📊 Provisioning Job Status

════════════════════════════════════════════════════════════════
  FastOrder Pre-Flight Validation Checks
════════════════════════════════════════════════════════════════

[INFO] Checking SSH connectivity to target host...
[✓] Target is localhost, skipping SSH check

[INFO] Checking available disk space...
[⚠] Disk space limited: 20GB available (recommended: 50GB)
  → PostgreSQL + Elasticsearch may experience space pressure

[INFO] Checking available memory...
[⚠] Memory low: 7GB (minimum: 4GB, recommended: 16GB)
  → Suitable for development/testing only
  → Reduce component counts: use 1 ES node, 1 PG worker, minimal standby nodes
  → Production environments require 16GB+

[INFO] Checking critical port availability...
[✓] Port 5432 available (PostgreSQL)
[✓] Port 9200 available (Elasticsearch)
[✓] Port 9300 available (Elasticsearch cluster)
[✓] Port 9092 available (Kafka)
[✓] Port 2181 available (Zookeeper)

[INFO] Checking DNS resolution...
[✓] DNS resolution working: google.com
[✓] DNS resolution working: github.com
[✓] DNS resolution working: archive.ubuntu.com

[INFO] Checking required system commands...
[✓] Command available: curl
[✓] Command available: wget
[✓] Command available: git
[✓] Command available: sudo
[✓] Command available: systemctl
[✓] Command available: apt-get

[INFO] Checking current system load...
[✓] System load normal: 0.06 (4 CPUs)

[INFO] Checking for existing environment conflicts...
[✓] No conflicting services found for: identity-uae-main-dev

════════════════════════════════════════════════════════════════
  Pre-Flight Check Summary
════════════════════════════════════════════════════════════════
[⚠] 2 warning(s) detected

⚠️  Environment can proceed with caution
   Review warnings above and consider remediation

[INFO] Using web-provided environment: identity-universe-main-dev
[INFO] Auto-creating state directory for identity-universe-main-dev...
[ OK ] Created topology.json for identity-universe-main-dev
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=10.100.1.51)
[2026-01-02_04:22:39] Starting Terraform provisioning step
[2026-01-02_04:22:39] Service: identity
[2026-01-02_04:22:39] Zone: universe
[2026-01-02_04:22:39] Environment: dev
[2026-01-02_04:22:39] Resource: web-03
[2026-01-02_04:22:39] Terraform binary: /home/ab/bin/terraform
[2026-01-02_04:22:39] HOME: /home/www-data
[2026-01-02_04:22:39] AWS Config: /home/ab/.aws/config
[2026-01-02_04:22:39] AWS Credentials: /home/ab/.aws/credentials
[2026-01-02_04:22:39] Terraform directory: /opt/fastorder/cli/terraform/examples/citus-production
[2026-01-02_04:22:39] Running terraform init...

Initializing the backend...
Upgrading modules...
- citus_cluster in ../../modules/citus_cluster

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 5.0"...
- Using previously-installed hashicorp/aws v5.100.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
[2026-01-02_04:22:46] ✓ Terraform init succeeded
[2026-01-02_04:22:46] Running terraform validate...
Success! The configuration is valid.

[2026-01-02_04:22:51] ✓ Terraform validate succeeded
[2026-01-02_04:22:51] Running terraform plan...
module.citus_cluster.data.aws_caller_identity.current: Reading...
module.citus_cluster.data.aws_caller_identity.current: Read complete after 0s [id=464621692046]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.citus_cluster.aws_iam_instance_profile.citus will be created
  + resource "aws_iam_instance_profile" "citus" {
      + arn         = (known after apply)
      + create_date = (known after apply)
      + id          = (known after apply)
      + name        = (known after apply)
      + name_prefix = "citus-prod-"
      + path        = "/"
      + role        = (known after apply)
      + tags        = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-prod"
        }
      + tags_all    = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
        }
      + unique_id   = (known after apply)
    }

  # module.citus_cluster.aws_iam_role.citus will be created
  + resource "aws_iam_role" "citus" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRole"
                      + Effect    = "Allow"
                      + Principal = {
                          + Service = "ec2.amazonaws.com"
                        }
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = (known after apply)
      + name_prefix           = "citus-prod-"
      + path                  = "/"
      + tags                  = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-prod"
        }
      + tags_all              = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
        }
      + unique_id             = (known after apply)
    }

  # module.citus_cluster.aws_iam_role_policy.secrets_manager[0] will be created
  + resource "aws_iam_role_policy" "secrets_manager" {
      + id          = (known after apply)
      + name        = (known after apply)
      + name_prefix = "secrets-access-"
      + policy      = jsonencode(
            {
              + Statement = [
                  + {
                      + Action   = [
                          + "secretsmanager:GetSecretValue",
                          + "secretsmanager:DescribeSecret",
                        ]
                      + Effect   = "Allow"
                      + Resource = "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/db/web/ksa/main/dev/postgresqladmin/ksa/prod*"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + role        = (known after apply)
    }

  # module.citus_cluster.aws_iam_role_policy_attachment.cloudwatch will be created
  + resource "aws_iam_role_policy_attachment" "cloudwatch" {
      + id         = (known after apply)
      + policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
      + role       = (known after apply)
    }

  # module.citus_cluster.aws_iam_role_policy_attachment.ssm will be created
  + resource "aws_iam_role_policy_attachment" "ssm" {
      + id         = (known after apply)
      + policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
      + role       = (known after apply)
    }

  # module.citus_cluster.aws_instance.coordinator will be created
  + resource "aws_instance" "coordinator" {
      + ami                                  = "ami-0b2aae5f4283c0df2"
      + arn                                  = (known after apply)
      + associate_public_ip_address          = (known after apply)
      + availability_zone                    = (known after apply)
      + cpu_core_count                       = (known after apply)
      + cpu_threads_per_core                 = (known after apply)
      + disable_api_stop                     = (known after apply)
      + disable_api_termination              = (known after apply)
      + ebs_optimized                        = (known after apply)
      + enable_primary_ipv6                  = (known after apply)
      + get_password_data                    = false
      + host_id                              = (known after apply)
      + host_resource_group_arn              = (known after apply)
      + iam_instance_profile                 = (known after apply)
      + id                                   = (known after apply)
      + instance_initiated_shutdown_behavior = (known after apply)
      + instance_lifecycle                   = (known after apply)
      + instance_state                       = (known after apply)
      + instance_type                        = "r6i.2xlarge"
      + ipv6_address_count                   = (known after apply)
      + ipv6_addresses                       = (known after apply)
      + key_name                             = (known after apply)
      + monitoring                           = (known after apply)
      + outpost_arn                          = (known after apply)
      + password_data                        = (known after apply)
      + placement_group                      = (known after apply)
      + placement_partition_number           = (known after apply)
      + primary_network_interface_id         = (known after apply)
      + private_dns                          = (known after apply)
      + private_ip                           = (known after apply)
      + public_dns                           = (known after apply)
      + public_ip                            = (known after apply)
      + secondary_private_ips                = (known after apply)
      + security_groups                      = (known after apply)
      + source_dest_check                    = true
      + spot_instance_request_id             = (known after apply)
      + subnet_id                            = "subnet-0a1f5a9a74ed030cf"
      + tags                                 = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-coordinator-prod"
          + "Role"        = "coordinator"
          + "Service"     = "citus"
        }
      + tags_all                             = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-coordinator-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
          + "Role"        = "coordinator"
          + "Service"     = "citus"
        }
      + tenancy                              = (known after apply)
      + user_data                            = "2a9e41ea765dcf3b3046ee10d2f458c18f00e430"
      + user_data_base64                     = (known after apply)
      + user_data_replace_on_change          = false
      + vpc_security_group_ids               = (known after apply)

      + ebs_block_device {
          + delete_on_termination = false
          + device_name           = "/dev/sdf"
          + encrypted             = true
          + iops                  = 3000
          + kms_key_id            = (known after apply)
          + snapshot_id           = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-coordinator-prod-data"
            }
          + tags_all              = (known after apply)
          + throughput            = 125
          + volume_id             = (known after apply)
          + volume_size           = 500
          + volume_type           = "gp3"
        }

      + root_block_device {
          + delete_on_termination = false
          + device_name           = (known after apply)
          + encrypted             = true
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-coordinator-prod-root"
            }
          + tags_all              = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = 100
          + volume_type           = "gp3"
        }
    }

  # module.citus_cluster.aws_instance.workers[0] will be created
  + resource "aws_instance" "workers" {
      + ami                                  = "ami-0b2aae5f4283c0df2"
      + arn                                  = (known after apply)
      + associate_public_ip_address          = (known after apply)
      + availability_zone                    = (known after apply)
      + cpu_core_count                       = (known after apply)
      + cpu_threads_per_core                 = (known after apply)
      + disable_api_stop                     = (known after apply)
      + disable_api_termination              = (known after apply)
      + ebs_optimized                        = (known after apply)
      + enable_primary_ipv6                  = (known after apply)
      + get_password_data                    = false
      + host_id                              = (known after apply)
      + host_resource_group_arn              = (known after apply)
      + iam_instance_profile                 = (known after apply)
      + id                                   = (known after apply)
      + instance_initiated_shutdown_behavior = (known after apply)
      + instance_lifecycle                   = (known after apply)
      + instance_state                       = (known after apply)
      + instance_type                        = "r6i.2xlarge"
      + ipv6_address_count                   = (known after apply)
      + ipv6_addresses                       = (known after apply)
      + key_name                             = (known after apply)
      + monitoring                           = (known after apply)
      + outpost_arn                          = (known after apply)
      + password_data                        = (known after apply)
      + placement_group                      = (known after apply)
      + placement_partition_number           = (known after apply)
      + primary_network_interface_id         = (known after apply)
      + private_dns                          = (known after apply)
      + private_ip                           = (known after apply)
      + public_dns                           = (known after apply)
      + public_ip                            = (known after apply)
      + secondary_private_ips                = (known after apply)
      + security_groups                      = (known after apply)
      + source_dest_check                    = true
      + spot_instance_request_id             = (known after apply)
      + subnet_id                            = "subnet-0a1f5a9a74ed030cf"
      + tags                                 = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-worker-0-prod"
          + "Role"        = "worker"
          + "Service"     = "citus"
          + "WorkerIndex" = "0"
        }
      + tags_all                             = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-worker-0-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
          + "Role"        = "worker"
          + "Service"     = "citus"
          + "WorkerIndex" = "0"
        }
      + tenancy                              = (known after apply)
      + user_data                            = "7b4bd87c9982aab7fa463c8d12e99399661f8bde"
      + user_data_base64                     = (known after apply)
      + user_data_replace_on_change          = false
      + vpc_security_group_ids               = (known after apply)

      + ebs_block_device {
          + delete_on_termination = false
          + device_name           = "/dev/sdf"
          + encrypted             = true
          + iops                  = 3000
          + kms_key_id            = (known after apply)
          + snapshot_id           = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-worker-0-prod-data"
            }
          + tags_all              = (known after apply)
          + throughput            = 125
          + volume_id             = (known after apply)
          + volume_size           = 500
          + volume_type           = "gp3"
        }

      + root_block_device {
          + delete_on_termination = false
          + device_name           = (known after apply)
          + encrypted             = true
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-worker-0-prod-root"
            }
          + tags_all              = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = 100
          + volume_type           = "gp3"
        }
    }

  # module.citus_cluster.aws_instance.workers[1] will be created
  + resource "aws_instance" "workers" {
      + ami                                  = "ami-0b2aae5f4283c0df2"
      + arn                                  = (known after apply)
      + associate_public_ip_address          = (known after apply)
      + availability_zone                    = (known after apply)
      + cpu_core_count                       = (known after apply)
      + cpu_threads_per_core                 = (known after apply)
      + disable_api_stop                     = (known after apply)
      + disable_api_termination              = (known after apply)
      + ebs_optimized                        = (known after apply)
      + enable_primary_ipv6                  = (known after apply)
      + get_password_data                    = false
      + host_id                              = (known after apply)
      + host_resource_group_arn              = (known after apply)
      + iam_instance_profile                 = (known after apply)
      + id                                   = (known after apply)
      + instance_initiated_shutdown_behavior = (known after apply)
      + instance_lifecycle                   = (known after apply)
      + instance_state                       = (known after apply)
      + instance_type                        = "r6i.2xlarge"
      + ipv6_address_count                   = (known after apply)
      + ipv6_addresses                       = (known after apply)
      + key_name                             = (known after apply)
      + monitoring                           = (known after apply)
      + outpost_arn                          = (known after apply)
      + password_data                        = (known after apply)
      + placement_group                      = (known after apply)
      + placement_partition_number           = (known after apply)
      + primary_network_interface_id         = (known after apply)
      + private_dns                          = (known after apply)
      + private_ip                           = (known after apply)
      + public_dns                           = (known after apply)
      + public_ip                            = (known after apply)
      + secondary_private_ips                = (known after apply)
      + security_groups                      = (known after apply)
      + source_dest_check                    = true
      + spot_instance_request_id             = (known after apply)
      + subnet_id                            = "subnet-02c930351cde1e9c3"
      + tags                                 = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-worker-1-prod"
          + "Role"        = "worker"
          + "Service"     = "citus"
          + "WorkerIndex" = "1"
        }
      + tags_all                             = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-worker-1-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
          + "Role"        = "worker"
          + "Service"     = "citus"
          + "WorkerIndex" = "1"
        }
      + tenancy                              = (known after apply)
      + user_data                            = "7b4bd87c9982aab7fa463c8d12e99399661f8bde"
      + user_data_base64                     = (known after apply)
      + user_data_replace_on_change          = false
      + vpc_security_group_ids               = (known after apply)

      + ebs_block_device {
          + delete_on_termination = false
          + device_name           = "/dev/sdf"
          + encrypted             = true
          + iops                  = 3000
          + kms_key_id            = (known after apply)
          + snapshot_id           = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-worker-1-prod-data"
            }
          + tags_all              = (known after apply)
          + throughput            = 125
          + volume_id             = (known after apply)
          + volume_size           = 500
          + volume_type           = "gp3"
        }

      + root_block_device {
          + delete_on_termination = false
          + device_name           = (known after apply)
          + encrypted             = true
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-worker-1-prod-root"
            }
          + tags_all              = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = 100
          + volume_type           = "gp3"
        }
    }

  # module.citus_cluster.aws_security_group.citus will be created
  + resource "aws_security_group" "citus" {
      + arn                    = (known after apply)
      + description            = "Security group for Citus cluster"
      + egress                 = [
          + {
              + cidr_blocks      = [
                  + "0.0.0.0/0",
                ]
              + description      = "Allow all outbound"
              + from_port        = 0
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "-1"
              + security_groups  = []
              + self             = false
              + to_port          = 0
            },
        ]
      + id                     = (known after apply)
      + ingress                = [
          + {
              + cidr_blocks      = [
                  + "10.0.0.0/8",
                ]
              + description      = "PgBouncer access"
              + from_port        = 6432
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 6432
            },
          + {
              + cidr_blocks      = [
                  + "10.0.0.0/8",
                ]
              + description      = "PostgreSQL access"
              + from_port        = 5432
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 5432
            },
          + {
              + cidr_blocks      = [
                  + "10.0.0.0/8",
                ]
              + description      = "SSH access"
              + from_port        = 22
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 22
            },
          + {
              + cidr_blocks      = []
              + description      = "Internal cluster communication"
              + from_port        = 0
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = true
              + to_port          = 65535
            },
        ]
      + name                   = (known after apply)
      + name_prefix            = "citus-prod-"
      + owner_id               = (known after apply)
      + revoke_rules_on_delete = false
      + tags                   = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-prod"
          + "Service"     = "citus"
        }
      + tags_all               = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
          + "Service"     = "citus"
        }
      + vpc_id                 = "vpc-0af7da1e7d94d62bd"
    }

Plan: 9 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + connection_string = (sensitive value)
  + coordinator_ip    = (known after apply)
  + worker_ips        = [
      + (known after apply),
      + (known after apply),
    ]

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"
[2026-01-02_04:22:57] ✓ Terraform plan succeeded
[2026-01-02_04:22:57] Generating plan JSON...
[2026-01-02_04:23:02] ✓ Terraform provisioning step completed successfully

Next step: Review the plan and apply with 'terraform apply tfplan'

[INFO] FastOrder Environment Preparation
[INFO] Service: identity
[INFO] Zone: universe
[INFO] Environment: dev
[INFO] Branch: main
[INFO] State Directory: /opt/fastorder/bash/scripts/env_app_setup/state
[INFO] Library: /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] IP: 142.93.238.16 (specified)

[INFO] Creating environment using fo-env...
[INFO] Creating new FastOrder environment (v1 topology)
[INFO] Generated environment ID: identity-universe-main-dev
[INFO] Using provided IP: 142.93.238.16
[INFO] Allocated interface: eth0:16
[INFO] Configuring network interface for VM IP: 142.93.238.16
[INFO] VM IP 142.93.238.16 is already configured on eth0:16
[CONFIG] No web configuration found for environment: identity-universe-main-dev
[CONFIG] Using defaults: ES_NODES=1, PG_WORKERS=1
[INFO] Service enabled flags: db=no, eventbus=no, search=yes
[ OK ] Created topology.json at /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
[ OK ] Generated overlay configurations in /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/generated/
[ OK ] Updated environments.json
[ OK ] Updated setup.json
[ OK ] Environment created successfully!
[INFO] 
[INFO] Environment Details:
[INFO]   ID: identity-universe-main-dev
[INFO]   Service: identity
[INFO]   zone: universe
[INFO]   Environment: dev
[INFO]   Branch: main
[INFO]   IP: 142.93.238.16
[INFO]   Interface: eth0:16
[INFO] 
[INFO] Configuration files:
[INFO]   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
[INFO]   Generated: /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/generated/*.env
[INFO]   Overrides: /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/overrides/*.env
[INFO] 
[INFO] To use this environment:
[INFO]   export ENV_ID="identity-universe-main-dev"
[INFO]   source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO]   init_environment
[ OK ] Environment preparation completed successfully!
[INFO] Creating topology from web form submission...
[INFO] Loaded from topology.json: identity-universe-main-dev
[2026-01-02 04:23:04] Loaded environment: identity-universe-main-dev
[2026-01-02 04:23:04] Service: identity, Zone: universe, Branch: main, Env: dev
[2026-01-02 04:23:04] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-02 04:23:04] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-02 04:23:04] PostgreSQL HA Nodes: 1, Citus Enabled: yes
[ OK ] Environment initialized successfully (mode: general)
[INFO] Creating topology.json from web form submission...
[INFO] DEBUG: Service enabled flags...
[INFO]   DB_ENABLED=no
[INFO]   EVENTBUS_ENABLED=no
[INFO]   SEARCH_ENABLED=yes
[INFO] DEBUG: Checking for form submission variables...
[INFO]   service_es_ip=10.100.1.4
[INFO]   service_es_fqdn=search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com
[INFO]   service_pg_coordinator_ip=NOT SET
[WARN] IP 10.100.1.4 is already allocated, allocating new IP for search
[INFO] Adding search: search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.178) [reallocated from 10.100.1.4]
[WARN] IP 10.100.1.6 is already allocated, allocating new IP for search-node-01
[INFO] Adding search-node-01: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.179) [reallocated from 10.100.1.6]
[INFO] Skipping Kafka/EventBus - EVENTBUS_ENABLED=no
[INFO] Skipping PostgreSQL/PgBouncer - DB_ENABLED=no
[WARN] IP 10.100.1.8 is already allocated, allocating new IP for obs
[INFO] Adding obs: obs-identity-universe-main-dev.fastorder.com (10.100.1.180) [reallocated from 10.100.1.8]
[ OK ] Topology created from form data
[INFO] Applications registered:
  ✓ obs: obs-identity-universe-main-dev.fastorder.com (10.100.1.180)
  ✓ search: search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.178)
  ✓ search-node-01: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.179)
[ OK ] Topology created from form data

[INFO] Next steps:
[INFO] 1. Review the generated topology.json and configurations
[INFO] 2. Customize overrides/*.env files if needed
[INFO] 3. Run subsequent installation steps (02-install-postgresql, etc.)

[INFO] To use this environment in other scripts:
[INFO]   export ENV_ID="$(fo-env list | tail -n1 | awk '{print $1}')"
[INFO]   source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO]   init_environment

[INFO] Using IAM engine from IAM_ENGINE environment variable: keycloak
[INFO] Cleaning up any existing locks...

Starting IAM engine: keycloak
═══════════════════════════════════════════════

[INFO] Logging to: /var/log/fastorder/iam/keycloak-deploy-20260102-042307.log
[INFO] Loaded from topology.json: identity-universe-main-dev
[2026-01-02 04:23:08] Loaded environment: identity-universe-main-dev
[2026-01-02 04:23:08] Service: identity, Zone: universe, Branch: main, Env: dev
[2026-01-02 04:23:08] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-02 04:23:08] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-02 04:23:08] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
[INFO] Keycloak Configuration:
[INFO]   Version: 24.0.4
[INFO]   Node Role: controller
[INFO]   HA Cluster: false
[INFO]   Storage Backend: domain-db
[INFO]   Session Backend: redis
[INFO]   mTLS Enabled: true

[INFO] Starting Keycloak IAM setup process...
[INFO] Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/02-iam/engine/keycloak/steps
[INFO] Environment: iam-identity-universe-main-dev

[INFO] Found 12 step(s) to execute

[INFO] 🔐 Step 1/12: install keycloak...
[INFO] Allocated Keycloak IP from IAM subnet: 10.100.60.2
[INFO] Installing Keycloak 24.0.4 for environment: iam-identity-universe-main-dev
[INFO]   FQDN: keycloak-iam-identity-universe-main-dev.fastorder.com
[INFO]   IP: 10.100.60.2
[INFO] Checking Java installation...
[OK] Java installed: openjdk version "17.0.17" 2025-10-21
[INFO] Configuring network interface for Keycloak IP: 10.100.60.2
[INFO] Adding IP 10.100.60.2 to loopback interface...
[2026-01-02 04:23:08 UTC] USER=www-data EUID=0 PID=1375797 ACTION=configure-network-interface ARGS=lo:keycloak 10.100.60.2
✓ lo:keycloak <- 10.100.60.2
[OK] IP 10.100.60.2 configured on lo:keycloak
[INFO] Creating persistent IP service: vm-ip-10-100-60-2
[2026-01-02 04:23:08 UTC] USER=www-data EUID=0 PID=1375825 ACTION=fsop ARGS=mv /tmp/vm-ip-10-100-60-2.service /etc/systemd/system/vm-ip-10-100-60-2.service
[2026-01-02 04:23:08 UTC] USER=www-data EUID=0 PID=1375846 ACTION=fsop ARGS=chmod 644 /etc/systemd/system/vm-ip-10-100-60-2.service
[2026-01-02 04:23:08 UTC] USER=www-data EUID=0 PID=1375867 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:23:09 UTC] USER=www-data EUID=0 PID=1375924 ACTION=passthru ARGS=systemctl enable vm-ip-10-100-60-2.service
Created symlink /etc/systemd/system/multi-user.target.wants/vm-ip-10-100-60-2.service -> /etc/systemd/system/vm-ip-10-100-60-2.service.
[2026-01-02 04:23:09 UTC] USER=www-data EUID=0 PID=1375981 ACTION=passthru ARGS=systemctl start vm-ip-10-100-60-2.service
[OK] Created and enabled vm-ip-10-100-60-2.service
[INFO] Setting up Keycloak user and group...
[INFO] Creating directory structure...
[2026-01-02 04:23:09 UTC] USER=www-data EUID=0 PID=1376006 ACTION=fsop ARGS=mkdir -p /opt/keycloak
[2026-01-02 04:23:09 UTC] USER=www-data EUID=0 PID=1376027 ACTION=fsop ARGS=mkdir -p /var/lib/keycloak
[2026-01-02 04:23:09 UTC] USER=www-data EUID=0 PID=1376048 ACTION=fsop ARGS=mkdir -p /var/log/keycloak
[2026-01-02 04:23:10 UTC] USER=www-data EUID=0 PID=1376069 ACTION=fsop ARGS=mkdir -p /etc/keycloak/iam-identity-universe-main-dev
[2026-01-02 04:23:10 UTC] USER=www-data EUID=0 PID=1376090 ACTION=fsop ARGS=mkdir -p /etc/keycloak/iam-identity-universe-main-dev/certs
[2026-01-02 04:23:10 UTC] USER=www-data EUID=0 PID=1376111 ACTION=fsop ARGS=mkdir -p /etc/keycloak/iam-identity-universe-main-dev/providers
[2026-01-02 04:23:10 UTC] USER=www-data EUID=0 PID=1376132 ACTION=fsop ARGS=mkdir -p /etc/keycloak/iam-identity-universe-main-dev/themes
[2026-01-02 04:23:10 UTC] USER=www-data EUID=0 PID=1376153 ACTION=fsop ARGS=mkdir -p /opt/keycloak/data/tmp
[OK] Created directories
[INFO] Downloading Keycloak 24.0.4...
[INFO] Extracting Keycloak...
[2026-01-02 04:23:13 UTC] USER=www-data EUID=0 PID=1376179 ACTION=passthru ARGS=tar -xzf /tmp/keycloak-24.0.4.tar.gz -C /opt/
[2026-01-02 04:23:15 UTC] USER=www-data EUID=0 PID=1376203 ACTION=fsop ARGS=cp -r /opt/keycloak-24.0.4/LICENSE.txt /opt/keycloak-24.0.4/README.md /opt/keycloak-24.0.4/bin /opt/keycloak-24.0.4/conf /opt/keycloak-24.0.4/lib /opt/keycloak-24.0.4/providers /opt/keycloak-24.0.4/themes /opt/keycloak-24.0.4/version.txt /opt/keycloak/
[2026-01-02 04:23:16 UTC] USER=www-data EUID=0 PID=1376226 ACTION=fsop ARGS=rm -rf /opt/keycloak-24.0.4
[OK] Keycloak installed to /opt/keycloak
[INFO] Setting permissions...
[2026-01-02 04:23:16 UTC] USER=www-data EUID=0 PID=1376248 ACTION=fsop ARGS=chown -R keycloak:keycloak /opt/keycloak
[2026-01-02 04:23:16 UTC] USER=www-data EUID=0 PID=1376269 ACTION=fsop ARGS=chown -R keycloak:keycloak /var/lib/keycloak
[2026-01-02 04:23:16 UTC] USER=www-data EUID=0 PID=1376290 ACTION=fsop ARGS=chown -R keycloak:keycloak /var/log/keycloak
[2026-01-02 04:23:16 UTC] USER=www-data EUID=0 PID=1376311 ACTION=fsop ARGS=chown -R keycloak:keycloak /etc/keycloak
[2026-01-02 04:23:16 UTC] USER=www-data EUID=0 PID=1376332 ACTION=fsop ARGS=chmod 750 /opt/keycloak
[2026-01-02 04:23:16 UTC] USER=www-data EUID=0 PID=1376353 ACTION=fsop ARGS=chmod 750 /var/lib/keycloak
[2026-01-02 04:23:16 UTC] USER=www-data EUID=0 PID=1376374 ACTION=fsop ARGS=chmod 750 /var/log/keycloak
[2026-01-02 04:23:17 UTC] USER=www-data EUID=0 PID=1376395 ACTION=fsop ARGS=chmod 750 /etc/keycloak
[OK] Permissions set
[INFO] Creating Keycloak configuration...
[2026-01-02 04:23:17 UTC] USER=www-data EUID=0 PID=1376418 ACTION=fsop ARGS=mv /tmp/keycloak-iam-identity-universe-main-dev.conf /etc/keycloak/iam-identity-universe-main-dev/keycloak.conf
[2026-01-02 04:23:17 UTC] USER=www-data EUID=0 PID=1376439 ACTION=fsop ARGS=chown keycloak:keycloak /etc/keycloak/iam-identity-universe-main-dev/keycloak.conf
[2026-01-02 04:23:17 UTC] USER=www-data EUID=0 PID=1376460 ACTION=fsop ARGS=chmod 640 /etc/keycloak/iam-identity-universe-main-dev/keycloak.conf
[2026-01-02 04:23:17 UTC] USER=www-data EUID=0 PID=1376481 ACTION=fsop ARGS=mkdir -p /var/log/keycloak/iam-identity-universe-main-dev
[2026-01-02 04:23:17 UTC] USER=www-data EUID=0 PID=1376502 ACTION=fsop ARGS=chown keycloak:keycloak /var/log/keycloak/iam-identity-universe-main-dev
[OK] Configuration created at /etc/keycloak/iam-identity-universe-main-dev/keycloak.conf
[INFO] Creating systemd service...
[2026-01-02 04:23:17 UTC] USER=www-data EUID=0 PID=1376527 ACTION=fsop ARGS=mv /tmp/keycloak-iam-identity-universe-main-dev.service /etc/systemd/system/keycloak-iam-identity-universe-main-dev.service
[2026-01-02 04:23:17 UTC] USER=www-data EUID=0 PID=1376548 ACTION=fsop ARGS=chmod 644 /etc/systemd/system/keycloak-iam-identity-universe-main-dev.service
[2026-01-02 04:23:17 UTC] USER=www-data EUID=0 PID=1376569 ACTION=fsop ARGS=mkdir -p /var/lib/keycloak/iam-identity-universe-main-dev
[2026-01-02 04:23:17 UTC] USER=www-data EUID=0 PID=1376590 ACTION=fsop ARGS=chown keycloak:keycloak /var/lib/keycloak/iam-identity-universe-main-dev
[2026-01-02 04:23:17 UTC] USER=www-data EUID=0 PID=1376611 ACTION=passthru ARGS=systemctl daemon-reload
[OK] Systemd service created: keycloak-iam-identity-universe-main-dev
[INFO] Configuring /etc/hosts for Keycloak FQDNs -> 10.100.60.2
[2026-01-02 04:23:18 UTC] USER=www-data EUID=0 PID=1376668 ACTION=fsop ARGS=sed -i /keycloak-iam-identity-universe-main-dev.fastorder.com/d /etc/hosts
[2026-01-02 04:23:18 UTC] USER=www-data EUID=0 PID=1376689 ACTION=fsop ARGS=sed -i /keycloak-iam-identity-universe-main-dev-controller.fastorder.com/d /etc/hosts
[OK] Added keycloak-iam-identity-universe-main-dev.fastorder.com to /etc/hosts
[OK] Added keycloak-iam-identity-universe-main-dev-controller.fastorder.com to /etc/hosts
[INFO] Building optimized Keycloak distribution...
2026-01-02 04:23:32,246 INFO  [io.qua.dep.QuarkusAugmentor] (main) Quarkus augmentation completed in 10967ms
Server configuration updated and persisted. Run the following command to review the configuration:

	kc.sh show-config

[OK] Keycloak build completed

[OK] Keycloak installation completed
[INFO]   Version: 24.0.4
[INFO]   FQDN: keycloak-iam-identity-universe-main-dev.fastorder.com
[INFO]   IP: 10.100.60.2
[INFO]   Home: /opt/keycloak
[INFO]   Config: /etc/keycloak/iam-identity-universe-main-dev
[INFO]   Data: /var/lib/keycloak/iam-identity-universe-main-dev
[INFO]   Logs: /var/log/keycloak/iam-identity-universe-main-dev
[INFO]   Service: keycloak-iam-identity-universe-main-dev

[WARN] Next steps:
[WARN]   1. Configure TLS certificates (02-setup-tls.sh)
[WARN]   2. Configure database connection (03-database-setup.sh)
[WARN]   3. Create admin credentials (04-create-admin.sh)
[WARN]   4. Start service: sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl start keycloak-iam-identity-universe-main-dev
[OK] ✅ Step 1 completed: 01-install-keycloak.sh

[INFO] 🔐 Step 2/12: setup tls...
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
[CREDS] Using AWS credentials from: /var/www/.aws/credentials
[CREDS] Credential management library loaded (region: me-central-1)
[INFO] Using Keycloak internal IP: 10.100.60.2
[INFO] Using Keycloak FQDN: keycloak-iam-identity-universe-main-dev.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Keycloak TLS Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Zone:        universe (FastOrder Universe)
  FQDN:        keycloak-iam-identity-universe-main-dev.fastorder.com
  Internal IP: 10.100.60.2
  Country:     US
  State:       Virginia
  City:        Ashburn
  mTLS:        true
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Zone 'universe' has public DNS - using Let's Encrypt for trusted certificates
[INFO] Delegating to Let's Encrypt TLS setup...
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Keycloak TLS Setup - Let's Encrypt
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  FQDN:        keycloak-iam-identity-universe-main-dev.fastorder.com
  VM_IP:       142.93.238.16
  Email:       admin@fastorder.com
  Webroot:     /var/www/html
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Installing certbot and Apache...
[2026-01-02 04:23:33 UTC] USER=www-data EUID=0 PID=1377044 ACTION=pkg ARGS=apt-get update -qq
ERROR: pkg subcommand not allowed: apt-get
[OK] Certbot and Apache installed
[INFO] Creating certificate directories...
[2026-01-02 04:23:33 UTC] USER=www-data EUID=0 PID=1377128 ACTION=fsop ARGS=mkdir -p /etc/keycloak/iam-identity-universe-main-dev/certs
[2026-01-02 04:23:33 UTC] USER=www-data EUID=0 PID=1377149 ACTION=fsop ARGS=mkdir -p /etc/keycloak/iam-identity-universe-main-dev/certs/letsencrypt
[2026-01-02 04:23:34 UTC] USER=www-data EUID=0 PID=1377170 ACTION=fsop ARGS=chown -R keycloak:keycloak /etc/keycloak/iam-identity-universe-main-dev/certs
[2026-01-02 04:23:34 UTC] USER=www-data EUID=0 PID=1377191 ACTION=fsop ARGS=chmod 750 /etc/keycloak/iam-identity-universe-main-dev/certs
[2026-01-02 04:23:34 UTC] USER=www-data EUID=0 PID=1377213 ACTION=passthru ARGS=mkdir -p /var/www/html/.well-known/acme-challenge
[2026-01-02 04:23:34 UTC] USER=www-data EUID=0 PID=1377234 ACTION=passthru ARGS=chown -R www-data:www-data /var/www/html/.well-known
[OK] Directories created
[INFO] Configuring Apache for ACME challenge...
[2026-01-02 04:23:34 UTC] USER=www-data EUID=0 PID=1377301 ACTION=passthru ARGS=systemctl reload apache2
[OK] Apache HTTP vhost configured for ACME
[INFO] Obtaining Let's Encrypt certificate for keycloak-iam-identity-universe-main-dev.fastorder.com...
[WARN] Certificate expires in -20455 days, renewing...
[2026-01-02 04:23:34 UTC] USER=www-data EUID=0 PID=1377363 ACTION=certbot ARGS=certonly --webroot -w /var/www/html -d keycloak-iam-identity-universe-main-dev.fastorder.com --cert-name keycloak-iam-identity-universe-main-dev.fastorder.com --agree-tos --email admin@fastorder.com --non-interactive --keep-until-expiring
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Certificate not yet due for renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate not yet due for renewal; no action taken.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[OK] Let's Encrypt certificate obtained

Certificate details:
[2026-01-02 04:23:36 UTC] USER=www-data EUID=0 PID=1377389 ACTION=fsop ARGS=openssl x509 -in /etc/letsencrypt/live/keycloak-iam-identity-universe-main-dev.fastorder.com/fullchain.pem -noout -subject -issuer -dates
subject=CN = keycloak-iam-identity-universe-main-dev.fastorder.com
issuer=C = US, O = Let's Encrypt, CN = R12
notBefore=Dec  8 06:46:33 2025 GMT
notAfter=Mar  8 06:46:32 2026 GMT
[INFO] Creating PKCS12 keystore for Keycloak...
[2026-01-02 04:23:36 UTC] USER=www-data EUID=0 PID=1377414 ACTION=fsop ARGS=rm -f /etc/keycloak/iam-identity-universe-main-dev/certs/keystore.p12
[2026-01-02 04:23:36 UTC] USER=www-data EUID=0 PID=1377435 ACTION=fsop ARGS=openssl pkcs12 -export -in /etc/letsencrypt/live/keycloak-iam-identity-universe-main-dev.fastorder.com/fullchain.pem -inkey /etc/letsencrypt/live/keycloak-iam-identity-universe-main-dev.fastorder.com/privkey.pem -out /tmp/keycloak-keystore.p12 -name keycloak-server -password pass:nPQ9vS67YQES5nCHs71icSAURadmWWGA
[2026-01-02 04:23:36 UTC] USER=www-data EUID=0 PID=1377456 ACTION=fsop ARGS=cp /tmp/keycloak-keystore.p12 /etc/keycloak/iam-identity-universe-main-dev/certs/keystore.p12
[2026-01-02 04:23:37 UTC] USER=www-data EUID=0 PID=1377477 ACTION=fsop ARGS=chmod 600 /etc/keycloak/iam-identity-universe-main-dev/certs/keystore.p12
[2026-01-02 04:23:37 UTC] USER=www-data EUID=0 PID=1377498 ACTION=fsop ARGS=chown keycloak:keycloak /etc/keycloak/iam-identity-universe-main-dev/certs/keystore.p12
rm: cannot remove '/tmp/keycloak-keystore.p12': Operation not permitted
[2026-01-02 04:23:37 UTC] USER=www-data EUID=0 PID=1377543 ACTION=fsop ARGS=chmod 600 /etc/keycloak/iam-identity-universe-main-dev/certs/keystore.password
[2026-01-02 04:23:37 UTC] USER=www-data EUID=0 PID=1377564 ACTION=fsop ARGS=chown keycloak:keycloak /etc/keycloak/iam-identity-universe-main-dev/certs/keystore.password
[OK] PKCS12 keystore created
[INFO] Copying certificates for Keycloak...
[2026-01-02 04:23:37 UTC] USER=www-data EUID=0 PID=1377585 ACTION=passthru ARGS=cp /etc/letsencrypt/live/keycloak-iam-identity-universe-main-dev.fastorder.com/fullchain.pem /tmp/server.crt
[2026-01-02 04:23:37 UTC] USER=www-data EUID=0 PID=1377606 ACTION=passthru ARGS=cp /etc/letsencrypt/live/keycloak-iam-identity-universe-main-dev.fastorder.com/privkey.pem /tmp/server.key
[2026-01-02 04:23:37 UTC] USER=www-data EUID=0 PID=1377627 ACTION=fsop ARGS=cp /tmp/server.crt /etc/keycloak/iam-identity-universe-main-dev/certs/server.crt
[2026-01-02 04:23:37 UTC] USER=www-data EUID=0 PID=1377648 ACTION=fsop ARGS=cp /tmp/server.key /etc/keycloak/iam-identity-universe-main-dev/certs/server.key
[2026-01-02 04:23:37 UTC] USER=www-data EUID=0 PID=1377669 ACTION=fsop ARGS=chmod 644 /etc/keycloak/iam-identity-universe-main-dev/certs/server.crt
[2026-01-02 04:23:37 UTC] USER=www-data EUID=0 PID=1377690 ACTION=fsop ARGS=chmod 600 /etc/keycloak/iam-identity-universe-main-dev/certs/server.key
[2026-01-02 04:23:37 UTC] USER=www-data EUID=0 PID=1377711 ACTION=fsop ARGS=chown keycloak:keycloak /etc/keycloak/iam-identity-universe-main-dev/certs/server.crt /etc/keycloak/iam-identity-universe-main-dev/certs/server.key
rm: cannot remove '/tmp/server.crt': Operation not permitted
rm: cannot remove '/tmp/server.key': Operation not permitted
[2026-01-02 04:23:38 UTC] USER=www-data EUID=0 PID=1377733 ACTION=passthru ARGS=cp /etc/letsencrypt/live/keycloak-iam-identity-universe-main-dev.fastorder.com/chain.pem /tmp/ca.crt
[2026-01-02 04:23:38 UTC] USER=www-data EUID=0 PID=1377754 ACTION=fsop ARGS=mkdir -p /etc/keycloak/iam-identity-universe-main-dev/certs/ca
[2026-01-02 04:23:38 UTC] USER=www-data EUID=0 PID=1377775 ACTION=fsop ARGS=cp /tmp/ca.crt /etc/keycloak/iam-identity-universe-main-dev/certs/ca/ca.crt
[2026-01-02 04:23:38 UTC] USER=www-data EUID=0 PID=1377796 ACTION=fsop ARGS=chmod 644 /etc/keycloak/iam-identity-universe-main-dev/certs/ca/ca.crt
[2026-01-02 04:23:38 UTC] USER=www-data EUID=0 PID=1377817 ACTION=fsop ARGS=chown keycloak:keycloak /etc/keycloak/iam-identity-universe-main-dev/certs/ca/ca.crt
rm: cannot remove '/tmp/ca.crt': Operation not permitted
[OK] Certificates copied
[INFO] Configuring Apache HTTPS reverse proxy...
[2026-01-02 04:23:38 UTC] USER=www-data EUID=0 PID=1377910 ACTION=passthru ARGS=systemctl reload apache2
[OK] Apache HTTPS reverse proxy configured
[INFO] Updating Keycloak configuration...
[2026-01-02 04:23:39 UTC] USER=www-data EUID=0 PID=1377972 ACTION=fsop ARGS=chown keycloak:keycloak /etc/keycloak/iam-identity-universe-main-dev/keycloak.conf
[OK] Keycloak configuration updated
[INFO] Setting up automatic certificate renewal...
[2026-01-02 04:23:39 UTC] USER=www-data EUID=0 PID=1377993 ACTION=passthru ARGS=mkdir -p /etc/letsencrypt/renewal-hooks/deploy
[2026-01-02 04:23:39 UTC] USER=www-data EUID=0 PID=1378035 ACTION=passthru ARGS=chmod +x /etc/letsencrypt/renewal-hooks/deploy/keycloak-iam-identity-universe-main-dev.sh
[2026-01-02 04:23:39 UTC] USER=www-data EUID=0 PID=1378056 ACTION=passthru ARGS=systemctl enable certbot.timer
[2026-01-02 04:23:39 UTC] USER=www-data EUID=0 PID=1378113 ACTION=passthru ARGS=systemctl start certbot.timer
[OK] Automatic renewal configured
[INFO] Storing credentials in Secrets Manager...
[WARN] Secrets manager library not found, password stored locally only

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Keycloak Let's Encrypt TLS Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: iam-identity-universe-main-dev
Domain:      keycloak-iam-identity-universe-main-dev.fastorder.com

Certificate files:
  📜 Server cert:    /etc/keycloak/iam-identity-universe-main-dev/certs/server.crt
  🔑 Server key:     /etc/keycloak/iam-identity-universe-main-dev/certs/server.key
  🏛️  CA chain:       /etc/keycloak/iam-identity-universe-main-dev/certs/ca/ca.crt
  📦 Keystore:       /etc/keycloak/iam-identity-universe-main-dev/certs/keystore.p12
  🔄 Let's Encrypt:  /etc/letsencrypt/live/keycloak-iam-identity-universe-main-dev.fastorder.com/

Apache vhosts:
  HTTP:  /etc/apache2/sites-available/keycloak-iam-identity-universe-main-dev.fastorder.com.conf
  HTTPS: /etc/apache2/sites-available/keycloak-iam-identity-universe-main-dev.fastorder.com-ssl.conf

Access URLs:
  🌐 https://keycloak-iam-identity-universe-main-dev.fastorder.com/admin
  🌐 https://keycloak-iam-identity-universe-main-dev.fastorder.com/realms/master

Renewal:
  📅 Automatic renewal via certbot timer
  🔧 Hook script: /etc/letsencrypt/renewal-hooks/deploy/keycloak-iam-identity-universe-main-dev.sh

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Step 2 completed: 02-setup-tls.sh

[INFO] 🔐 Step 3/12: observability cell setup...
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║              IAM Observability Cell Provisioning                            ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] Environment: iam-identity-universe-main-dev
[INFO] Service: iam-identity
[INFO] Identifier: controller

[INFO] Setting up observability cell for IAM...
[INFO]   Service: iam-identity (SERVICE=iam-identity)
[INFO]   Environment: iam-identity-universe-main-dev

[INFO] Executing observability cell setup...
[INFO]   Passing SERVICE=iam-identity to observability-cell

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🚀 OBSERVABILITY CELL PROVISIONING STARTED
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Script: 02-observability-cell/run.sh
[INFO] Timestamp: 2026-01-02 04:23:40 UTC
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Ensuring correct permissions for observability deployment...
[2026-01-02 04:23:40 UTC] USER=www-data EUID=0 PID=1378151 ACTION=fsop ARGS=chmod 775 /var/log/fastorder
[2026-01-02 04:23:40 UTC] USER=www-data EUID=0 PID=1378160 ACTION=fsop ARGS=chown www-data:www-data /var/log/fastorder
[2026-01-02 04:23:40 UTC] USER=www-data EUID=0 PID=1378169 ACTION=fsop ARGS=touch /var/log/fastorder/provisioning-elevated.log
[2026-01-02 04:23:40 UTC] USER=www-data EUID=0 PID=1378178 ACTION=fsop ARGS=chmod 666 /var/log/fastorder/provisioning-elevated.log
[2026-01-02 04:23:40 UTC] USER=www-data EUID=0 PID=1378187 ACTION=fsop ARGS=chown www-data:www-data /var/log/fastorder/provisioning-elevated.log
[OK]   Log directory: /var/log/fastorder (775)
[OK]   Log file: provisioning-elevated.log (666)
[2026-01-02 04:23:40 UTC] USER=www-data EUID=0 PID=1378196 ACTION=fsop ARGS=chmod 775 /opt/fastorder/bash/scripts/env_app_setup/state
[OK]   State directory: 775
[2026-01-02 04:23:40 UTC] USER=www-data EUID=0 PID=1378205 ACTION=fsop ARGS=mkdir -p /etc/fastorder/observability/certs
[2026-01-02 04:23:40 UTC] USER=www-data EUID=0 PID=1378214 ACTION=fsop ARGS=chmod 750 /etc/fastorder/observability/certs
[OK]   Cert directory: /etc/fastorder/observability/certs (750 - secure)
[OK]   Lib scripts: executable (755)
[OK]   All deployment scripts: executable (755)
[OK]   All directories: accessible (755)
[OK]   ✅ All permissions verified and fixed
[CREDS] Using AWS credentials from: /var/www/.aws/credentials
[CREDS] Credential management library loaded (region: me-central-1)
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
═══════════════════════════════════════════════════════════════════════════════
OBSERVABILITY CELL PROVISIONING
═══════════════════════════════════════════════════════════════════════════════
[INFO] Application Cell: iam-identity-universe-main-dev
[INFO] Observability Cell: obs-iam-identity-universe-main-dev
[INFO] Service: iam-identity | Zone: universe | Env: dev

[INFO] Step 1/10: Provisioning network infrastructure...
[INFO]   Allocated new IP for obs: 10.100.1.181
[WARN] Topology file not found: /opt/fastorder/bash/scripts/env_app_setup/state/iam-identity-universe-main-dev/topology.json
[INFO]   Allocated new IP for metrics: 10.100.1.182
[WARN] Topology file not found: /opt/fastorder/bash/scripts/env_app_setup/state/iam-identity-universe-main-dev/topology.json
[INFO]   Allocated new IP for dashboards: 10.100.1.183
[WARN] Topology file not found: /opt/fastorder/bash/scripts/env_app_setup/state/iam-identity-universe-main-dev/topology.json
[INFO]   Allocated new IP for logstore: 10.100.1.184
[WARN] Topology file not found: /opt/fastorder/bash/scripts/env_app_setup/state/iam-identity-universe-main-dev/topology.json
[INFO]   Allocated new IP for traces: 10.100.1.186
[WARN] Topology file not found: /opt/fastorder/bash/scripts/env_app_setup/state/iam-identity-universe-main-dev/topology.json
[INFO]   Allocated new IP for alerts: 10.100.1.187
[WARN] Topology file not found: /opt/fastorder/bash/scripts/env_app_setup/state/iam-identity-universe-main-dev/topology.json
[INFO]   Allocated new IP for telemetry: 10.100.1.188
[WARN] Topology file not found: /opt/fastorder/bash/scripts/env_app_setup/state/iam-identity-universe-main-dev/topology.json
[INFO]   Allocated observability IPs:
[INFO]     metrics: 10.100.1.182
[INFO]     alerts: 10.100.1.187
[INFO]     dashboards: 10.100.1.183
[INFO]     traces: 10.100.1.186
[INFO]     telemetry: 10.100.1.188
[INFO]     logstore: 10.100.1.184
[INFO]     proxy: 10.100.1.181
[INFO]     obs: 10.100.1.181
[ OK ] Network infrastructure allocated
[INFO] Cleaning up ports from previous environments...
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-iam-identity-universe-main-dev
[INFO] IP Address: 10.100.1.181
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[OK]   No old observability services found
[INFO] Checking for remaining processes on IP 10.100.1.181...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup completed successfully
[INFO] Configuring IP aliases on network interface...
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING NETWORK IP ALIASES
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Interface: lo
[INFO] IP Count: 8

[INFO] Configuring: metrics → 10.100.1.182
[INFO]   Configuring IP alias: 10.100.1.182/32 on lo
[OK]     ✅ IP 10.100.1.182 configured successfully on lo
[OK]     ✅ IP 10.100.1.182 verified on network interface
[INFO] Configuring: alerts → 10.100.1.187
[INFO]   Configuring IP alias: 10.100.1.187/32 on lo
[OK]     ✅ IP 10.100.1.187 configured successfully on lo
[OK]     ✅ IP 10.100.1.187 verified on network interface
[INFO] Configuring: dashboards → 10.100.1.183
[INFO]   Configuring IP alias: 10.100.1.183/32 on lo
[OK]     ✅ IP 10.100.1.183 configured successfully on lo
[OK]     ✅ IP 10.100.1.183 verified on network interface
[INFO] Configuring: traces → 10.100.1.186
[INFO]   Configuring IP alias: 10.100.1.186/32 on lo
[OK]     ✅ IP 10.100.1.186 configured successfully on lo
[OK]     ✅ IP 10.100.1.186 verified on network interface
[INFO] Configuring: telemetry → 10.100.1.188
[INFO]   Configuring IP alias: 10.100.1.188/32 on lo
[OK]     ✅ IP 10.100.1.188 configured successfully on lo
[OK]     ✅ IP 10.100.1.188 verified on network interface
[INFO] Configuring: logstore → 10.100.1.184
[INFO]   Configuring IP alias: 10.100.1.184/32 on lo
[OK]     ✅ IP 10.100.1.184 configured successfully on lo
[OK]     ✅ IP 10.100.1.184 verified on network interface
[INFO] Configuring: proxy → 10.100.1.181
[INFO]   Configuring IP alias: 10.100.1.181/32 on lo
[OK]     ✅ IP 10.100.1.181 configured successfully on lo
[OK]     ✅ IP 10.100.1.181 verified on network interface
[INFO] Configuring: obs → 10.100.1.181
[INFO]   IP 10.100.1.181 already configured on network interface

[OK]   ═══════════════════════════════════════════════════════════════
[OK]   ✅ All IP aliases configured successfully
[OK]   ═══════════════════════════════════════════════════════════════
[INFO] Current IP configuration on lo:
      inet 127.0.0.1/8 scope host lo
      inet 10.100.60.2/32 scope global lo
      inet 10.100.1.182/32 scope global lo
      inet 10.100.1.187/32 scope global lo
      inet 10.100.1.183/32 scope global lo
      inet 10.100.1.186/32 scope global lo
      inet 10.100.1.188/32 scope global lo
      inet 10.100.1.184/32 scope global lo
      inet 10.100.1.181/32 scope global lo

[OK]   IP aliases configured on network interface
[INFO] Step 2/10: Creating DNS entries...
[INFO] Configuring DNS entries in /etc/hosts...
[INFO]   Added: metrics-iam-identity-universe-main-dev-prometheus.fastorder.com → 10.100.1.182
[INFO]   Added: alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com → 10.100.1.187
[INFO]   Added: dashboards-iam-identity-universe-main-dev-grafana.fastorder.com → 10.100.1.183
[INFO]   Added: traces-iam-identity-universe-main-dev-tempo.fastorder.com → 10.100.1.186
[INFO]   Added: telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com → 10.100.1.188
[INFO]   Added: logstore-iam-identity-universe-main-dev-clickhouse.fastorder.com → 10.100.1.184
[INFO]   Added: observe-iam-identity-universe-main-dev.fastorder.com → 10.100.1.181
[INFO] Adding observability integration aliases...
[INFO]   Added alias: metrics-iam-identity-universe-main-dev.fastorder.com → 10.100.1.182
[INFO]   Added alias: alerts-iam-identity-universe-main-dev.fastorder.com → 10.100.1.187
[INFO]   Added alias: dashboards-iam-identity-universe-main-dev.fastorder.com → 10.100.1.183
[INFO]   Added alias: traces-iam-identity-universe-main-dev.fastorder.com → 10.100.1.186
[INFO]   Added alias: telemetry-iam-identity-universe-main-dev.fastorder.com → 10.100.1.188
[INFO]   Added alias: logstore-iam-identity-universe-main-dev.fastorder.com → 10.100.1.184
[2026-01-02 04:23:42 UTC] USER=www-data EUID=0 PID=1378967 ACTION=fsop ARGS=sed -i /observe-iam-identity-universe-main-dev.fastorder.com/d /etc/hosts
[INFO]   Added alias: observe-iam-identity-universe-main-dev.fastorder.com → 10.100.1.181
[OK]   DNS entries created
[INFO] Step 3/10: Creating AWS Secrets Manager structure...
[INFO] Creating AWS Secrets Manager structure
[INFO]   Base path: fastorder/observability/iam-identity/universe/dev
[INFO]   Observability Cell: obs-iam-identity-universe-main-dev
[INFO]   Application Cell: iam-identity-universe-main-dev
[INFO]   Exists: fastorder/observability/iam-identity/universe/dev/metrics
[INFO]   Exists: fastorder/observability/iam-identity/universe/dev/dashboards
[INFO]   Exists: fastorder/observability/iam-identity/universe/dev/logstore
[INFO]   Exists: fastorder/observability/iam-identity/universe/dev/traces
[INFO]   Exists: fastorder/observability/iam-identity/universe/dev/telemetry
[INFO]   Exists: fastorder/observability/iam-identity/universe/dev/alerts
[INFO] Secrets structure created successfully
[OK]   Secrets structure created
[INFO] Step 4/10: Generating mTLS certificates...
[INFO] Generating mTLS certificates for observability cell
[INFO]   Observability Cell: obs-iam-identity-universe-main-dev
[INFO]   Components: prometheus,grafana,loki,tempo,otlp_collector,clickhouse,alertmanager
[INFO]   Creating certificate directory: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[2026-01-02 04:23:52 UTC] USER=www-data EUID=0 PID=1379018 ACTION=fsop ARGS=mkdir -p /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[2026-01-02 04:23:52 UTC] USER=www-data EUID=0 PID=1379027 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[INFO]   Generating CA certificate for obs-iam-identity-universe-main-dev
[2026-01-02 04:23:52 UTC] USER=www-data EUID=0 PID=1379036 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-key.pem 4096
[2026-01-02 04:23:54 UTC] USER=www-data EUID=0 PID=1379047 ACTION=fsop ARGS=openssl req -new -x509 -days 3650 -key /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-key.pem -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=obs-iam-identity-universe-main-dev-ca
[2026-01-02 04:23:54 UTC] USER=www-data EUID=0 PID=1379056 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-key.pem
[2026-01-02 04:23:54 UTC] USER=www-data EUID=0 PID=1379065 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem
[INFO]   CA certificate created: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem
[INFO]   Generating certificate for: prometheus
[2026-01-02 04:23:54 UTC] USER=www-data EUID=0 PID=1379074 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/prometheus-key.pem 2048
[2026-01-02 04:23:55 UTC] USER=www-data EUID=0 PID=1379083 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/prometheus-key.pem -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/prometheus-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=prometheus.obs-iam-identity-universe-main-dev
[2026-01-02 04:23:55 UTC] USER=www-data EUID=0 PID=1379092 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/prometheus-csr.pem -CA /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/prometheus-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = prometheus.obs-iam-identity-universe-main-dev
[2026-01-02 04:23:55 UTC] USER=www-data EUID=0 PID=1379101 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/prometheus-key.pem
[2026-01-02 04:23:55 UTC] USER=www-data EUID=0 PID=1379110 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/prometheus-cert.pem
[2026-01-02 04:23:55 UTC] USER=www-data EUID=0 PID=1379119 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/prometheus-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/prometheus-cert.pem
[INFO]   Generating certificate for: grafana
[2026-01-02 04:23:55 UTC] USER=www-data EUID=0 PID=1379128 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/grafana-key.pem 2048
[2026-01-02 04:23:55 UTC] USER=www-data EUID=0 PID=1379137 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/grafana-key.pem -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/grafana-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=grafana.obs-iam-identity-universe-main-dev
[2026-01-02 04:23:55 UTC] USER=www-data EUID=0 PID=1379146 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/grafana-csr.pem -CA /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/grafana-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = grafana.obs-iam-identity-universe-main-dev
[2026-01-02 04:23:55 UTC] USER=www-data EUID=0 PID=1379155 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/grafana-key.pem
[2026-01-02 04:23:55 UTC] USER=www-data EUID=0 PID=1379164 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/grafana-cert.pem
[2026-01-02 04:23:55 UTC] USER=www-data EUID=0 PID=1379173 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/grafana-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/grafana-cert.pem
[INFO]   Generating certificate for: loki
[2026-01-02 04:23:55 UTC] USER=www-data EUID=0 PID=1379182 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/loki-key.pem 2048
[2026-01-02 04:23:56 UTC] USER=www-data EUID=0 PID=1379191 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/loki-key.pem -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/loki-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=loki.obs-iam-identity-universe-main-dev
[2026-01-02 04:23:56 UTC] USER=www-data EUID=0 PID=1379200 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/loki-csr.pem -CA /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/loki-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = loki.obs-iam-identity-universe-main-dev
[2026-01-02 04:23:56 UTC] USER=www-data EUID=0 PID=1379209 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/loki-key.pem
[2026-01-02 04:23:56 UTC] USER=www-data EUID=0 PID=1379218 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/loki-cert.pem
[2026-01-02 04:23:56 UTC] USER=www-data EUID=0 PID=1379227 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/loki-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/loki-cert.pem
[INFO]   Generating certificate for: tempo
[2026-01-02 04:23:56 UTC] USER=www-data EUID=0 PID=1379236 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/tempo-key.pem 2048
[2026-01-02 04:23:56 UTC] USER=www-data EUID=0 PID=1379245 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/tempo-key.pem -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/tempo-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=tempo.obs-iam-identity-universe-main-dev
[2026-01-02 04:23:56 UTC] USER=www-data EUID=0 PID=1379254 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/tempo-csr.pem -CA /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/tempo-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = tempo.obs-iam-identity-universe-main-dev
[2026-01-02 04:23:56 UTC] USER=www-data EUID=0 PID=1379263 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/tempo-key.pem
[2026-01-02 04:23:56 UTC] USER=www-data EUID=0 PID=1379272 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/tempo-cert.pem
[2026-01-02 04:23:56 UTC] USER=www-data EUID=0 PID=1379281 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/tempo-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/tempo-cert.pem
[INFO]   Generating certificate for: otlp_collector
[2026-01-02 04:23:56 UTC] USER=www-data EUID=0 PID=1379290 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/otlp_collector-key.pem 2048
[2026-01-02 04:23:57 UTC] USER=www-data EUID=0 PID=1379299 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/otlp_collector-key.pem -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/otlp_collector-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=otlp_collector.obs-iam-identity-universe-main-dev
[2026-01-02 04:23:57 UTC] USER=www-data EUID=0 PID=1379308 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/otlp_collector-csr.pem -CA /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/otlp_collector-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = otlp_collector.obs-iam-identity-universe-main-dev
[2026-01-02 04:23:57 UTC] USER=www-data EUID=0 PID=1379317 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/otlp_collector-key.pem
[2026-01-02 04:23:57 UTC] USER=www-data EUID=0 PID=1379326 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/otlp_collector-cert.pem
[2026-01-02 04:23:57 UTC] USER=www-data EUID=0 PID=1379335 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/otlp_collector-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/otlp_collector-cert.pem
[INFO]   Generating certificate for: clickhouse
[2026-01-02 04:23:57 UTC] USER=www-data EUID=0 PID=1379344 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/clickhouse-key.pem 2048
[2026-01-02 04:23:57 UTC] USER=www-data EUID=0 PID=1379353 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/clickhouse-key.pem -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/clickhouse-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=clickhouse.obs-iam-identity-universe-main-dev
[2026-01-02 04:23:57 UTC] USER=www-data EUID=0 PID=1379363 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/clickhouse-csr.pem -CA /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/clickhouse-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = clickhouse.obs-iam-identity-universe-main-dev
[2026-01-02 04:23:57 UTC] USER=www-data EUID=0 PID=1379372 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/clickhouse-key.pem
[2026-01-02 04:23:57 UTC] USER=www-data EUID=0 PID=1379381 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/clickhouse-cert.pem
[2026-01-02 04:23:57 UTC] USER=www-data EUID=0 PID=1379390 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/clickhouse-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/clickhouse-cert.pem
[INFO]   Generating certificate for: alertmanager
[2026-01-02 04:23:58 UTC] USER=www-data EUID=0 PID=1379399 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/alertmanager-key.pem 2048
[2026-01-02 04:23:58 UTC] USER=www-data EUID=0 PID=1379408 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/alertmanager-key.pem -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/alertmanager-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=alertmanager.obs-iam-identity-universe-main-dev
[2026-01-02 04:23:58 UTC] USER=www-data EUID=0 PID=1379417 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/alertmanager-csr.pem -CA /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/alertmanager-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = alertmanager.obs-iam-identity-universe-main-dev
[2026-01-02 04:23:58 UTC] USER=www-data EUID=0 PID=1379426 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/alertmanager-key.pem
[2026-01-02 04:23:58 UTC] USER=www-data EUID=0 PID=1379435 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/alertmanager-cert.pem
[2026-01-02 04:23:58 UTC] USER=www-data EUID=0 PID=1379444 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/alertmanager-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/alertmanager-cert.pem
[INFO]   Generating PHP client certificate for metrics service...
[2026-01-02 04:23:58 UTC] USER=www-data EUID=0 PID=1379453 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/php-client-key.pem 2048
[2026-01-02 04:23:59 UTC] USER=www-data EUID=0 PID=1379462 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/php-client-key.pem -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/php-client-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Dashboard/CN=php-metrics-client.obs-iam-identity-universe-main-dev
[2026-01-02 04:23:59 UTC] USER=www-data EUID=0 PID=1379471 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/php-client-csr.pem -CA /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/php-client-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Dashboard, CN = php-metrics-client.obs-iam-identity-universe-main-dev
[2026-01-02 04:23:59 UTC] USER=www-data EUID=0 PID=1379480 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/php-client-key.pem
[2026-01-02 04:23:59 UTC] USER=www-data EUID=0 PID=1379489 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/php-client-cert.pem
[2026-01-02 04:23:59 UTC] USER=www-data EUID=0 PID=1379498 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/php-client-key.pem
[2026-01-02 04:23:59 UTC] USER=www-data EUID=0 PID=1379507 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/php-client-cert.pem
[2026-01-02 04:23:59 UTC] USER=www-data EUID=0 PID=1379516 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/php-client-csr.pem
[INFO]   PHP client certificate created: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/php-client-cert.pem
[INFO]   Generating Apache client certificate for mTLS reverse proxy...
[2026-01-02 04:23:59 UTC] USER=www-data EUID=0 PID=1379525 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/apache-client-key.pem 2048
[2026-01-02 04:24:00 UTC] USER=www-data EUID=0 PID=1379534 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/apache-client-key.pem -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/apache-client-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=ReverseProxy/CN=apache-proxy.obs-iam-identity-universe-main-dev
[2026-01-02 04:24:00 UTC] USER=www-data EUID=0 PID=1379543 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/apache-client-csr.pem -CA /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/apache-client-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = ReverseProxy, CN = apache-proxy.obs-iam-identity-universe-main-dev
[2026-01-02 04:24:00 UTC] USER=www-data EUID=0 PID=1379570 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/apache-client-key.pem
[2026-01-02 04:24:00 UTC] USER=www-data EUID=0 PID=1379579 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/apache-client-combined.pem
[2026-01-02 04:24:00 UTC] USER=www-data EUID=0 PID=1379588 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/apache-client-cert.pem
[2026-01-02 04:24:00 UTC] USER=www-data EUID=0 PID=1379597 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/apache-client-key.pem
[2026-01-02 04:24:00 UTC] USER=www-data EUID=0 PID=1379606 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/apache-client-cert.pem
[2026-01-02 04:24:00 UTC] USER=www-data EUID=0 PID=1379615 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/apache-client-combined.pem
[2026-01-02 04:24:00 UTC] USER=www-data EUID=0 PID=1379624 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/apache-client-csr.pem
[INFO]   Apache client certificate created: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/apache-client-cert.pem
[INFO]   Apache combined cert+key: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/apache-client-combined.pem
[INFO]   Storing mTLS certificates in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/iam-identity/universe/main/dev/mtls/php-client-UFSlUf",
    "Name": "fastorder/observability/iam-identity/universe/main/dev/mtls/php-client",
    "VersionId": "59eb9fc3-7c19-482c-9b74-4a0584dc9e81"
}
[INFO]   mTLS certificates stored in Secrets Manager: fastorder/observability/iam-identity/universe/main/dev/mtls/php-client
[INFO] mTLS certificates generated successfully
[INFO]   Certificate directory: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[INFO]   PHP client cert: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/php-client-cert.pem
[INFO]   PHP client key: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/php-client-key.pem
[INFO]   Apache client cert: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/apache-client-cert.pem
[INFO]   Apache combined (for SSLProxyMachineCertificateFile): /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/apache-client-combined.pem
[OK]   mTLS certificates generated
[INFO] Step 5/10: Deploying log storage backend...
[INFO]   Provider: clickhouse (selected)
[INFO]   Note: Deployed before telemetry (OtelCol depends on log storage)
[INFO]   FQDN: logstore-iam-identity-universe-main-dev-clickhouse.fastorder.com
[INFO]   IP: 10.100.1.184
[INFO] Deploying log backend: clickhouse...
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] LOG STORAGE BACKEND DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: clickhouse
[INFO] Observability Cell: obs-iam-identity-universe-main-dev
[INFO] FQDN: logstore-iam-identity-universe-main-dev-clickhouse.fastorder.com
[INFO] IP: 10.100.1.184
[INFO] S3 Bucket: fastorder-logs-universe-dev
[INFO] Retention: 90 days
[INFO] ═══════════════════════════════════════════════════════════════

[2026-01-02 04:24:04 UTC] USER=unknown EUID=33 PID=1379680 ACTION=fsop ARGS=chmod +x /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh
/bin/chmod: changing permissions of '/opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh': Operation not permitted
[INFO] Using provider: clickhouse
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=iam-identity, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-iam-identity-universe-main-dev...
[2026-01-02 04:24:04 UTC] USER=www-data EUID=0 PID=1379697 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-02 04:24:04 UTC] USER=www-data EUID=0 PID=1379706 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-02 04:24:04 UTC] USER=www-data EUID=0 PID=1379715 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-02 04:24:04 UTC] USER=www-data EUID=0 PID=1379724 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-iam-identity-universe-main-dev
[INFO] IP Address: 10.100.1.184
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[OK]   No old observability services found
[INFO] Checking for remaining processes on IP 10.100.1.184...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[INFO] Binding ClickHouse to allocated IP: 10.100.1.184
[INFO] Deploying ClickHouse for obs-iam-identity-universe-main-dev
[INFO]   FQDN: logstore-iam-identity-universe-main-dev-clickhouse.fastorder.com
[INFO]   Allocated IP: 10.100.1.184
[INFO]   VM IP: 10.100.1.184
[INFO]   Ports: HTTP=8123 TCP=9000 Interserver=9009
[INFO]   S3 Bucket: fastorder-logs-universe-dev (region=me-central-1)
[INFO]   Retention: 90 days
[INFO] Checking if ClickHouse is installed...
[OK]   ClickHouse already installed
[2026-01-02 04:24:04 UTC] USER=www-data EUID=0 PID=1379822 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-server-obs-iam-identity-universe-main-dev/config.d
[2026-01-02 04:24:04 UTC] USER=www-data EUID=0 PID=1379831 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-server-obs-iam-identity-universe-main-dev/users.d
[2026-01-02 04:24:04 UTC] USER=www-data EUID=0 PID=1379840 ACTION=fsop ARGS=mkdir -p /var/lib/clickhouse-obs-iam-identity-universe-main-dev
[2026-01-02 04:24:04 UTC] USER=www-data EUID=0 PID=1379849 ACTION=fsop ARGS=mkdir -p /var/log/clickhouse-server-obs-iam-identity-universe-main-dev
[2026-01-02 04:24:04 UTC] USER=www-data EUID=0 PID=1379858 ACTION=passthru ARGS=chmod 755 /etc/clickhouse-server-obs-iam-identity-universe-main-dev
[2026-01-02 04:24:04 UTC] USER=www-data EUID=0 PID=1379867 ACTION=passthru ARGS=chmod 700 /var/lib/clickhouse-obs-iam-identity-universe-main-dev
[2026-01-02 04:24:04 UTC] USER=www-data EUID=0 PID=1379876 ACTION=passthru ARGS=chmod 750 /var/log/clickhouse-server-obs-iam-identity-universe-main-dev
[INFO] Found existing logs_writer credentials in Secrets Manager - reusing to maintain sync
[INFO] Found existing metrics_reader credentials in Secrets Manager - reusing to maintain sync
[INFO] TLS configuration exported for clickhouse
[INFO]   Cert: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/clickhouse-cert.pem
[INFO]   Key:  /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/clickhouse-key.pem
[INFO]   CA:   /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem
[INFO] Configuring certificate permissions for clickhouse (user: clickhouse)
[INFO] Initializing certificate directory for obs-iam-identity-universe-main-dev...
[2026-01-02 04:24:08 UTC] USER=www-data EUID=0 PID=1379923 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-02 04:24:08 UTC] USER=www-data EUID=0 PID=1379932 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-02 04:24:08 UTC] USER=www-data EUID=0 PID=1379941 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-02 04:24:08 UTC] USER=www-data EUID=0 PID=1379950 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[INFO]   Setting file permissions...
[2026-01-02 04:24:08 UTC] USER=www-data EUID=0 PID=1379960 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/clickhouse-cert.pem
[2026-01-02 04:24:08 UTC] USER=www-data EUID=0 PID=1379969 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem
[2026-01-02 04:24:08 UTC] USER=www-data EUID=0 PID=1379978 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/clickhouse-key.pem
[INFO]   Setting file ownership...
[2026-01-02 04:24:08 UTC] USER=www-data EUID=0 PID=1379987 ACTION=passthru ARGS=chown root:clickhouse /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/clickhouse-key.pem
[2026-01-02 04:24:08 UTC] USER=www-data EUID=0 PID=1379996 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/clickhouse-cert.pem /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for clickhouse
[INFO] Creating ClickHouse configuration...
[2026-01-02 04:24:08 UTC] USER=www-data EUID=0 PID=1380042 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /etc/clickhouse-server-obs-iam-identity-universe-main-dev
[2026-01-02 04:24:09 UTC] USER=www-data EUID=0 PID=1380051 ACTION=passthru ARGS=bash -c chmod 640 /etc/clickhouse-server-obs-iam-identity-universe-main-dev/*.xml
[OK]   ClickHouse configuration created
[INFO] Creating logs table schema...
[2026-01-02 04:24:09 UTC] USER=www-data EUID=0 PID=1380069 ACTION=passthru ARGS=sed -i s/__RETENTION_DAYS__/90/g /etc/clickhouse-server-obs-iam-identity-universe-main-dev/logs_schema.sql
[2026-01-02 04:24:09 UTC] USER=www-data EUID=0 PID=1380078 ACTION=passthru ARGS=chmod 644 /etc/clickhouse-server-obs-iam-identity-universe-main-dev/logs_schema.sql
[OK]   Logs schema created
[INFO] Creating systemd service...
[2026-01-02 04:24:09 UTC] USER=www-data EUID=0 PID=1380096 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /var/lib/clickhouse-obs-iam-identity-universe-main-dev
[2026-01-02 04:24:09 UTC] USER=www-data EUID=0 PID=1380105 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /var/log/clickhouse-server-obs-iam-identity-universe-main-dev
[2026-01-02 04:24:09 UTC] USER=www-data EUID=0 PID=1380114 ACTION=passthru ARGS=chmod 700 /var/lib/clickhouse-obs-iam-identity-universe-main-dev
[OK]   Systemd service created
[INFO] Starting ClickHouse service...
[2026-01-02 04:24:09 UTC] USER=www-data EUID=0 PID=1380123 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:24:09 UTC] USER=www-data EUID=0 PID=1380169 ACTION=passthru ARGS=systemctl enable clickhouse-server-obs-iam-identity-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/clickhouse-server-obs-iam-identity-universe-main-dev.service -> /etc/systemd/system/clickhouse-server-obs-iam-identity-universe-main-dev.service.
[2026-01-02 04:24:10 UTC] USER=www-data EUID=0 PID=1380214 ACTION=passthru ARGS=systemctl start clickhouse-server-obs-iam-identity-universe-main-dev.service
[INFO] Waiting for ClickHouse to be ready...
[OK]   ClickHouse is ready
[INFO] Initializing database schema...
[OK]   Schema initialized
[INFO] Storing ClickHouse credentials in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/iam-identity/universe/main/dev/clickhouse/server/logs_writer-VNgHBp",
    "Name": "fastorder/observability/iam-identity/universe/main/dev/clickhouse/server/logs_writer",
    "VersionId": "4bcb1c14-0c97-4c24-b02b-75408a3a2225"
}
[OK]   logs_writer credentials stored and verified in Secrets Manager
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/iam-identity/universe/main/dev/clickhouse/server/metrics_reader-2jRlU2",
    "Name": "fastorder/observability/iam-identity/universe/main/dev/clickhouse/server/metrics_reader",
    "VersionId": "153260e4-6e94-4adc-9afb-f2c29dbe2875"
}
[OK]   metrics_reader credentials stored and verified in Secrets Manager
[INFO] Validating ClickHouse deployment...
[INFO] ClickHouse version: 25.10.1.3832
[INFO] Tables created: .inner_id.45bd1836-acd7-4c70-9cbf-df34dcdf5222
.inner_id.4e6b83a9-cdf6-42f8-8a00-2294bf03dd81
application_logs
error_logs_mv
iam_audit_event
metrics_all
otel_logs
request_logs_mv
security_access
[INFO] Test log inserted. Total logs: 1
[OK]   ✅ ClickHouse deployment validated

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ ClickHouse Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   FQDN: logstore-iam-identity-universe-main-dev-clickhouse.fastorder.com
[INFO]   IP: 10.100.1.184
[INFO]   HTTP Port: 8123
[INFO]   Native Port: 9000
[INFO]   Database: logs
[INFO]   Retention: 90 days
[INFO]   Storage: Tiered (Local → S3: fastorder-logs-universe-dev in me-central-1)
[INFO] 
[INFO] Credentials stored in AWS Secrets Manager:
[INFO]   Writers: fastorder/observability/iam-identity/universe/main/dev/clickhouse/server/logs_writer
[INFO]   Readers: fastorder/observability/iam-identity/universe/main/dev/clickhouse/server/metrics_reader (for PHP metrics service)
[INFO] 
[INFO] Example queries (using credentials from Secrets Manager):
[INFO]   # Write logs:
[INFO]   clickhouse-client --host logstore-iam-identity-universe-main-dev-clickhouse.fastorder.com --port 9000 --user logs_writer --password '***' --query 'SELECT 1'
[INFO] 
[INFO]   # Read metrics (PHP metrics service):
[INFO]   clickhouse-client --host logstore-iam-identity-universe-main-dev-clickhouse.fastorder.com --port 9000 --user metrics_reader --password '***' --query 'SELECT * FROM system.metrics'
[INFO] 
[INFO] HTTPS Setup (run on web-03/skeleton server):
[INFO]   # Set up HTTPS reverse proxy with Let's Encrypt:
[INFO]   OBS_CELL=obs-iam-identity-universe-main-dev BACKEND_IP=10.100.1.184 sudo bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/../https/setup-clickhouse-https.sh
[INFO] 
[INFO]   # Or add --setup-https flag when running this script
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Log Storage Backend Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: clickhouse
[INFO] FQDN: logstore-iam-identity-universe-main-dev-clickhouse.fastorder.com
[INFO] IP: 10.100.1.184
[INFO] Retention: 90 days
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering ClickHouse in monitoring dashboard...
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       ClickHouse
[INFO]   Identifier:        iam-identity-universe-main-dev-clickhouse
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.184
[INFO]   Port:              8443
[INFO]   FQDN:              logstore-iam-identity-universe-main-dev-clickhouse.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ INVALID REQUEST
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] Response: {"success":false,"error":"Invalid JSON: Control character error, possibly incorrectly encoded"}
[ERROR] 
[ERROR] Request payload:
  {
    "env_id": "iam-identity-universe-main-dev",
    "application": "ClickHouse",
    "identifier": "iam-identity-universe-main-dev-clickhouse",
    "identifier_parent": "cluster",
    "ip": "10.100.1.184",
    "port": 8443,
    "fqdn": "logstore-iam-identity-universe-main-dev-clickhouse.fastorder.com",
    "status": "running",
    "meta": {
      "role": "log_storage",
      "provider": "clickhouse",
      "version": "25.10
  1.3832",
      "http_port": 8123,
      "native_port": 9000,
      "https_port": 8443,
      "protocol": "https",
      "metrics_enabled": true,
      "metrics_port": 8123,
      "metrics_path": "/metrics",
      "health_endpoint": "https://logstore-iam-identity-universe-main-dev-clickhouse.fastorder.com/ping",
      "retention_days": 90,
      "s3_bucket": "fastorder-logs-universe-dev"
  }
  }
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[WARN] ⚠️  Failed to register ClickHouse (service is running)
[OK]   clickhouse deployed successfully
[OK]   Log storage backend deployed
[INFO] Step 6/10: Deploying telemetry collector...
[INFO]   Provider: otlp (backend implementation - internal)
[INFO]   Endpoint: telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com (stable, exposed to clients)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] TELEMETRY COLLECTOR DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: otlp
[INFO] Observability Cell: obs-iam-identity-universe-main-dev
[INFO] FQDN: telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com
[INFO] IP: 10.100.1.188
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: otlp
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Telemetry/provider/otlp.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=iam-identity, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-iam-identity-universe-main-dev...
[2026-01-02 04:24:23 UTC] USER=www-data EUID=0 PID=1381062 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-02 04:24:23 UTC] USER=www-data EUID=0 PID=1381071 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-02 04:24:23 UTC] USER=www-data EUID=0 PID=1381080 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-02 04:24:23 UTC] USER=www-data EUID=0 PID=1381089 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-iam-identity-universe-main-dev
[INFO] IP Address: 10.100.1.188
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-iam-identity-universe-main-dev.service belongs to current cell (skipping)
[INFO] Found 1 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.188...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[INFO] Binding to allocated IP: 10.100.1.188
[INFO] Deploying OpenTelemetry Collector for observability cell: obs-iam-identity-universe-main-dev
[INFO] FQDN:         telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com
[INFO] Allocated IP: 10.100.1.188
[INFO] VM IP:        10.100.1.188
[INFO] Ports:        gRPC=4317 HTTP=4318 Metrics=8888 Prom=8889
[OK]   User 'otelcol' already exists
[INFO] Checking if OpenTelemetry Collector is installed...
[OK]   OpenTelemetry Collector already installed at /usr/local/bin/otelcol-contrib
[INFO] Creating configuration/data directories...
[2026-01-02 04:24:24 UTC] USER=www-data EUID=0 PID=1381192 ACTION=passthru ARGS=mkdir -p /etc/otelcol/obs-iam-identity-universe-main-dev
[2026-01-02 04:24:24 UTC] USER=www-data EUID=0 PID=1381201 ACTION=passthru ARGS=mkdir -p /var/lib/otelcol/obs-iam-identity-universe-main-dev
[2026-01-02 04:24:24 UTC] USER=www-data EUID=0 PID=1381210 ACTION=passthru ARGS=chown -R otelcol:otelcol /etc/otelcol/obs-iam-identity-universe-main-dev /var/lib/otelcol/obs-iam-identity-universe-main-dev
[2026-01-02 04:24:24 UTC] USER=www-data EUID=0 PID=1381219 ACTION=passthru ARGS=chmod 0750 /etc/otelcol/obs-iam-identity-universe-main-dev
[2026-01-02 04:24:24 UTC] USER=www-data EUID=0 PID=1381228 ACTION=passthru ARGS=chmod 0750 /var/lib/otelcol/obs-iam-identity-universe-main-dev
[INFO] Retrieving ClickHouse credentials from Secrets Manager...
[OK]   Retrieved ClickHouse credentials from Secrets Manager
[INFO] Creating OpenTelemetry Collector configuration...
[INFO] ClickHouse exporter enabled: tcp://logstore-iam-identity-universe-main-dev-clickhouse.fastorder.com:9000
[2026-01-02 04:24:25 UTC] USER=www-data EUID=0 PID=1381267 ACTION=passthru ARGS=chown otelcol:otelcol /etc/otelcol/obs-iam-identity-universe-main-dev/config.yaml
[2026-01-02 04:24:26 UTC] USER=www-data EUID=0 PID=1381276 ACTION=passthru ARGS=chmod 0640 /etc/otelcol/obs-iam-identity-universe-main-dev/config.yaml
[OK]   Configuration created at /etc/otelcol/obs-iam-identity-universe-main-dev/config.yaml
[INFO] Setting up TLS certificate permissions...
[INFO] Configuring certificate permissions for otlp_collector (user: otelcol)
[INFO] Initializing certificate directory for obs-iam-identity-universe-main-dev...
[2026-01-02 04:24:26 UTC] USER=www-data EUID=0 PID=1381285 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-02 04:24:26 UTC] USER=www-data EUID=0 PID=1381294 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-02 04:24:26 UTC] USER=www-data EUID=0 PID=1381303 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-02 04:24:26 UTC] USER=www-data EUID=0 PID=1381312 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[INFO]   Setting file permissions...
[2026-01-02 04:24:26 UTC] USER=www-data EUID=0 PID=1381322 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/otlp_collector-cert.pem
[2026-01-02 04:24:26 UTC] USER=www-data EUID=0 PID=1381331 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem
[2026-01-02 04:24:26 UTC] USER=www-data EUID=0 PID=1381340 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/otlp_collector-key.pem
[INFO]   Setting file ownership...
[2026-01-02 04:24:26 UTC] USER=www-data EUID=0 PID=1381349 ACTION=passthru ARGS=chown root:otelcol /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/otlp_collector-key.pem
[2026-01-02 04:24:26 UTC] USER=www-data EUID=0 PID=1381358 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/otlp_collector-cert.pem /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for otlp_collector
[OK]   Certificate permissions configured
[INFO] Creating systemd service: otelcol-obs-iam-identity-universe-main-dev
[OK]   Systemd service created at /etc/systemd/system/otelcol-obs-iam-identity-universe-main-dev.service
[INFO] Adding /etc/hosts entry for telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com -> 10.100.1.188
[2026-01-02 04:24:26 UTC] USER=www-data EUID=0 PID=1381378 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com/10.100.1.188    telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com/ /etc/hosts
[OK]   Updated /etc/hosts entry to use VM_IP
[INFO] Storing OTLP configuration metadata in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/iam-identity/universe/main/dev/otlp/collector-n72sGS",
    "Name": "fastorder/observability/iam-identity/universe/main/dev/otlp/collector",
    "VersionId": "41a1d33d-c8f9-483b-a22b-379528cdda43"
}
[OK]   Configuration metadata stored/updated in AWS Secrets Manager: fastorder/observability/iam-identity/universe/main/dev/otlp/collector
[INFO] Enabling and starting OpenTelemetry Collector service...
[2026-01-02 04:24:27 UTC] USER=www-data EUID=0 PID=1381392 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:24:28 UTC] USER=www-data EUID=0 PID=1381437 ACTION=passthru ARGS=systemctl enable otelcol-obs-iam-identity-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/otelcol-obs-iam-identity-universe-main-dev.service -> /etc/systemd/system/otelcol-obs-iam-identity-universe-main-dev.service.
[2026-01-02 04:24:28 UTC] USER=www-data EUID=0 PID=1381482 ACTION=passthru ARGS=systemctl restart otelcol-obs-iam-identity-universe-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-01-02 04:24:31 UTC] USER=www-data EUID=0 PID=1381508 ACTION=passthru ARGS=systemctl is-active --quiet otelcol-obs-iam-identity-universe-main-dev.service
[OK]   ✅ OpenTelemetry Collector is running
[OK]   ✅ gRPC endpoint listening on port 4317
[OK]   ✅ HTTP endpoint listening on port 4318
[OK]   ✅ Prometheus metrics endpoint listening on port 8889
[INFO] Service logs (last 10 lines):
[2026-01-02 04:24:32 UTC] USER=www-data EUID=0 PID=1381523 ACTION=passthru ARGS=journalctl -u otelcol-obs-iam-identity-universe-main-dev.service -n 10 --no-pager
Jan 02 04:24:29 web-03 otelcol-obs-iam-identity-universe-main-dev[1381489]: 2026-01-02T04:24:29.703Z        info        internal/resourcedetection.go:125        began detecting resource information        {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics"}
Jan 02 04:24:29 web-03 otelcol-obs-iam-identity-universe-main-dev[1381489]: 2026-01-02T04:24:29.706Z        info        system/system.go:201        This attribute changed from int to string. Temporarily switch back to int using the feature gate.        {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics", "attribute": "host.cpu.family", "feature gate": "processor.resourcedetection.hostCPUModelAndFamilyAsString"}
Jan 02 04:24:29 web-03 otelcol-obs-iam-identity-universe-main-dev[1381489]: 2026-01-02T04:24:29.708Z        info        system/system.go:220        This attribute changed from int to string. Temporarily switch back to int using the feature gate.        {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics", "attribute": "host.cpu.model.id", "feature gate": "processor.resourcedetection.hostCPUModelAndFamilyAsString"}
Jan 02 04:24:29 web-03 otelcol-obs-iam-identity-universe-main-dev[1381489]: 2026-01-02T04:24:29.708Z        info        internal/resourcedetection.go:139        detected resource information        {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics", "resource": {"host.name":"web-03","os.type":"linux"}}
Jan 02 04:24:29 web-03 otelcol-obs-iam-identity-universe-main-dev[1381489]: 2026-01-02T04:24:29.708Z        info        prometheusreceiver@v0.91.0/metrics_receiver.go:231        Scrape job added        {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "jobName": "otel-collector"}
Jan 02 04:24:29 web-03 otelcol-obs-iam-identity-universe-main-dev[1381489]: 2026-01-02T04:24:29.708Z        info        prometheusreceiver@v0.91.0/metrics_receiver.go:240        Starting discovery manager        {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
Jan 02 04:24:29 web-03 otelcol-obs-iam-identity-universe-main-dev[1381489]: 2026-01-02T04:24:29.710Z        info        prometheusreceiver@v0.91.0/metrics_receiver.go:282        Starting scrape manager        {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
Jan 02 04:24:29 web-03 otelcol-obs-iam-identity-universe-main-dev[1381489]: 2026-01-02T04:24:29.753Z        info        otlpreceiver@v0.91.0/otlp.go:83        Starting GRPC server        {"kind": "receiver", "name": "otlp", "data_type": "logs", "endpoint": "10.100.1.188:4317"}
Jan 02 04:24:29 web-03 otelcol-obs-iam-identity-universe-main-dev[1381489]: 2026-01-02T04:24:29.753Z        info        otlpreceiver@v0.91.0/otlp.go:101        Starting HTTP server        {"kind": "receiver", "name": "otlp", "data_type": "logs", "endpoint": "10.100.1.188:4318"}
Jan 02 04:24:29 web-03 otelcol-obs-iam-identity-universe-main-dev[1381489]: 2026-01-02T04:24:29.755Z        info        service@v0.91.0/service.go:171        Everything is ready. Begin running and processing data.

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Telemetry Collector Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: otlp
[INFO] FQDN: telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com
[INFO] IP: 10.100.1.188
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering OpenTelemetry Collector in monitoring dashboard...
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       OpenTelemetry Collector
[INFO]   Identifier:        iam-identity-universe-main-dev-opentelemetry
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.188
[INFO]   Port:              4317
[INFO]   FQDN:              telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 99feae51-7464-44b8-9c06-ec4d2e3893a4
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✅ OpenTelemetry Collector registered in dashboard
[INFO] Setting up OpenTelemetry Collector metrics collection timer...
[2026-01-02 04:24:32 UTC] USER=www-data EUID=0 PID=1381580 ACTION=passthru ARGS=mv /tmp/otelcol-metrics-iam-identity-universe-main-dev.service /etc/systemd/system/
[2026-01-02 04:24:32 UTC] USER=www-data EUID=0 PID=1381589 ACTION=passthru ARGS=mv /tmp/otelcol-metrics-iam-identity-universe-main-dev.timer /etc/systemd/system/
[2026-01-02 04:24:32 UTC] USER=www-data EUID=0 PID=1381598 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:24:33 UTC] USER=www-data EUID=0 PID=1381643 ACTION=passthru ARGS=systemctl enable otelcol-metrics-iam-identity-universe-main-dev.timer
Created symlink /etc/systemd/system/timers.target.wants/otelcol-metrics-iam-identity-universe-main-dev.timer -> /etc/systemd/system/otelcol-metrics-iam-identity-universe-main-dev.timer.
[2026-01-02 04:24:33 UTC] USER=www-data EUID=0 PID=1381688 ACTION=passthru ARGS=systemctl start otelcol-metrics-iam-identity-universe-main-dev.timer
[OK]   ✅ Metrics collection timer installed and started
[OK]   Telemetry collector (otlp) deployed successfully

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Step 7/10: METRICS BACKEND DEPLOYMENT
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Provider: prometheus
[INFO]   OBS Cell: obs-iam-identity-universe-main-dev
[INFO]   FQDN: metrics-iam-identity-universe-main-dev-prometheus.fastorder.com
[INFO]   IP: 10.100.1.182
[INFO]   Script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/deploy-metrics.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📊 METRICS DEPLOYMENT WRAPPER STARTED
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Script: deploy-metrics.sh
[INFO] Timestamp: 2026-01-02 04:24:33 UTC
[INFO] Arguments: --provider prometheus --obs-cell obs-iam-identity-universe-main-dev --fqdn metrics-iam-identity-universe-main-dev-prometheus.fastorder.com --ip 10.100.1.182

[INFO] ═══════════════════════════════════════════════════════════════
[INFO] METRICS DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: prometheus
[INFO] Observability Cell: obs-iam-identity-universe-main-dev
[INFO] FQDN: metrics-iam-identity-universe-main-dev-prometheus.fastorder.com
[INFO] IP: 10.100.1.182
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: prometheus
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/provider/prometheus.sh

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/provider/prometheus.sh
[INFO]   OBS_CELL: obs-iam-identity-universe-main-dev
[INFO]   FQDN: metrics-iam-identity-universe-main-dev-prometheus.fastorder.com
[INFO]   IP: 10.100.1.182
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🚀 PROMETHEUS DEPLOYMENT STARTED
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Script: prometheus.sh
[INFO] Timestamp: 2026-01-02 04:24:33 UTC
[INFO] Arguments: --obs-cell obs-iam-identity-universe-main-dev --fqdn metrics-iam-identity-universe-main-dev-prometheus.fastorder.com --ip 10.100.1.182

[INFO] Parsed: SERVICE=iam-identity, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Step 1/12: Sourcing centralized libraries...
[INFO]   Library directory: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/lib
[INFO]   Sourcing port_allocator.sh...
[OK]     ✓ port_allocator.sh loaded
[INFO]   Sourcing cert_permissions.sh...
[OK]     ✓ cert_permissions.sh loaded
[INFO]   Sourcing port_cleanup.sh...
[OK]     ✓ port_cleanup.sh loaded
[OK]   Step 1/12: Libraries sourced successfully

[INFO] Step 2/12: Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-iam-identity-universe-main-dev...
[2026-01-02 04:24:33 UTC] USER=www-data EUID=0 PID=1381759 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-02 04:24:33 UTC] USER=www-data EUID=0 PID=1381795 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-02 04:24:34 UTC] USER=www-data EUID=0 PID=1381809 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-02 04:24:34 UTC] USER=www-data EUID=0 PID=1381818 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-iam-identity-universe-main-dev
[INFO] IP Address: 10.100.1.182
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-iam-identity-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-iam-identity-universe-main-dev.service belongs to current cell (skipping)
[INFO] Found 3 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.182...

[INFO] Scanning 15 ports...

[INFO]   ⚠️  NodeExporter: 10.100.1.182:9100 - OCCUPIED

[WARN] Found 1 occupied port(s) out of 15 total
[WARN] Will attempt to free occupied ports...

[OK]   Port 10.100.1.182:9100 occupied but service obs-iam-identity-universe-main-dev is running (OK - idempotent)

[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Port Cleanup Summary for obs-iam-identity-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Total ports checked:  15
[INFO] Already free:         14
[INFO] Occupied (cleaned):   1
[OK]   Successfully freed:   1

[OK]   ✅ All ports are now FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[OK]   Step 2/12: Port cleanup completed

[INFO] Step 3/12: Allocating ports...
[OK]   Step 3/12: Ports allocated

[INFO] Step 4/12: Setting up configuration...
[INFO]   Observability cell: obs-iam-identity-universe-main-dev
[INFO]   FQDN: metrics-iam-identity-universe-main-dev-prometheus.fastorder.com
[INFO]   IP: 10.100.1.182
[INFO]   Prometheus Port: 9090
[INFO] Step 5/12: Checking if Prometheus is installed...
[OK]   Prometheus already installed at /usr/local/bin/prometheus
[OK]   Step 5/12: Prometheus binary ready

[INFO] Step 5.1/12: Creating configuration directories early (required for Node Exporter config)...
[INFO]   Config: /etc/prometheus/obs-iam-identity-universe-main-dev
[INFO]   Data: /var/lib/prometheus/obs-iam-identity-universe-main-dev
[INFO]   Rules: /etc/prometheus/obs-iam-identity-universe-main-dev/rules
[2026-01-02 04:24:34 UTC] USER=www-data EUID=0 PID=1381971 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-iam-identity-universe-main-dev
[2026-01-02 04:24:34 UTC] USER=www-data EUID=0 PID=1381980 ACTION=passthru ARGS=mkdir -p /var/lib/prometheus/obs-iam-identity-universe-main-dev
[2026-01-02 04:24:34 UTC] USER=www-data EUID=0 PID=1381989 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-iam-identity-universe-main-dev/rules
[2026-01-02 04:24:34 UTC] USER=www-data EUID=0 PID=1381998 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-iam-identity-universe-main-dev/targets
[OK]   Step 5.1/12: Directories created early

[INFO] Step 6/12: Setting up Node Exporter...
[INFO] Checking if Node Exporter is installed...
[OK]   Node Exporter already installed at /usr/local/bin/node_exporter
[INFO] Creating Node Exporter TLS web config...
[INFO] Creating Node Exporter systemd service with TLS...
[2026-01-02 04:24:34 UTC] USER=www-data EUID=0 PID=1382025 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:24:35 UTC] USER=www-data EUID=0 PID=1382071 ACTION=passthru ARGS=systemctl enable node_exporter-obs-iam-identity-universe-main-dev.service
[2026-01-02 04:24:35 UTC] USER=www-data EUID=0 PID=1382116 ACTION=passthru ARGS=systemctl restart node_exporter-obs-iam-identity-universe-main-dev.service
[OK]   Step 6/12: Node Exporter ready

[INFO] Step 7/12: Creating configuration directories...
[INFO]   Config: /etc/prometheus/obs-iam-identity-universe-main-dev
[INFO]   Data: /var/lib/prometheus/obs-iam-identity-universe-main-dev
[INFO]   Rules: /etc/prometheus/obs-iam-identity-universe-main-dev/rules
[2026-01-02 04:24:35 UTC] USER=www-data EUID=0 PID=1382130 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-iam-identity-universe-main-dev
[2026-01-02 04:24:35 UTC] USER=www-data EUID=0 PID=1382139 ACTION=passthru ARGS=mkdir -p /var/lib/prometheus/obs-iam-identity-universe-main-dev
[2026-01-02 04:24:35 UTC] USER=www-data EUID=0 PID=1382148 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-iam-identity-universe-main-dev/rules
[OK]   Step 7/12: Directories created

[INFO] Step 8/12: Creating Prometheus configuration...
[INFO] Generated FQDNs:
[INFO]   Prometheus:   metrics-iam-identity-universe-main-dev-prometheus.fastorder.com
[INFO]   Alertmanager: alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com
[INFO]   Grafana:      dashboards-iam-identity-universe-main-dev-grafana.fastorder.com
[INFO]   Otelcol:      telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com
[OK]   Step 8/12: Configuration created at /etc/prometheus/obs-iam-identity-universe-main-dev/prometheus.yml

[INFO] Step 9/12: Creating TLS/HTTPS web config...
[OK]   Step 9/12: Web config created at /etc/prometheus/obs-iam-identity-universe-main-dev/web-config.yml
[INFO]   TLS cert: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/prometheus-cert.pem
[INFO]   TLS key: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/prometheus-key.pem
[INFO]   CA cert: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem

[INFO] Creating basic alerting rules...
[OK]   Alerting rules created
[2026-01-02 04:24:36 UTC] USER=www-data EUID=0 PID=1382184 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-iam-identity-universe-main-dev/targets
[2026-01-02 04:24:36 UTC] USER=www-data EUID=0 PID=1382193 ACTION=passthru ARGS=bash -c cat > '/etc/prometheus/obs-iam-identity-universe-main-dev/targets/.placeholder.yml' << 'EOF'
# Placeholder file to prevent file_sd_configs warning
# Application targets will be added here automatically
[]
EOF
[INFO] Step 10/12: Creating systemd service...
[INFO]   Service: prometheus-obs-iam-identity-universe-main-dev
[INFO] Binding to: 10.100.1.182:9090
[OK]   Step 10/12: Systemd service created at /etc/systemd/system/prometheus-obs-iam-identity-universe-main-dev.service

[INFO] Step 11/12: Configuring certificate permissions...
[INFO]   Looking for certificates in: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[OK]     ✓ All certificate files exist
[INFO] Configuring certificate permissions for prometheus (user: root)
[INFO] Initializing certificate directory for obs-iam-identity-universe-main-dev...
[2026-01-02 04:24:36 UTC] USER=www-data EUID=0 PID=1382212 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-02 04:24:36 UTC] USER=www-data EUID=0 PID=1382221 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-02 04:24:36 UTC] USER=www-data EUID=0 PID=1382230 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-02 04:24:36 UTC] USER=www-data EUID=0 PID=1382239 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[INFO]   Setting file permissions...
[2026-01-02 04:24:36 UTC] USER=www-data EUID=0 PID=1382249 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/prometheus-cert.pem
[2026-01-02 04:24:36 UTC] USER=www-data EUID=0 PID=1382258 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem
[2026-01-02 04:24:36 UTC] USER=www-data EUID=0 PID=1382267 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/prometheus-key.pem
[INFO]   Setting file ownership...
[2026-01-02 04:24:36 UTC] USER=www-data EUID=0 PID=1382276 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/prometheus-key.pem
[2026-01-02 04:24:36 UTC] USER=www-data EUID=0 PID=1382285 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/prometheus-cert.pem /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for prometheus
[OK]   Step 11/12: Certificate permissions configured

[INFO] Adding /etc/hosts entry for metrics-iam-identity-universe-main-dev-prometheus.fastorder.com -> 10.100.1.182
[2026-01-02 04:24:36 UTC] USER=www-data EUID=0 PID=1382296 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*metrics-iam-identity-universe-main-dev-prometheus.fastorder.com/10.100.1.182    metrics-iam-identity-universe-main-dev-prometheus.fastorder.com/ /etc/hosts
[OK]   Updated /etc/hosts entry to use VM_IP
[INFO] Validating Prometheus configuration...
Checking /etc/prometheus/obs-iam-identity-universe-main-dev/prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: /etc/prometheus/obs-iam-identity-universe-main-dev/prometheus.yml is valid prometheus config file syntax

Checking /etc/prometheus/obs-iam-identity-universe-main-dev/rules/basic_alerts.yml
  SUCCESS: 4 rules found

[OK]   ✅ Configuration is valid
[INFO] Storing Prometheus configuration in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/iam-identity/universe/main/dev/prometheus/server-QRqt7T",
    "Name": "fastorder/observability/iam-identity/universe/main/dev/prometheus/server",
    "VersionId": "625a6e8e-e0b8-4487-93a5-b1f3aec39a1c"
}
[OK]   Configuration stored in AWS Secrets Manager
[INFO] Step 12/12: Starting Prometheus service...
[INFO]   Reloading systemd daemon...
[2026-01-02 04:24:38 UTC] USER=www-data EUID=0 PID=1382318 ACTION=passthru ARGS=systemctl daemon-reload
[OK]     ✓ Systemd daemon reloaded
[INFO]   Enabling service...
[2026-01-02 04:24:38 UTC] USER=www-data EUID=0 PID=1382363 ACTION=passthru ARGS=systemctl enable prometheus-obs-iam-identity-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus-obs-iam-identity-universe-main-dev.service -> /etc/systemd/system/prometheus-obs-iam-identity-universe-main-dev.service.
[OK]     ✓ Service enabled
[INFO]   Starting service...
[2026-01-02 04:24:39 UTC] USER=www-data EUID=0 PID=1382408 ACTION=passthru ARGS=systemctl restart prometheus-obs-iam-identity-universe-main-dev.service
[OK]     ✓ Service start command issued

[INFO] Validating Prometheus deployment...
[2026-01-02 04:24:42 UTC] USER=www-data EUID=0 PID=1382437 ACTION=passthru ARGS=systemctl is-active --quiet prometheus-obs-iam-identity-universe-main-dev.service
[OK]   ✅ Prometheus is running
[OK]   ✅ Prometheus web interface listening on port 9090
[OK]   ✅ Prometheus health check passed (HTTPS)
[INFO] ═══════════════════════════════════════════════════════════════
[OK]   Prometheus Web UI: https://metrics-iam-identity-universe-main-dev-prometheus.fastorder.com:9090
[OK]   Targets: https://metrics-iam-identity-universe-main-dev-prometheus.fastorder.com:9090/targets
[OK]   Alerts: https://metrics-iam-identity-universe-main-dev-prometheus.fastorder.com:9090/alerts
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service logs (last 10 lines):
[2026-01-02 04:24:44 UTC] USER=www-data EUID=0 PID=1382452 ACTION=passthru ARGS=journalctl -u prometheus-obs-iam-identity-universe-main-dev.service -n 10 --no-pager
Jan 02 04:24:39 web-03 prometheus-obs-iam-identity-universe-main-dev[1382415]: ts=2026-01-02T04:24:39.392Z caller=head.go:682 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.194µs
Jan 02 04:24:39 web-03 prometheus-obs-iam-identity-universe-main-dev[1382415]: ts=2026-01-02T04:24:39.392Z caller=head.go:690 level=info component=tsdb msg="Replaying WAL, this may take a while"
Jan 02 04:24:39 web-03 prometheus-obs-iam-identity-universe-main-dev[1382415]: ts=2026-01-02T04:24:39.393Z caller=head.go:761 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
Jan 02 04:24:39 web-03 prometheus-obs-iam-identity-universe-main-dev[1382415]: ts=2026-01-02T04:24:39.393Z caller=head.go:798 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=43.537µs wal_replay_duration=546.423µs wbl_replay_duration=219ns total_replay_duration=618.021µs
Jan 02 04:24:39 web-03 prometheus-obs-iam-identity-universe-main-dev[1382415]: ts=2026-01-02T04:24:39.396Z caller=main.go:1045 level=info fs_type=EXT4_SUPER_MAGIC
Jan 02 04:24:39 web-03 prometheus-obs-iam-identity-universe-main-dev[1382415]: ts=2026-01-02T04:24:39.396Z caller=main.go:1048 level=info msg="TSDB started"
Jan 02 04:24:39 web-03 prometheus-obs-iam-identity-universe-main-dev[1382415]: ts=2026-01-02T04:24:39.396Z caller=main.go:1230 level=info msg="Loading configuration file" filename=/etc/prometheus/obs-iam-identity-universe-main-dev/prometheus.yml
Jan 02 04:24:39 web-03 prometheus-obs-iam-identity-universe-main-dev[1382415]: ts=2026-01-02T04:24:39.398Z caller=main.go:1267 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/obs-iam-identity-universe-main-dev/prometheus.yml totalDuration=2.399104ms db_storage=1.616µs remote_storage=1.596µs web_handler=573ns query_engine=1.653µs scrape=323.924µs scrape_sd=84.838µs notify=32.319µs notify_sd=17.939µs rules=1.334008ms tracing=8.057µs
Jan 02 04:24:39 web-03 prometheus-obs-iam-identity-universe-main-dev[1382415]: ts=2026-01-02T04:24:39.398Z caller=main.go:1009 level=info msg="Server is ready to receive web requests."
Jan 02 04:24:39 web-03 prometheus-obs-iam-identity-universe-main-dev[1382415]: ts=2026-01-02T04:24:39.398Z caller=manager.go:1012 level=info component="rule manager" msg="Starting rule manager..."
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Provider script completed with exit code: 0
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Metrics Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: prometheus
[INFO] FQDN: metrics-iam-identity-universe-main-dev-prometheus.fastorder.com
[INFO] IP: 10.100.1.182
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Step 7/10: METRICS DEPLOYMENT RESULT
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Exit code: 0
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ Metrics backend (prometheus) deployed successfully
[INFO] Step 8/10: Deploying traces backend...
[INFO]   Provider: tempo (selected)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] TRACES DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: tempo
[INFO] Observability Cell: obs-iam-identity-universe-main-dev
[INFO] FQDN: traces-iam-identity-universe-main-dev-tempo.fastorder.com
[INFO] IP: 10.100.1.186
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: tempo
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Traces/provider/tempo.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=iam-identity, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-iam-identity-universe-main-dev...
[2026-01-02 04:24:44 UTC] USER=www-data EUID=0 PID=1382472 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-02 04:24:44 UTC] USER=www-data EUID=0 PID=1382481 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-02 04:24:44 UTC] USER=www-data EUID=0 PID=1382490 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-02 04:24:44 UTC] USER=www-data EUID=0 PID=1382499 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-iam-identity-universe-main-dev
[INFO] IP Address: 10.100.1.186
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-iam-identity-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-iam-identity-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service prometheus-obs-iam-identity-universe-main-dev.service belongs to current cell (skipping)
[INFO] Found 4 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.186...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[INFO] Binding Tempo to allocated IP: 10.100.1.186
[INFO] Deploying Grafana Tempo for observability cell: obs-iam-identity-universe-main-dev
[INFO] FQDN: traces-iam-identity-universe-main-dev-tempo.fastorder.com
[INFO] Allocated IP: 10.100.1.186
[INFO] VM IP: 10.100.1.186
[INFO] Ports: HTTP=3200 gRPC=9095, OTLP gRPC=4317, OTLP HTTP=4318
[INFO] Checking if Grafana Tempo is installed...
[OK]   Grafana Tempo already installed at /usr/local/bin/tempo
[INFO] Preparing configuration and data directories...
[2026-01-02 04:24:44 UTC] USER=www-data EUID=0 PID=1382616 ACTION=passthru ARGS=mkdir -p /etc/tempo/obs-iam-identity-universe-main-dev
[2026-01-02 04:24:44 UTC] USER=www-data EUID=0 PID=1382625 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-iam-identity-universe-main-dev
[2026-01-02 04:24:44 UTC] USER=www-data EUID=0 PID=1382634 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-iam-identity-universe-main-dev/wal
[2026-01-02 04:24:44 UTC] USER=www-data EUID=0 PID=1382643 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-iam-identity-universe-main-dev/blocks
[2026-01-02 04:24:44 UTC] USER=www-data EUID=0 PID=1382652 ACTION=passthru ARGS=chown -R tempo:tempo /etc/tempo/obs-iam-identity-universe-main-dev /var/lib/tempo/obs-iam-identity-universe-main-dev
[2026-01-02 04:24:44 UTC] USER=www-data EUID=0 PID=1382661 ACTION=passthru ARGS=chmod 750 /etc/tempo/obs-iam-identity-universe-main-dev /var/lib/tempo/obs-iam-identity-universe-main-dev
[INFO] Creating Grafana Tempo configuration...
[INFO] TLS configuration exported for tempo
[INFO]   Cert: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/tempo-cert.pem
[INFO]   Key:  /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/tempo-key.pem
[INFO]   CA:   /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem
[INFO] Setting up certificate permissions for Tempo...
[INFO] Configuring certificate permissions for tempo (user: tempo)
[INFO] Initializing certificate directory for obs-iam-identity-universe-main-dev...
[2026-01-02 04:24:44 UTC] USER=www-data EUID=0 PID=1382676 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-02 04:24:44 UTC] USER=www-data EUID=0 PID=1382685 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-02 04:24:45 UTC] USER=www-data EUID=0 PID=1382694 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-02 04:24:45 UTC] USER=www-data EUID=0 PID=1382703 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[INFO]   Setting file permissions...
[2026-01-02 04:24:45 UTC] USER=www-data EUID=0 PID=1382713 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/tempo-cert.pem
[2026-01-02 04:24:45 UTC] USER=www-data EUID=0 PID=1382722 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem
[2026-01-02 04:24:45 UTC] USER=www-data EUID=0 PID=1382731 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/tempo-key.pem
[INFO]   Setting file ownership...
[2026-01-02 04:24:45 UTC] USER=www-data EUID=0 PID=1382740 ACTION=passthru ARGS=chown root:tempo /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/tempo-key.pem
[2026-01-02 04:24:45 UTC] USER=www-data EUID=0 PID=1382749 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/tempo-cert.pem /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for tempo
[2026-01-02 04:24:45 UTC] USER=www-data EUID=0 PID=1382768 ACTION=passthru ARGS=chown tempo:tempo /etc/tempo/obs-iam-identity-universe-main-dev/config.yaml
[2026-01-02 04:24:45 UTC] USER=www-data EUID=0 PID=1382777 ACTION=passthru ARGS=chmod 640 /etc/tempo/obs-iam-identity-universe-main-dev/config.yaml
[OK]   Configuration created at /etc/tempo/obs-iam-identity-universe-main-dev/config.yaml
[INFO] Creating systemd service: tempo-obs-iam-identity-universe-main-dev
[OK]   Systemd service created
[INFO] Adding /etc/hosts entry for traces-iam-identity-universe-main-dev-tempo.fastorder.com -> 10.100.1.186
[2026-01-02 04:24:45 UTC] USER=www-data EUID=0 PID=1382796 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*traces-iam-identity-universe-main-dev-tempo.fastorder.com/10.100.1.186    traces-iam-identity-universe-main-dev-tempo.fastorder.com/ /etc/hosts
[OK]   Updated /etc/hosts entry to use VM_IP
[INFO] Storing Tempo configuration in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/iam-identity/universe/main/dev/tempo/server-rJ5OOT",
    "Name": "fastorder/observability/iam-identity/universe/main/dev/tempo/server",
    "VersionId": "696d8fe2-21b4-472c-a279-f85c3d1e3cb3"
}
[OK]   Tempo configuration stored/updated in AWS Secrets Manager: fastorder/observability/iam-identity/universe/main/dev/tempo/server
[WARN] Port cleanup library not found, skipping automatic cleanup
[INFO] Adding iptables redirect for Tempo internal communication (optional)...
[2026-01-02 04:24:47 UTC] USER=www-data EUID=0 PID=1382812 ACTION=passthru ARGS=iptables -t nat -A OUTPUT -p tcp -d 127.0.0.1 --dport 9095 -j DNAT --to-destination 10.100.1.186:9095
ERROR: passthru not allowed: iptables
[WARN] Could not add iptables redirect (iptables not allowed in wrapper)
[WARN] Tempo will still work - clients should connect to 10.100.1.186:9095 directly
[INFO] Enabling and starting Grafana Tempo service...
[2026-01-02 04:24:47 UTC] USER=www-data EUID=0 PID=1382820 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:24:47 UTC] USER=www-data EUID=0 PID=1382865 ACTION=passthru ARGS=systemctl enable tempo-obs-iam-identity-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/tempo-obs-iam-identity-universe-main-dev.service -> /etc/systemd/system/tempo-obs-iam-identity-universe-main-dev.service.
[2026-01-02 04:24:48 UTC] USER=www-data EUID=0 PID=1382910 ACTION=passthru ARGS=systemctl restart tempo-obs-iam-identity-universe-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-01-02 04:24:51 UTC] USER=www-data EUID=0 PID=1382932 ACTION=passthru ARGS=systemctl is-active --quiet tempo-obs-iam-identity-universe-main-dev.service
[OK]   ✅ Grafana Tempo is running
[OK]   ✅ HTTP endpoint listening on port 3200
[OK]   ✅ OTLP gRPC endpoint listening on port 4317
[OK]   ✅ OTLP HTTP endpoint listening on port 4318
[INFO] Service logs (last 10 lines):
[2026-01-02 04:24:51 UTC] USER=www-data EUID=0 PID=1382947 ACTION=passthru ARGS=journalctl -u tempo-obs-iam-identity-universe-main-dev.service -n 10 --no-pager
Jan 02 04:24:48 web-03 tempo-obs-iam-identity-universe-main-dev[1382917]: level=info ts=2026-01-02T04:24:48.433158798Z caller=basic_lifecycler.go:297 msg="instance not found in the ring" instance=web-03 ring=metrics-generator
Jan 02 04:24:48 web-03 tempo-obs-iam-identity-universe-main-dev[1382917]: level=info ts=2026-01-02T04:24:48.433216004Z caller=basic_lifecycler.go:297 msg="instance not found in the ring" instance=web-03 ring=compactor
Jan 02 04:24:48 web-03 tempo-obs-iam-identity-universe-main-dev[1382917]: level=info ts=2026-01-02T04:24:48.433168122Z caller=ring.go:297 msg="ring doesn't exist in KV store yet"
Jan 02 04:24:48 web-03 tempo-obs-iam-identity-universe-main-dev[1382917]: level=info ts=2026-01-02T04:24:48.43353667Z caller=module_service.go:82 msg=starting module=distributor
Jan 02 04:24:48 web-03 tempo-obs-iam-identity-universe-main-dev[1382917]: level=info ts=2026-01-02T04:24:48.434005923Z caller=compactor.go:127 msg="waiting until compactor is ACTIVE in the ring"
Jan 02 04:24:48 web-03 tempo-obs-iam-identity-universe-main-dev[1382917]: level=info ts=2026-01-02T04:24:48.434346904Z caller=worker.go:246 msg="total worker concurrency updated" totalConcurrency=20
Jan 02 04:24:48 web-03 tempo-obs-iam-identity-universe-main-dev[1382917]: ts=2026-01-02T04:24:48Z level=info msg="Starting GRPC server" component=tempo endpoint=10.100.1.186:4317
Jan 02 04:24:48 web-03 tempo-obs-iam-identity-universe-main-dev[1382917]: ts=2026-01-02T04:24:48Z level=info msg="Starting HTTP server" component=tempo endpoint=10.100.1.186:4318
Jan 02 04:24:48 web-03 tempo-obs-iam-identity-universe-main-dev[1382917]: level=info ts=2026-01-02T04:24:48.55051463Z caller=compactor.go:133 msg="compactor is ACTIVE in the ring"
Jan 02 04:24:48 web-03 tempo-obs-iam-identity-universe-main-dev[1382917]: level=info ts=2026-01-02T04:24:48.550929766Z caller=compactor.go:142 msg="waiting until compactor ring topology is stable" min_waiting=1m0s max_waiting=5m0s

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Traces Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: tempo
[INFO] FQDN: traces-iam-identity-universe-main-dev-tempo.fastorder.com
[INFO] IP: 10.100.1.186
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering Tempo in monitoring dashboard...
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Tempo
[INFO]   Identifier:        iam-identity-universe-main-dev-tempo
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.186
[INFO]   Port:              3200
[INFO]   FQDN:              traces-iam-identity-universe-main-dev-tempo.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: ca4299ea-758f-4b03-afaa-ef64981f5219
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✅ Tempo registered in dashboard
[OK]   Traces backend (tempo) deployed successfully
[INFO] Step 9/10: Deploying dashboards...
[INFO]   Provider: grafana (selected)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] DASHBOARDS DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: grafana
[INFO] Observability Cell: obs-iam-identity-universe-main-dev
[INFO] FQDN: dashboards-iam-identity-universe-main-dev-grafana.fastorder.com
[INFO] IP: 10.100.1.183
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: grafana
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Dashboards/provider/grafana.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=iam-identity, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Binding to allocated IP: 10.100.1.183
[INFO] Deploying Grafana for observability cell: obs-iam-identity-universe-main-dev
[INFO] FQDN: dashboards-iam-identity-universe-main-dev-grafana.fastorder.com
[INFO] Allocated IP: 10.100.1.183
[INFO] VM IP: 10.100.1.183
[INFO] HTTP Port: 3000
[INFO] Checking if Grafana is installed...
[OK]   Grafana already installed
[INFO] Installing Grafana plugins...
[INFO] Installing ClickHouse datasource plugin...
[WARN] Failed to install ClickHouse plugin (may need internet access)
[INFO] Validating TLS certificate and key...
[INFO] Setting certificate permissions...
[OK]   TLS cert/key found and permissions set
[INFO] Creating configuration and data directories...
[2026-01-02 04:24:52 UTC] USER=www-data EUID=0 PID=1383016 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-iam-identity-universe-main-dev
[2026-01-02 04:24:52 UTC] USER=www-data EUID=0 PID=1383025 ACTION=passthru ARGS=mkdir -p /var/lib/grafana/obs-iam-identity-universe-main-dev
[2026-01-02 04:24:52 UTC] USER=www-data EUID=0 PID=1383034 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-iam-identity-universe-main-dev/provisioning/datasources
[2026-01-02 04:24:52 UTC] USER=www-data EUID=0 PID=1383043 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-iam-identity-universe-main-dev/provisioning/dashboards
[2026-01-02 04:24:52 UTC] USER=www-data EUID=0 PID=1383052 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-iam-identity-universe-main-dev/provisioning/notifiers
[INFO] Creating Grafana configuration at /etc/grafana/obs-iam-identity-universe-main-dev/grafana.ini...
[OK]   Configuration created
[INFO] Creating Prometheus datasource provisioning...
[OK]   Prometheus datasource provisioned
[INFO] Creating Tempo datasource provisioning...
[OK]   Tempo datasource provisioned
[INFO] Creating Loki datasource provisioning...
[OK]   Loki datasource provisioned
[INFO] Creating ClickHouse datasource provisioning...
[OK]   Retrieved ClickHouse credentials from Secrets Manager
[OK]   ClickHouse datasource provisioned
[INFO] Creating systemd service: grafana-obs-iam-identity-universe-main-dev
[OK]   Systemd service created
[2026-01-02 04:24:54 UTC] USER=www-data EUID=0 PID=1383138 ACTION=passthru ARGS=chown -R grafana:grafana /etc/grafana/obs-iam-identity-universe-main-dev
[2026-01-02 04:24:54 UTC] USER=www-data EUID=0 PID=1383147 ACTION=passthru ARGS=chown -R grafana:grafana /var/lib/grafana/obs-iam-identity-universe-main-dev
[2026-01-02 04:24:54 UTC] USER=www-data EUID=0 PID=1383156 ACTION=passthru ARGS=chmod 750 /etc/grafana/obs-iam-identity-universe-main-dev /var/lib/grafana/obs-iam-identity-universe-main-dev
[INFO] Adding /etc/hosts entry for dashboards-iam-identity-universe-main-dev-grafana.fastorder.com -> 10.100.1.183
[WARN] /etc/hosts entry already exists
[INFO] Storing Grafana credentials in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/iam-identity/universe/main/dev/grafana/admin-eCnTPz",
    "Name": "fastorder/observability/iam-identity/universe/main/dev/grafana/admin",
    "VersionId": "45df3e5f-a812-4f94-9253-f433e4c58bc0"
}
[OK]   Credentials stored in AWS Secrets Manager: fastorder/observability/iam-identity/universe/main/dev/grafana/admin
[INFO] Enabling and starting Grafana service...
[2026-01-02 04:24:55 UTC] USER=www-data EUID=0 PID=1383171 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:24:56 UTC] USER=www-data EUID=0 PID=1383216 ACTION=passthru ARGS=systemctl enable grafana-obs-iam-identity-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/grafana-obs-iam-identity-universe-main-dev.service -> /etc/systemd/system/grafana-obs-iam-identity-universe-main-dev.service.
[2026-01-02 04:24:57 UTC] USER=www-data EUID=0 PID=1383261 ACTION=passthru ARGS=systemctl restart grafana-obs-iam-identity-universe-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-01-02 04:25:02 UTC] USER=www-data EUID=0 PID=1383327 ACTION=passthru ARGS=systemctl is-active --quiet grafana-obs-iam-identity-universe-main-dev.service
[OK]   ✅ Grafana is running
[WARN] ⚠️  Grafana web interface not yet listening on port 3000
[INFO] ═══════════════════════════════════════════════════════════════
[OK]   Grafana Dashboard URL: https://dashboards-iam-identity-universe-main-dev-grafana.fastorder.com:3000
[OK]   Username: admin
[OK]   Password is stored in AWS Secrets Manager at: fastorder/observability/iam-identity/universe/main/dev/grafana/admin
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service logs (last 10 lines):
[2026-01-02 04:25:02 UTC] USER=www-data EUID=0 PID=1383339 ACTION=passthru ARGS=journalctl -u grafana-obs-iam-identity-universe-main-dev.service -n 10 --no-pager
Jan 02 04:25:02 web-03 grafana-obs-iam-identity-universe-main-dev[1383270]: logger=migrator t=2026-01-02T04:25:02.093377608Z level=info msg="Migration successfully executed" id="create index IDX_correlation_source_uid - v2" duration=1.663005ms
Jan 02 04:25:02 web-03 grafana-obs-iam-identity-universe-main-dev[1383270]: logger=migrator t=2026-01-02T04:25:02.102474467Z level=info msg="Executing migration" id="create index IDX_correlation_org_id - v2"
Jan 02 04:25:02 web-03 grafana-obs-iam-identity-universe-main-dev[1383270]: logger=migrator t=2026-01-02T04:25:02.104226075Z level=info msg="Migration successfully executed" id="create index IDX_correlation_org_id - v2" duration=1.755947ms
Jan 02 04:25:02 web-03 grafana-obs-iam-identity-universe-main-dev[1383270]: logger=migrator t=2026-01-02T04:25:02.111004385Z level=info msg="Executing migration" id="copy correlation v1 to v2"
Jan 02 04:25:02 web-03 grafana-obs-iam-identity-universe-main-dev[1383270]: logger=migrator t=2026-01-02T04:25:02.111539016Z level=info msg="Migration successfully executed" id="copy correlation v1 to v2" duration=538.236µs
Jan 02 04:25:02 web-03 grafana-obs-iam-identity-universe-main-dev[1383270]: logger=migrator t=2026-01-02T04:25:02.117528445Z level=info msg="Executing migration" id="drop correlation_tmp_qwerty"
Jan 02 04:25:02 web-03 grafana-obs-iam-identity-universe-main-dev[1383270]: logger=migrator t=2026-01-02T04:25:02.119669567Z level=info msg="Migration successfully executed" id="drop correlation_tmp_qwerty" duration=2.136597ms
Jan 02 04:25:02 web-03 grafana-obs-iam-identity-universe-main-dev[1383270]: logger=migrator t=2026-01-02T04:25:02.125912353Z level=info msg="Executing migration" id="add provisioning column"
Jan 02 04:25:02 web-03 grafana-obs-iam-identity-universe-main-dev[1383270]: logger=migrator t=2026-01-02T04:25:02.136626008Z level=info msg="Migration successfully executed" id="add provisioning column" duration=10.706654ms
Jan 02 04:25:02 web-03 grafana-obs-iam-identity-universe-main-dev[1383270]: logger=migrator t=2026-01-02T04:25:02.143624678Z level=info msg="Executing migration" id="add type column"

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Dashboards Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: grafana
[INFO] FQDN: dashboards-iam-identity-universe-main-dev-grafana.fastorder.com
[INFO] IP: 10.100.1.183
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering Grafana in monitoring dashboard...
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Grafana
[INFO]   Identifier:        iam-identity-universe-main-dev-grafana
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.183
[INFO]   Port:              3000
[INFO]   FQDN:              dashboards-iam-identity-universe-main-dev-grafana.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 6dc2f7d7-f9d7-49b6-9602-baa6d5321469
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✅ Grafana registered in dashboard
[OK]   Dashboards (grafana) deployed successfully
[INFO] Step 10/10: Deploying alerting...
[INFO]   Provider: alertmanager (selected)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] ALERTING DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: alertmanager
[INFO] Observability Cell: obs-iam-identity-universe-main-dev
[INFO] FQDN: alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com
[INFO] IP: 10.100.1.187
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Ports: Web=9093 Cluster=9094 (bound to IP: 10.100.1.187)

[INFO] Using provider: alertmanager
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Alerting/provider/alertmanager.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=iam-identity, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Binding to allocated IP: 10.100.1.187
[INFO] Deploying Alertmanager for observability cell: obs-iam-identity-universe-main-dev
[INFO] FQDN: alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com
[INFO] Allocated IP: 10.100.1.187
[INFO] VM IP: 10.100.1.187
[INFO] Ports: Web=9093 Cluster=9094
[INFO] Checking if Alertmanager is installed...
[OK]   Alertmanager already installed at /usr/local/bin/alertmanager
[INFO] Validating TLS certificate and key...
[OK]   TLS cert/key found in /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev
[INFO] Creating configuration and data directories...
[2026-01-02 04:25:03 UTC] USER=www-data EUID=0 PID=1383410 ACTION=passthru ARGS=mkdir -p /etc/alertmanager/obs-iam-identity-universe-main-dev
[2026-01-02 04:25:03 UTC] USER=www-data EUID=0 PID=1383419 ACTION=passthru ARGS=mkdir -p /var/lib/alertmanager/obs-iam-identity-universe-main-dev
[2026-01-02 04:25:03 UTC] USER=www-data EUID=0 PID=1383428 ACTION=passthru ARGS=mkdir -p /etc/alertmanager/obs-iam-identity-universe-main-dev/templates
[INFO] Creating Alertmanager configuration...
[OK]   Alertmanager configuration created at /etc/alertmanager/obs-iam-identity-universe-main-dev/alertmanager.yml
[INFO] Creating notification templates...
[OK]   Notification templates created
[INFO] Creating Alertmanager web TLS configuration with mTLS...
[OK]   Web mTLS configuration created at /etc/alertmanager/obs-iam-identity-universe-main-dev/web-config.yml
[INFO] Validating Alertmanager configuration...
[2026-01-02 04:25:03 UTC] USER=www-data EUID=0 PID=1383464 ACTION=passthru ARGS=chmod 755 /etc/alertmanager/obs-iam-identity-universe-main-dev
[2026-01-02 04:25:03 UTC] USER=www-data EUID=0 PID=1383473 ACTION=passthru ARGS=chmod 644 /etc/alertmanager/obs-iam-identity-universe-main-dev/alertmanager.yml
Checking '/etc/alertmanager/obs-iam-identity-universe-main-dev/alertmanager.yml'  SUCCESS
Found:
 - global config
 - route
 - 6 inhibit rules
 - 5 receivers
 - 1 templates
  SUCCESS

[OK]   ✅ Configuration is valid
[INFO] Creating systemd service: alertmanager-obs-iam-identity-universe-main-dev
[OK]   Systemd service created
[2026-01-02 04:25:03 UTC] USER=www-data EUID=0 PID=1383499 ACTION=passthru ARGS=chown alertmanager:alertmanager /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/alertmanager-key.pem
[2026-01-02 04:25:03 UTC] USER=www-data EUID=0 PID=1383508 ACTION=passthru ARGS=chown alertmanager:alertmanager /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/alertmanager-cert.pem
[2026-01-02 04:25:03 UTC] USER=www-data EUID=0 PID=1383517 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-iam-identity-universe-main-dev/ca-cert.pem
[2026-01-02 04:25:03 UTC] USER=www-data EUID=0 PID=1383526 ACTION=passthru ARGS=chown -R alertmanager:alertmanager /etc/alertmanager/obs-iam-identity-universe-main-dev
[2026-01-02 04:25:03 UTC] USER=www-data EUID=0 PID=1383535 ACTION=passthru ARGS=chown -R alertmanager:alertmanager /var/lib/alertmanager/obs-iam-identity-universe-main-dev
[2026-01-02 04:25:03 UTC] USER=www-data EUID=0 PID=1383544 ACTION=passthru ARGS=chmod 750 /etc/alertmanager/obs-iam-identity-universe-main-dev /var/lib/alertmanager/obs-iam-identity-universe-main-dev
[INFO] Adding /etc/hosts entry for alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com -> 10.100.1.187
[WARN] /etc/hosts entry already exists
[INFO] Storing Alertmanager configuration in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/iam-identity/universe/main/dev/alertmanager/server-LDQLp5",
    "Name": "fastorder/observability/iam-identity/universe/main/dev/alertmanager/server",
    "VersionId": "b55f3513-6a7c-498e-a003-f51fca8a95d5"
}
[OK]   Configuration stored in AWS Secrets Manager: fastorder/observability/iam-identity/universe/main/dev/alertmanager/server
[INFO] Enabling and starting Alertmanager service...
[2026-01-02 04:25:05 UTC] USER=www-data EUID=0 PID=1383563 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:25:06 UTC] USER=www-data EUID=0 PID=1383618 ACTION=passthru ARGS=systemctl enable alertmanager-obs-iam-identity-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/alertmanager-obs-iam-identity-universe-main-dev.service -> /etc/systemd/system/alertmanager-obs-iam-identity-universe-main-dev.service.
[2026-01-02 04:25:06 UTC] USER=www-data EUID=0 PID=1383665 ACTION=passthru ARGS=systemctl restart alertmanager-obs-iam-identity-universe-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-01-02 04:25:09 UTC] USER=www-data EUID=0 PID=1383685 ACTION=passthru ARGS=systemctl is-active --quiet alertmanager-obs-iam-identity-universe-main-dev.service
[OK]   ✅ Alertmanager is running
[OK]   ✅ Alertmanager HTTPS web interface listening on port 9093
[OK]   ✅ Alertmanager cluster port listening on port 9094
[WARN] ⚠️  Alertmanager health check not responding yet (HTTPS)
[INFO] ═══════════════════════════════════════════════════════════════
[OK]   Alertmanager Web UI: https://alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com:9093
[OK]   API Endpoint:        https://alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com:9093/api/v2
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service logs (last 10 lines):
[2026-01-02 04:25:09 UTC] USER=www-data EUID=0 PID=1383699 ACTION=passthru ARGS=journalctl -u alertmanager-obs-iam-identity-universe-main-dev.service -n 10 --no-pager
Jan 02 04:25:06 web-03 systemd[1]: Started Alertmanager - obs-iam-identity-universe-main-dev.
Jan 02 04:25:06 web-03 alertmanager-obs-iam-identity-universe-main-dev[1383672]: ts=2026-01-02T04:25:06.947Z caller=main.go:245 level=info msg="Starting Alertmanager" version="(version=0.26.0, branch=HEAD, revision=d7b4f0c7322e7151d6e3b1e31cbc15361e295d8d)"
Jan 02 04:25:06 web-03 alertmanager-obs-iam-identity-universe-main-dev[1383672]: ts=2026-01-02T04:25:06.947Z caller=main.go:246 level=info build_context="(go=go1.20.7, platform=linux/amd64, user=root@df8d7debeef4, date=20230824-11:11:58, tags=netgo)"
Jan 02 04:25:06 web-03 alertmanager-obs-iam-identity-universe-main-dev[1383672]: ts=2026-01-02T04:25:06.949Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s
Jan 02 04:25:07 web-03 alertmanager-obs-iam-identity-universe-main-dev[1383672]: ts=2026-01-02T04:25:07.023Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/obs-iam-identity-universe-main-dev/alertmanager.yml
Jan 02 04:25:07 web-03 alertmanager-obs-iam-identity-universe-main-dev[1383672]: ts=2026-01-02T04:25:07.027Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/obs-iam-identity-universe-main-dev/alertmanager.yml
Jan 02 04:25:07 web-03 alertmanager-obs-iam-identity-universe-main-dev[1383672]: ts=2026-01-02T04:25:07.034Z caller=tls_config.go:274 level=info msg="Listening on" address=10.100.1.187:9093
Jan 02 04:25:07 web-03 alertmanager-obs-iam-identity-universe-main-dev[1383672]: ts=2026-01-02T04:25:07.035Z caller=tls_config.go:310 level=info msg="TLS is enabled." http2=true address=10.100.1.187:9093
Jan 02 04:25:08 web-03 alertmanager-obs-iam-identity-universe-main-dev[1383672]: ts=2026-01-02T04:25:08.950Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001115024s
Jan 02 04:25:09 web-03 alertmanager-obs-iam-identity-universe-main-dev[1383672]: 2026/01/02 04:25:09 http: TLS handshake error from 10.100.1.187:38760: tls: client didn't provide a certificate

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Alerting Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: alertmanager
[INFO] FQDN: alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com
[INFO] IP: 10.100.1.187
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Setting up HTTPS reverse proxy...
[INFO] Backend port: 9093
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Alertmanager HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-iam-identity-universe-main-dev
  FQDN:         alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com
  Backend:      https://alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com:9093/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.187
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[ERROR] This script must be run as root or with sudo
[WARN] ⚠️  HTTPS setup failed (Alertmanager is still running on HTTP)
[INFO] Registering Alertmanager in monitoring dashboard...
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Alertmanager
[INFO]   Identifier:        iam-identity-universe-main-dev-alertmanager
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.187
[INFO]   Port:              9093
[INFO]   FQDN:              alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 22029e5d-7ca9-4f4a-9026-b97425827c60
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✅ Alertmanager registered in dashboard
[OK]   Alerting (alertmanager) deployed successfully
[INFO] Step 10.5: Deploying Blackbox Exporter for synthetic monitoring...
[BLACKBOX] Starting Blackbox Exporter deployment for obs-iam-identity-universe-main-dev
[BLACKBOX] VM IP: 10.100.1.182
[BLACKBOX] Version: 0.25.0
[BLACKBOX] Checking prerequisites...
[BLACKBOX] Creating directories...
[BLACKBOX] Downloading Blackbox Exporter v0.25.0...
Sorry, user www-data is not allowed to execute '/usr/bin/mv /tmp/tmp.AinWY88nP7/blackbox_exporter-0.25.0.linux-amd64/blackbox_exporter /usr/local/bin/' as root on web-03.
[WARN] Blackbox Exporter deployment failed (non-fatal, synthetic monitoring disabled)
[INFO] Step 11/13: Configuring HTTPS reverse proxies...
[INFO] Setting up Prometheus HTTPS proxy...
[2026-01-02 04:25:11 UTC] USER=www-data EUID=0 PID=1383775 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/https/setup-prometheus-https.sh --obs-cell obs-iam-identity-universe-main-dev --backend-ip 10.100.1.182
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Prometheus HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-iam-identity-universe-main-dev
  FQDN:         metrics-iam-identity-universe-main-dev-prometheus.fastorder.com
  Backend:      https://metrics-iam-identity-universe-main-dev-prometheus.fastorder.com:9090/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.182
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity (will retry up to 60s)...
[OK] Backend is accessible
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate exists and is valid for 69 more days
[INFO] Creating HTTPS VirtualHost with reverse proxy...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Setting up certificate auto-renewal...
[OK] Auto-renewal configured
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated
[INFO] Verifying HTTPS setup...
[OK] HTTPS endpoint is working

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Prometheus HTTPS Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HTTPS Endpoints:
  Health:   https://metrics-iam-identity-universe-main-dev-prometheus.fastorder.com/-/healthy
  Ready:    https://metrics-iam-identity-universe-main-dev-prometheus.fastorder.com/-/ready
  Graph:    https://metrics-iam-identity-universe-main-dev-prometheus.fastorder.com/graph
  Targets:  https://metrics-iam-identity-universe-main-dev-prometheus.fastorder.com/targets
  Alerts:   https://metrics-iam-identity-universe-main-dev-prometheus.fastorder.com/alerts
  API:      https://metrics-iam-identity-universe-main-dev-prometheus.fastorder.com/api/v1/...

Apache VirtualHosts:
  HTTP:  /etc/apache2/sites-available/metrics-iam-identity-universe-main-dev-prometheus.fastorder.com.conf
  HTTPS: /etc/apache2/sites-available/metrics-iam-identity-universe-main-dev-prometheus.fastorder.com-ssl.conf

Certificate:
  Path:     /etc/letsencrypt/live/metrics-iam-identity-universe-main-dev-prometheus.fastorder.com/
  Renewal:  certbot renew --cert-name metrics-iam-identity-universe-main-dev-prometheus.fastorder.com

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   Prometheus HTTPS proxy configured
[INFO] Setting up Grafana HTTPS proxy...
[2026-01-02 04:25:14 UTC] USER=www-data EUID=0 PID=1383912 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Dashboards/https/setup-grafana-https.sh --obs-cell obs-iam-identity-universe-main-dev --backend-ip 10.100.1.183
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-iam-identity-universe-main-dev
  FQDN:         dashboards-iam-identity-universe-main-dev-grafana.fastorder.com
  Backend:      https://dashboards-iam-identity-universe-main-dev-grafana.fastorder.com:3000/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.183
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate already exists
[INFO] Creating HTTPS VirtualHost...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana HTTPS Setup Complete!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  Grafana URL: https://dashboards-iam-identity-universe-main-dev-grafana.fastorder.com/
  Metrics:     https://dashboards-iam-identity-universe-main-dev-grafana.fastorder.com/metrics

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   Grafana HTTPS proxy configured
[INFO] Setting up OpenTelemetry Collector HTTPS proxy...
[2026-01-02 04:25:15 UTC] USER=www-data EUID=0 PID=1383997 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Telemetry/https/setup-otelcol-https.sh --obs-cell obs-iam-identity-universe-main-dev --backend-ip 10.100.1.188
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OpenTelemetry Collector HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-iam-identity-universe-main-dev
  FQDN:         telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com
  Backend:      http://telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com:8888/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.188
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[OK] Backend is accessible and returning metrics via HTTPS
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate exists and is valid for 69 more days
[INFO] Creating HTTPS VirtualHost with reverse proxy...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Setting up certificate auto-renewal...
[OK] Auto-renewal configured
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated
[INFO] Verifying HTTPS setup...
[OK] HTTPS endpoint is working and returning metrics

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OpenTelemetry Collector HTTPS Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HTTPS Endpoints:
  Metrics: https://telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com/metrics

Apache VirtualHosts:
  HTTP:  /etc/apache2/sites-available/telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com.conf
  HTTPS: /etc/apache2/sites-available/telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com-ssl.conf

Certificate:
  Path:     /etc/letsencrypt/live/telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com/
  Renewal:  certbot renew --cert-name telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   OpenTelemetry Collector HTTPS proxy configured
[INFO] Setting up ClickHouse HTTPS proxy...
[2026-01-02 04:25:18 UTC] USER=www-data EUID=0 PID=1384133 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/https/setup-clickhouse-https.sh --obs-cell obs-iam-identity-universe-main-dev --backend-ip 10.100.1.184
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ClickHouse HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-iam-identity-universe-main-dev
  FQDN:         logstore-iam-identity-universe-main-dev.fastorder.com
  Backend:      http://logstore-iam-identity-universe-main-dev.fastorder.com:8123/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.184
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity (will retry up to 60s)...
[OK] Backend is accessible
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate exists and is valid for 67 more days
[INFO] Creating HTTPS VirtualHost with reverse proxy...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Setting up certificate auto-renewal...
[OK] Auto-renewal configured
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated
[INFO] Verifying HTTPS setup...
[OK] HTTPS endpoint is working

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ ClickHouse HTTPS Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HTTPS Endpoints:
  Health:     https://logstore-iam-identity-universe-main-dev.fastorder.com/
  Dashboard:  https://logstore-iam-identity-universe-main-dev.fastorder.com/dashboard
  Playground: https://logstore-iam-identity-universe-main-dev.fastorder.com/play
  Metrics:    https://logstore-iam-identity-universe-main-dev.fastorder.com/metrics

Login Instructions:
  1. Get credentials from skeleton: POST /api/monitoring/clickhouse/credentials
  2. Use auto-login URL: https://logstore-iam-identity-universe-main-dev.fastorder.com/dashboard#user=<USER>&password=<PASS>
  3. Or use skeleton monitoring dashboard for one-click access

Apache VirtualHosts:
  HTTP:  /etc/apache2/sites-available/logstore-iam-identity-universe-main-dev.fastorder.com.conf
  HTTPS: /etc/apache2/sites-available/logstore-iam-identity-universe-main-dev.fastorder.com-ssl.conf

Certificate:
  Path: /etc/letsencrypt/live/logstore-iam-identity-universe-main-dev.fastorder.com/
  Auto-renewal: Enabled via certbot.timer

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ClickHouse HTTPS proxy configured
[INFO] Setting up Tempo HTTPS proxy...
[2026-01-02 04:25:21 UTC] USER=www-data EUID=0 PID=1384267 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Traces/https/setup-tempo-https.sh --obs-cell obs-iam-identity-universe-main-dev --backend-ip 10.100.1.186
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana Tempo HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-iam-identity-universe-main-dev
  FQDN:         traces-iam-identity-universe-main-dev-tempo.fastorder.com
  Backend:      https://traces-iam-identity-universe-main-dev-tempo.fastorder.com:3200/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.186
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[WARN] Cannot verify Tempo health endpoint (it may not be running yet), continuing anyway...
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate already exists
[INFO] Generating Apache client certificate for mTLS backend connection...
[OK] Apache client certificate already exists
[INFO] Creating HTTPS VirtualHost with mTLS backend...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana Tempo HTTPS Setup Complete!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  Tempo URL:    https://traces-iam-identity-universe-main-dev-tempo.fastorder.com/
  Ready:        https://traces-iam-identity-universe-main-dev-tempo.fastorder.com/ready
  Metrics:      https://traces-iam-identity-universe-main-dev-tempo.fastorder.com/metrics
  Build Info:   https://traces-iam-identity-universe-main-dev-tempo.fastorder.com/api/status/buildinfo

  Note: Tempo backend must be running at traces-iam-identity-universe-main-dev-tempo.fastorder.com:3200 (10.100.1.186)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   Tempo HTTPS proxy configured
[INFO] Setting up Alertmanager HTTPS proxy...
[2026-01-02 04:25:21 UTC] USER=www-data EUID=0 PID=1384341 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Alerting/https/setup-alertmanager-https.sh --obs-cell obs-iam-identity-universe-main-dev --backend-ip 10.100.1.187
[INFO] Backend port: 9093
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Alertmanager HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-iam-identity-universe-main-dev
  FQDN:         alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com
  Backend:      https://alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com:9093/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.187
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[WARN] Backend health check inconclusive - proceeding anyway
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate exists and is valid for 69 more days
[INFO] Creating HTTPS VirtualHost with reverse proxy...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Setting up certificate auto-renewal...
[OK] Auto-renewal configured
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated
[INFO] Verifying HTTPS setup...
OK[OK] HTTPS endpoint is working

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Alertmanager HTTPS Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HTTPS Endpoints:
  Health:   https://alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com/-/healthy
  Ready:    https://alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com/-/ready
  Web UI:   https://alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com/
  API:      https://alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com/api/v2/...

Apache VirtualHosts:
  HTTP:  /etc/apache2/sites-available/alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com.conf
  HTTPS: /etc/apache2/sites-available/alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com-ssl.conf

Certificate:
  Path:     /etc/letsencrypt/live/alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com/
  Renewal:  certbot renew --cert-name alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   Alertmanager HTTPS proxy configured
[OK]   HTTPS reverse proxies configured
[INFO] Step 12/13: Configuring firewall rules (network segmentation)...

[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING FIREWALL RULES FOR OBSERVABILITY CELL
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Cell ID: obs-iam-identity-universe-main-dev
[INFO] Internal Network: 10.0.0.0/8

[INFO] Discovering dashboard/skeleton VM IPs...
[INFO]   Discovered skeleton IP: 142.93.238.16 (skeleton.fastorder.com)
[INFO] Authorized dashboard IPs:
[INFO]   - 10.100.60.2
[INFO]   - 142.93.238.16

[INFO] Configuring UFW firewall rules...
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384486 ACTION=passthru ARGS=ufw --force enable
ERROR: passthru not allowed: ufw
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384494 ACTION=passthru ARGS=ufw default deny incoming
ERROR: passthru not allowed: ufw
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384502 ACTION=passthru ARGS=ufw default allow outgoing
ERROR: passthru not allowed: ufw
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384510 ACTION=passthru ARGS=ufw allow 22/tcp comment SSH
ERROR: passthru not allowed: ufw
[INFO]   Allowing prometheus (port 9090) from internal network...
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384518 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 9090 proto tcp comment Obs: prometheus from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing alertmanager (port 9093) from internal network...
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384526 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 9093 proto tcp comment Obs: alertmanager from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing clickhouse (port 8123) from internal network...
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384534 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 8123 proto tcp comment Obs: clickhouse from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing grafana (port 3000) from internal network...
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384542 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 3000 proto tcp comment Obs: grafana from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing otelcol (port 4318) from internal network...
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384550 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 4318 proto tcp comment Obs: otelcol from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing loki (port 3100) from internal network...
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384558 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 3100 proto tcp comment Obs: loki from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing tempo (port 3200) from internal network...
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384566 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 3200 proto tcp comment Obs: tempo from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing dashboard access from 10.100.60.2...
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384574 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 9090 proto tcp comment Dashboard: prometheus
ERROR: passthru not allowed: ufw
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384582 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 9093 proto tcp comment Dashboard: alertmanager
ERROR: passthru not allowed: ufw
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384590 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 8123 proto tcp comment Dashboard: clickhouse
ERROR: passthru not allowed: ufw
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384598 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3000 proto tcp comment Dashboard: grafana
ERROR: passthru not allowed: ufw
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384606 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 4318 proto tcp comment Dashboard: otelcol
ERROR: passthru not allowed: ufw
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384614 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3100 proto tcp comment Dashboard: loki
ERROR: passthru not allowed: ufw
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384622 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3200 proto tcp comment Dashboard: tempo
ERROR: passthru not allowed: ufw
[INFO]   Allowing dashboard access from 142.93.238.16...
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384630 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 9090 proto tcp comment Dashboard: prometheus
ERROR: passthru not allowed: ufw
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384638 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 9093 proto tcp comment Dashboard: alertmanager
ERROR: passthru not allowed: ufw
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384646 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 8123 proto tcp comment Dashboard: clickhouse
ERROR: passthru not allowed: ufw
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384654 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 3000 proto tcp comment Dashboard: grafana
ERROR: passthru not allowed: ufw
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384662 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 4318 proto tcp comment Dashboard: otelcol
ERROR: passthru not allowed: ufw
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384670 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 3100 proto tcp comment Dashboard: loki
ERROR: passthru not allowed: ufw
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384678 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 3200 proto tcp comment Dashboard: tempo
ERROR: passthru not allowed: ufw
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384686 ACTION=passthru ARGS=ufw allow 443/tcp comment HTTPS obs-proxy
ERROR: passthru not allowed: ufw
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384694 ACTION=passthru ARGS=ufw reload
ERROR: passthru not allowed: ufw
[OK]   UFW firewall rules configured

[OK]   ═══════════════════════════════════════════════════════════════
[OK]   ✅ Firewall configuration completed
[OK]   ═══════════════════════════════════════════════════════════════

[INFO] Current firewall status:
[2026-01-02 04:25:25 UTC] USER=www-data EUID=0 PID=1384704 ACTION=passthru ARGS=ufw status numbered
ERROR: passthru not allowed: ufw
[OK]   Firewall rules configured
[INFO] Step 13/13: Configuring OAuth/SSO...
[INFO] OAuth/SSO configuration script not found, skipping...

[INFO] Running validation checks...
[INFO] Validation script not found, skipping...

[INFO] Registering observability components to dashboard...
[INFO] Components to register: metrics alerts dashboards traces telemetry logstore proxy
[INFO]   Skipping metrics - registered by deploy script
[INFO]   Skipping alerts - registered by deploy script
[INFO]   Skipping dashboards - registered by deploy script
[INFO]   Skipping traces - registered by deploy script
[INFO]   Skipping telemetry - registered by deploy script
[INFO]   Skipping logstore - registered by deploy script
[INFO]   Processing component: proxy
[INFO] Registering: proxy (obs-iam-identity-universe-main-dev-proxy)
[INFO] Detected observability component, parsing: iam-identity-universe-main-dev-proxy
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Observability Proxy
[INFO]   Identifier:        obs-iam-identity-universe-main-dev-proxy
[INFO]   Identifier Parent: observability-cell
[INFO]   IP:                10.100.1.181
[INFO]   Port:              443
[INFO]   FQDN:              observe-iam-identity-universe-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 9f6bfd0e-29a1-480c-84b5-1508ebdbc00d
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✓ Registered: proxy
[INFO] Registering short DNS aliases...
[OK]   ✓ Observability components registration completed

[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] Verifying all observability services are running...
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[OK]     ✓ tempo-obs-iam-identity-universe-main-dev.service is running
[OK]     ✓ otelcol-obs-iam-identity-universe-main-dev.service is running
[OK]   ✓ All observability services verified running


═══════════════════════════════════════════════════════════════════════════════
[OK]   ✅ OBSERVABILITY CELL PROVISIONED: obs-iam-identity-universe-main-dev
═══════════════════════════════════════════════════════════════════════════════

[INFO] DNS Entries:
  metrics-iam-identity-universe-main-dev-prometheus.fastorder.com (10.100.1.182)
  alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com (10.100.1.187)
  dashboards-iam-identity-universe-main-dev-grafana.fastorder.com (10.100.1.183)
  traces-iam-identity-universe-main-dev-tempo.fastorder.com (10.100.1.186)
  telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com (10.100.1.188)
  logstore-iam-identity-universe-main-dev-clickhouse.fastorder.com (10.100.1.184)
  observe-iam-identity-universe-main-dev.fastorder.com (10.100.1.181)

[INFO] Secrets Path: fastorder/observability/iam-identity/universe/dev/*

[INFO] Access (Purpose-Oriented URLs):
  Dashboards: https://dashboards-iam-identity-universe-main-dev-grafana.fastorder.com (SSO enabled)
  Metrics: https://metrics-iam-identity-universe-main-dev-prometheus.fastorder.com (internal only)
  Alerts: https://alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com
  Log Storage: https://logstore-iam-identity-universe-main-dev-clickhouse.fastorder.com

[INFO] Backend Implementation (Internal - Not Exposed to Clients):
  Telemetry: otlp
  Metrics: prometheus
  Traces: tempo
  Dashboards: grafana
  Alerting: alertmanager
  Log Storage: clickhouse

[INFO] For applications in iam-identity-universe-main-dev:
  - Metrics: Push to telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com:4318 (OTLP/HTTP)
  - Logs: Push to telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com:4318 (OTLP/HTTP)
  - Traces: Push to telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com:4317 (OTLP/gRPC)
  - Query Metrics: https://metrics-iam-identity-universe-main-dev-prometheus.fastorder.com
  - Query Logs: https://logstore-iam-identity-universe-main-dev-clickhouse.fastorder.com
  - Query Traces: https://traces-iam-identity-universe-main-dev-tempo.fastorder.com

[INFO] Runbook: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/RUNBOOK.md
═══════════════════════════════════════════════════════════════════════════════
[OK] Observability cell setup completed for iam-identity

[INFO] Registering IAM node to skeleton observability system...
[INFO] IAM Environment: iam-identity-universe-main-dev
[INFO]   Registering to IAM environment: iam-identity-universe-main-dev
[INFO]   Application: Keycloak
[INFO]   Identifier: iam-identity-universe-main-dev-keycloak-controller
[INFO]   Identifier Parent: controller
[INFO]   IP: 142.93.238.16
[INFO]   Port: 8443
[INFO]   FQDN: keycloak-iam-identity-universe-main-dev.fastorder.com
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Keycloak
[INFO]   Identifier:        iam-identity-universe-main-dev-keycloak-controller
[INFO]   Identifier Parent: controller
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8443
[INFO]   FQDN:              keycloak-iam-identity-universe-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: d441f4c0-4040-4d7b-a86b-a41ff03cb256
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK] Keycloak node registered successfully to IAM tab

[INFO] ═══════════════════════════════════════════════════════════════════════════
[INFO] Registering Observability Components to Skeleton Dashboard
[INFO] ═══════════════════════════════════════════════════════════════════════════
[INFO] IAM Environment: iam-identity-universe-main-dev
[INFO] Checking topology file: /opt/fastorder/bash/scripts/env_app_setup/state/iam-identity-universe-main-dev/topology.json
[WARN] Topology file not found: /opt/fastorder/bash/scripts/env_app_setup/state/iam-identity-universe-main-dev/topology.json
[WARN] Attempting to read IPs from /etc/hosts instead...
[INFO] ────────────────────────────────────────────────────────────────────────
[INFO] Component: Prometheus (metrics)
[INFO]   FQDN: metrics-iam-identity-universe-main-dev-prometheus.fastorder.com
[INFO]   Port: 9090
[INFO]   IP:   10.100.1.182
[INFO]   Registering to IAM environment: iam-identity-universe-main-dev
[INFO]   Node Identifier: iam-identity-universe-main-dev-prometheus
[INFO]   Identifier Parent: observability
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Prometheus
[INFO]   Identifier:        iam-identity-universe-main-dev-prometheus
[INFO]   Identifier Parent: observability
[INFO]   IP:                10.100.1.182
[INFO]   Port:              9090
[INFO]   FQDN:              metrics-iam-identity-universe-main-dev-prometheus.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 38c5c0e7-47c3-4701-8e82-bc7cc0b8d3de
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✅ Prometheus registered successfully to IAM tab
[INFO] ────────────────────────────────────────────────────────────────────────
[INFO] Component: Alertmanager (alerts)
[INFO]   FQDN: alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com
[INFO]   Port: 9093
[INFO]   IP:   10.100.1.187
[INFO]   Registering to IAM environment: iam-identity-universe-main-dev
[INFO]   Node Identifier: iam-identity-universe-main-dev-alertmanager
[INFO]   Identifier Parent: observability
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Alertmanager
[INFO]   Identifier:        iam-identity-universe-main-dev-alertmanager
[INFO]   Identifier Parent: observability
[INFO]   IP:                10.100.1.187
[INFO]   Port:              9093
[INFO]   FQDN:              alerts-iam-identity-universe-main-dev-alertmanager.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 22029e5d-7ca9-4f4a-9026-b97425827c60
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✅ Alertmanager registered successfully to IAM tab
[INFO] ────────────────────────────────────────────────────────────────────────
[INFO] Component: Grafana (dashboards)
[INFO]   FQDN: dashboards-iam-identity-universe-main-dev-grafana.fastorder.com
[INFO]   Port: 3000
[INFO]   IP:   10.100.1.183
[INFO]   Registering to IAM environment: iam-identity-universe-main-dev
[INFO]   Node Identifier: iam-identity-universe-main-dev-grafana
[INFO]   Identifier Parent: observability
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Grafana
[INFO]   Identifier:        iam-identity-universe-main-dev-grafana
[INFO]   Identifier Parent: observability
[INFO]   IP:                10.100.1.183
[INFO]   Port:              3000
[INFO]   FQDN:              dashboards-iam-identity-universe-main-dev-grafana.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 6dc2f7d7-f9d7-49b6-9602-baa6d5321469
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✅ Grafana registered successfully to IAM tab
[INFO] ────────────────────────────────────────────────────────────────────────
[INFO] Component: Tempo (traces)
[INFO]   FQDN: traces-iam-identity-universe-main-dev-tempo.fastorder.com
[INFO]   Port: 3200
[INFO]   IP:   10.100.1.186
[INFO]   Registering to IAM environment: iam-identity-universe-main-dev
[INFO]   Node Identifier: iam-identity-universe-main-dev-tempo
[INFO]   Identifier Parent: observability
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Tempo
[INFO]   Identifier:        iam-identity-universe-main-dev-tempo
[INFO]   Identifier Parent: observability
[INFO]   IP:                10.100.1.186
[INFO]   Port:              3200
[INFO]   FQDN:              traces-iam-identity-universe-main-dev-tempo.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: ca4299ea-758f-4b03-afaa-ef64981f5219
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✅ Tempo registered successfully to IAM tab
[INFO] ────────────────────────────────────────────────────────────────────────
[INFO] Component: OpenTelemetry Collector (telemetry)
[INFO]   FQDN: telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com
[INFO]   Port: 4318
[INFO]   IP:   10.100.1.188
[INFO]   Registering to IAM environment: iam-identity-universe-main-dev
[INFO]   Node Identifier: iam-identity-universe-main-dev-opentelemetry
[INFO]   Identifier Parent: observability
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       OpenTelemetry Collector
[INFO]   Identifier:        iam-identity-universe-main-dev-opentelemetry
[INFO]   Identifier Parent: observability
[INFO]   IP:                10.100.1.188
[INFO]   Port:              4318
[INFO]   FQDN:              telemetry-iam-identity-universe-main-dev-opentelemetry.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 99feae51-7464-44b8-9c06-ec4d2e3893a4
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✅ OpenTelemetry Collector registered successfully to IAM tab
[INFO] ────────────────────────────────────────────────────────────────────────
[INFO] Component: ClickHouse (logstore)
[INFO]   FQDN: logstore-iam-identity-universe-main-dev-clickhouse.fastorder.com
[INFO]   Port: 8443
[INFO]   IP:   10.100.1.184
[INFO]   Registering to IAM environment: iam-identity-universe-main-dev
[INFO]   Node Identifier: iam-identity-universe-main-dev-clickhouse
[INFO]   Identifier Parent: observability
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       ClickHouse
[INFO]   Identifier:        iam-identity-universe-main-dev-clickhouse
[INFO]   Identifier Parent: observability
[INFO]   IP:                10.100.1.184
[INFO]   Port:              8443
[INFO]   FQDN:              logstore-iam-identity-universe-main-dev-clickhouse.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: c1ecc502-f764-4b48-a5d0-3e1c03b8c01a
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✅ ClickHouse registered successfully to IAM tab

[INFO] ═══════════════════════════════════════════════════════════════════════════
[INFO] Observability Components Registration Summary
[INFO] ═══════════════════════════════════════════════════════════════════════════
[INFO] Target: IAM tab (iam-identity-universe-main-dev)
[OK]   ✅ Registered: 6
[INFO] ═══════════════════════════════════════════════════════════════════════════

[INFO] ═══════════════════════════════════════════════════════════════════════════
[INFO] Linking IAM Sub-Service to Parent Environment
[INFO] ═══════════════════════════════════════════════════════════════════════════
[INFO] Linking iam-identity-universe-main-dev to parent identity-universe-main-dev...
[OK] IAM sub-service linked to parent environment successfully
[INFO] Dashboard URL: https://skeleton.dev.fastorder.com/dashboard/monitoring/environment/b6092921-3a75-44ac-9080-96a7ca43bec0#iam

════════════════════════════════════════════════════════════════════════════
[OK] IAM Observability Cell Provisioning Complete

[INFO] Components configured:
[INFO]   - Observability cell for iam-identity
[INFO]   - Node registration in skeleton dashboard
[INFO]   - Metrics and health endpoints configured

[INFO] Dashboard URL:
[INFO]   https://skeleton.dev.fastorder.com/dashboard/monitoring/environment/<env-uuid>#iam

[INFO] Next step: 04-database-setup.sh
[INFO]   - Provision PostgreSQL database for Keycloak
[INFO]   - Create IAM schemas (core, policy, audit)
════════════════════════════════════════════════════════════════════════════

[OK] ✅ Step 3 completed: 03-observability-cell-setup.sh

[INFO] 🔐 Step 4/12: database setup...
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║                    IAM Database Provisioning                                ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: coordinator

[INFO] Database engine: postgresql
[INFO] Setting up PostgreSQL for IAM...
[INFO]   Service: iam-identity
[INFO]   Environment: iam-identity-universe-main-dev

[INFO] Executing PostgreSQL setup for IAM...
[INFO] Using CURRENT_ENV_ID from environment: identity-universe-main-dev
[INFO] Loaded from topology.json: identity-universe-main-dev
[2026-01-02 04:25:47] Loaded environment: identity-universe-main-dev
[2026-01-02 04:25:47] Service: identity, Zone: universe, Branch: main, Env: dev
[2026-01-02 04:25:47] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-02 04:25:47] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-02 04:25:47] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
[INFO] Checking observability cell readiness: obs-iam-identity-universe-main-dev
[OK]   Observability cell endpoints registered for iam-identity-universe-main-dev
[INFO] Observability cell verified for iam-identity-universe-main-dev
[INFO] Monitoring will be configured after PostgreSQL deployment (step 10-monitoring-setup.sh)
[INFO] Citus mode ENABLED
[INFO] → Coordinator + 1 worker(s) + 1 standby node(s) per worker
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up coordinator (Citus control plane)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Auto-creating state directory for iam-identity-universe-main-dev...
[ OK ] Created topology.json for iam-identity-universe-main-dev
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-02 04:25:48 UTC] USER=unknown EUID=33 PID=1385210 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 04:25:48 UTC] USER=unknown EUID=33 PID=1385217 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 04:25:48 UTC] USER=unknown EUID=33 PID=1385224 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-02 04:25:48 UTC] USER=unknown EUID=33 PID=1385231 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 04:25:48 UTC] USER=unknown EUID=33 PID=1385238 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-02 04:25:48 UTC] USER=unknown EUID=33 PID=1385245 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for iam-identity in universe-dev...
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67 JOB_UUID=85222a65-779a-4529-84e6-94639bcc9bed

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[WARN] Could not find PostgreSQL IP for coordinator in topology.json, allocating new VM IP...
[INFO] Allocated new VM IP: 10.100.1.189 for db-coordinator-postgresql
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: coordinator
[INFO] PostgreSQL IP: 10.100.1.189
[INFO] Primary hostname: db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com

[INFO] Adding /etc/hosts entries for coordinator...
[INFO]   1. db-iam-identity-universe-main-dev-postgresql.fastorder.com → 10.100.1.189 (primary/short)
[INFO]   2. db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.189 (compatibility)

[INFO]   ➕ Adding db-iam-identity-universe-main-dev-postgresql.fastorder.com → 10.100.1.189
✅     ✅ Added: db-iam-identity-universe-main-dev-postgresql.fastorder.com → 10.100.1.189
[INFO]   ➕ Adding db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.189
✅     ✅ Added: db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.189

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.189    db-iam-identity-universe-main-dev-postgresql.fastorder.com
  10.100.1.189    db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  iam-identity-universe-main-dev.fastorder.com
  Alt CN:      iam-identity-universe-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-iam-identity-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 04:25:52 UTC] USER=www-data EUID=0 PID=1385710 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator and /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:25:52 UTC] USER=www-data EUID=0 PID=1385719 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-02 04:25:52 UTC] USER=www-data EUID=0 PID=1385729 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1385677
[2026-01-02 04:25:52 UTC] USER=www-data EUID=0 PID=1385738 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1385677/ra_root.crt
[2026-01-02 04:25:52 UTC] USER=www-data EUID=0 PID=1385747 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1385677/ra_root.key
[2026-01-02 04:25:52 UTC] USER=www-data EUID=0 PID=1385756 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1385677/ra_root.crt
[2026-01-02 04:25:52 UTC] USER=www-data EUID=0 PID=1385765 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1385677/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = iam-identity-universe-main-dev.fastorder.com
[2026-01-02 04:25:57 UTC] USER=www-data EUID=0 PID=1385802 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1385677/server.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.key
[2026-01-02 04:25:57 UTC] USER=www-data EUID=0 PID=1385811 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1385677/server.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt
[2026-01-02 04:25:57 UTC] USER=www-data EUID=0 PID=1385820 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-02 04:25:57 UTC] USER=www-data EUID=0 PID=1385829 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1385677/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:25:57 UTC] USER=www-data EUID=0 PID=1385838 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:25:57 UTC] USER=www-data EUID=0 PID=1385847 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:25:57 UTC] USER=www-data EUID=0 PID=1385856 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 04:25:57 UTC] USER=www-data EUID=0 PID=1385867 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.key
[2026-01-02 04:25:57 UTC] USER=www-data EUID=0 PID=1385876 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.key
[2026-01-02 04:25:57 UTC] USER=www-data EUID=0 PID=1385885 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt
[2026-01-02 04:25:57 UTC] USER=www-data EUID=0 PID=1385894 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt
[2026-01-02 04:25:57 UTC] USER=www-data EUID=0 PID=1385903 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:25:57 UTC] USER=www-data EUID=0 PID=1385912 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = iam-identity-universe-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:iam-identity-universe-main-dev.fastorder.com, DNS:iam-identity-universe-main-dev.fastorder.com, DNS:db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:db-iam-identity-universe-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-iam-identity-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-iam-identity-universe-main-dev-postgresql.fastorder.com, DNS:db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: iam-identity-universe-main-dev
Node:        coordinator
Primary CN:  iam-identity-universe-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@iam-identity-universe-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=iam-identity-universe-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
Environment: iam-identity-universe-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:25:58 UTC] USER=www-data EUID=0 PID=1385966 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-01-02 04:25:58 UTC] USER=www-data EUID=0 PID=1385975 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 04:25:58 UTC] USER=www-data EUID=0 PID=1385984 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-01-02 04:25:58 UTC] USER=www-data EUID=0 PID=1385993 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 04:25:58 UTC] USER=www-data EUID=0 PID=1386002 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:25:58 UTC] USER=www-data EUID=0 PID=1386016 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:25:58 UTC] USER=www-data EUID=0 PID=1386025 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:25:58 UTC] USER=www-data EUID=0 PID=1386034 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key
[2026-01-02 04:25:58 UTC] USER=www-data EUID=0 PID=1386043 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.crt
[2026-01-02 04:25:58 UTC] USER=www-data EUID=0 PID=1386052 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:25:58 UTC] USER=www-data EUID=0 PID=1386061 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:25:58 UTC] USER=www-data EUID=0 PID=1386070 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 04:25:58 UTC] USER=www-data EUID=0 PID=1386079 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres_der.key
[2026-01-02 04:25:58 UTC] USER=www-data EUID=0 PID=1386088 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key
[2026-01-02 04:25:58 UTC] USER=www-data EUID=0 PID=1386097 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386106 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386115 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386124 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386133 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres_der.key
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386142 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386151 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386177 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386186 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386195 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386204 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386213 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386222 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386231 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386240 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386249 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386258 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386267 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres_der.key
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386277 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386287 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386296 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386305 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386314 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386323 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386332 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386341 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386350 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386359 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386368 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386377 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres_der.key
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386387 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386397 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386406 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:25:59 UTC] USER=www-data EUID=0 PID=1386415 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386424 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386433 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386442 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386451 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386460 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386469 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386478 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386487 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres_der.key
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386497 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386507 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386516 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386525 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386534 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386543 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386552 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386561 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386570 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386579 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386588 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386597 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres_der.key
[2026-01-02 04:26:00 UTC] USER=www-data EUID=0 PID=1386607 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
🎉 All requested users processed.
ℹ️  Kafka certificates not found at /opt/kafka/secrets/iam-identity-universe-main-dev/coordinator/pem (Kafka may not be installed yet)

✅ Client certificate generated successfully!

Environment: iam-identity-universe-main-dev
User: postgres
Node: coordinator
FQDN: db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
Environment: iam-identity-universe-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:26:01 UTC] USER=www-data EUID=0 PID=1386648 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-01-02 04:26:01 UTC] USER=www-data EUID=0 PID=1386657 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 04:26:01 UTC] USER=www-data EUID=0 PID=1386666 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-01-02 04:26:01 UTC] USER=www-data EUID=0 PID=1386675 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-02 04:26:01 UTC] USER=www-data EUID=0 PID=1386684 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:01 UTC] USER=www-data EUID=0 PID=1386717 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:01 UTC] USER=www-data EUID=0 PID=1386726 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:01 UTC] USER=www-data EUID=0 PID=1386735 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key
[2026-01-02 04:26:01 UTC] USER=www-data EUID=0 PID=1386744 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.crt
[2026-01-02 04:26:01 UTC] USER=www-data EUID=0 PID=1386753 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:26:01 UTC] USER=www-data EUID=0 PID=1386762 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386771 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386780 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres_der.key
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386789 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386798 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386807 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres_der.key
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386816 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386825 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386834 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386843 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386852 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres_der.key
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386861 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386870 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386896 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386905 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386914 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386923 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386932 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386941 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386950 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386959 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386968 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386977 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386986 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres_der.key
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1386996 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1387006 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1387015 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1387024 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1387033 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1387042 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:02 UTC] USER=www-data EUID=0 PID=1387051 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387060 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387069 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387078 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387087 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387096 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres_der.key
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387106 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387116 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387125 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387134 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387143 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387152 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387161 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387170 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387179 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387188 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387197 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387206 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres_der.key
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387216 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387226 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387235 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387244 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387253 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387262 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387271 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387280 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387289 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387298 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387307 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key.pkcs1
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387316 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres_der.key
[2026-01-02 04:26:03 UTC] USER=www-data EUID=0 PID=1387326 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
🎉 All requested users processed.
ℹ️  Kafka certificates not found at /opt/kafka/secrets/iam-identity-universe-main-dev/coordinator/pem (Kafka may not be installed yet)

✅ Client certificate generated successfully!

Environment: iam-identity-universe-main-dev
User: postgres
Node: coordinator
FQDN: db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-coordinator-postgresql environment: db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com (10.100.1.189)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.189
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: coordinator
[INFO] Data dir:   /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator
[INFO] Port:       5432
[INFO] Hostname:   db-iam-identity-universe-main-dev-postgresql-coordinator
[2026-01-02 04:26:05 UTC] USER=www-data EUID=0 PID=1387422 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:05 UTC] USER=www-data EUID=0 PID=1387444 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:05 UTC] USER=www-data EUID=0 PID=1387465 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:05 UTC] USER=www-data EUID=0 PID=1387486 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  iam-identity-universe-main-dev.fastorder.com
  Alt CN:      iam-identity-universe-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-iam-identity-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 04:26:06 UTC] USER=www-data EUID=0 PID=1387527 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator and /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:06 UTC] USER=www-data EUID=0 PID=1387536 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-02 04:26:06 UTC] USER=www-data EUID=0 PID=1387546 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1387493
[2026-01-02 04:26:06 UTC] USER=www-data EUID=0 PID=1387555 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1387493/ra_root.crt
[2026-01-02 04:26:06 UTC] USER=www-data EUID=0 PID=1387564 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1387493/ra_root.key
[2026-01-02 04:26:06 UTC] USER=www-data EUID=0 PID=1387573 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1387493/ra_root.crt
[2026-01-02 04:26:06 UTC] USER=www-data EUID=0 PID=1387582 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1387493/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = iam-identity-universe-main-dev.fastorder.com
[2026-01-02 04:26:13 UTC] USER=www-data EUID=0 PID=1387621 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1387493/server.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.key
[2026-01-02 04:26:13 UTC] USER=www-data EUID=0 PID=1387630 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1387493/server.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt
[2026-01-02 04:26:13 UTC] USER=www-data EUID=0 PID=1387639 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-02 04:26:13 UTC] USER=www-data EUID=0 PID=1387648 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1387493/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:26:13 UTC] USER=www-data EUID=0 PID=1387657 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:26:13 UTC] USER=www-data EUID=0 PID=1387666 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:26:13 UTC] USER=www-data EUID=0 PID=1387675 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 04:26:13 UTC] USER=www-data EUID=0 PID=1387686 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.key
[2026-01-02 04:26:13 UTC] USER=www-data EUID=0 PID=1387695 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.key
[2026-01-02 04:26:13 UTC] USER=www-data EUID=0 PID=1387704 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt
[2026-01-02 04:26:13 UTC] USER=www-data EUID=0 PID=1387713 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt
[2026-01-02 04:26:13 UTC] USER=www-data EUID=0 PID=1387722 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:13 UTC] USER=www-data EUID=0 PID=1387731 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = iam-identity-universe-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:iam-identity-universe-main-dev.fastorder.com, DNS:iam-identity-universe-main-dev.fastorder.com, DNS:db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com, DNS:db-iam-identity-universe-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-iam-identity-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-universe-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-iam-identity-universe-main-dev-postgresql.fastorder.com, DNS:db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: iam-identity-universe-main-dev
Node:        coordinator
Primary CN:  iam-identity-universe-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@iam-identity-universe-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=iam-identity-universe-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-02 04:26:13 UTC] USER=www-data EUID=0 PID=1387760 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt
[2026-01-02 04:26:13 UTC] USER=www-data EUID=0 PID=1387769 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.key
[2026-01-02 04:26:13 UTC] USER=www-data EUID=0 PID=1387778 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-02 04:26:14 UTC] USER=www-data EUID=0 PID=1387799 ACTION=passthru ARGS=systemctl stop postgresql@iam-identity-universe-main-dev-coordinator.service
[2026-01-02 04:26:14 UTC] USER=www-data EUID=0 PID=1387820 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-iam-identity-universe-main-dev-coordinator
[2026-01-02 04:26:14 UTC] USER=www-data EUID=0 PID=1387851 ACTION=fsop ARGS=rm -rf /var/run/postgresql-iam-identity-universe-main-dev-coordinator
[OK]   No conflicting Postgres left on port 5432
[OK]   Generated new postgres password for initdb
[2026-01-02 04:26:37 UTC] USER=www-data EUID=0 PID=1388050 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.eGXHOV
[2026-01-02 04:26:37 UTC] USER=www-data EUID=0 PID=1388071 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.eGXHOV
[2026-01-02 04:26:37 UTC] USER=www-data EUID=0 PID=1388093 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/iam-identity-universe-main-dev
[2026-01-02 04:26:37 UTC] USER=www-data EUID=0 PID=1388115 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/iam-identity-universe-main-dev
[2026-01-02 04:26:38 UTC] USER=www-data EUID=0 PID=1388137 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/iam-identity-universe-main-dev
[INFO] Initializing cluster in /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator (SCRAM; pwfile)
[2026-01-02 04:26:38 UTC] USER=www-data EUID=0 PID=1388159 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:38 UTC] USER=www-data EUID=0 PID=1388180 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:38 UTC] USER=www-data EUID=0 PID=1388201 ACTION=fsop ARGS=chmod 700 /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:26:38 UTC] USER=www-data EUID=0 PID=1388222 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-iam-identity-universe-main-dev-coordinator
[2026-01-02 04:26:38 UTC] USER=www-data EUID=0 PID=1388243 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-iam-identity-universe-main-dev-coordinator
[2026-01-02 04:26:38 UTC] USER=www-data EUID=0 PID=1388264 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-iam-identity-universe-main-dev-coordinator
[2026-01-02 04:26:38 UTC] USER=www-data EUID=0 PID=1388273 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.eGXHOV
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator -l logfile start

[OK]   initdb complete
[2026-01-02 04:26:39 UTC] USER=www-data EUID=0 PID=1388308 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.eGXHOV
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-02 04:26:39 UTC] USER=www-data EUID=0 PID=1388355 ACTION=fsop ARGS=cp /tmp/tmp.pVxdV91x2J /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator/pg_hba.conf
[2026-01-02 04:26:39 UTC] USER=www-data EUID=0 PID=1388376 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator/pg_hba.conf
[2026-01-02 04:26:40 UTC] USER=www-data EUID=0 PID=1388397 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@iam-identity-universe-main-dev-coordinator.service
[2026-01-02 04:26:40 UTC] USER=www-data EUID=0 PID=1388422 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.8etZap /etc/systemd/system/postgresql@iam-identity-universe-main-dev-coordinator.service
[2026-01-02 04:26:40 UTC] USER=www-data EUID=0 PID=1388443 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@iam-identity-universe-main-dev-coordinator.service
[OK]   systemd unit written
[2026-01-02 04:26:40 UTC] USER=www-data EUID=0 PID=1388464 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 04:26:40 UTC] USER=www-data EUID=0 PID=1388485 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 04:26:40 UTC] USER=www-data EUID=0 PID=1388506 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-02 04:26:41 UTC] USER=www-data EUID=0 PID=1388620 ACTION=passthru ARGS=systemctl start postgresql@iam-identity-universe-main-dev-coordinator.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-02 04:26:42 UTC] USER=www-data EUID=0 PID=1388661 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@iam-identity-universe-main-dev-coordinator.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Provisioning application database and Debezium role (if not exists)...
[INFO] Checking if database fastorder_iam_identity_universe_main_dev_db exists...
[INFO] DB check result: exit_code=0, output='[2026-01-02 04:26:43 UTC] USER=www-data EUID=0 PID=1388816 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_iam_identity_universe_main_dev_db''
[INFO] Creating database fastorder_iam_identity_universe_main_dev_db...
[2026-01-02 04:26:43 UTC] USER=www-data EUID=0 PID=1388839 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_iam_identity_universe_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[OK]   Database fastorder_iam_identity_universe_main_dev_db created
[INFO] Checking if role debezium_user exists...
[INFO] Role check result: exit_code=0, output='[2026-01-02 04:26:43 UTC] USER=www-data EUID=0 PID=1388863 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[INFO] Creating role debezium_user...
[2026-01-02 04:26:44 UTC] USER=www-data EUID=0 PID=1388890 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'i9NHU+UQd+0gYtG0I0FnM4EI';
CREATE ROLE
[OK]   Role debezium_user created
[2026-01-02 04:26:44 UTC] USER=www-data EUID=0 PID=1388913 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_iam_identity_universe_main_dev_db" TO debezium_user;
GRANT
[OK]   Application DB (fastorder_iam_identity_universe_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[INFO] Applying connection and memory optimizations...
[INFO] Current settings: max_connections=100, work_mem=4MB
[INFO] Target settings (coordinator): max_connections=150, work_mem=8MB
[2026-01-02 04:26:44 UTC] USER=www-data EUID=0 PID=1388990 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 150;
ALTER SYSTEM
[2026-01-02 04:26:44 UTC] USER=www-data EUID=0 PID=1389013 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-02 04:26:44 UTC] USER=www-data EUID=0 PID=1389036 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   Settings applied to postgresql.auto.conf
[2026-01-02 04:26:45 UTC] USER=www-data EUID=0 PID=1389051 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator/standby.signal
[INFO] Service recently started (3s ago) - restarting to apply max_connections...
[INFO] Stopping service...
[2026-01-02 04:26:45 UTC] USER=www-data EUID=0 PID=1389073 ACTION=passthru ARGS=systemctl stop postgresql@iam-identity-universe-main-dev-coordinator.service
[INFO] Waiting for port 5432 to be released...
[OK]   Port 5432 released
[INFO] Starting service...
[2026-01-02 04:26:48 UTC] USER=www-data EUID=0 PID=1389107 ACTION=passthru ARGS=systemctl start postgresql@iam-identity-universe-main-dev-coordinator.service
[2026-01-02 04:26:54 UTC] USER=www-data EUID=0 PID=1389154 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@iam-identity-universe-main-dev-coordinator.service
[OK]   ✅ Optimization complete: max_connections=150, work_mem=8MB
[INFO] Setting postgres password via centralized script... for coordinator
[INFO] Temporarily disabling synchronous_commit on coordinator for password setting...
[OK]   Disabled synchronous_commit (was: on)
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    iam-identity
  Zone:       universe
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator

Connection Info:
  Socket Dir: /var/run/postgresql-iam-identity-universe-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Initial setup: Using password from initdb
✓ PostgreSQL password already set during initdb
Storing password in AWS Secrets Manager: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator
✅ Secret created: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator
✅ PostgreSQL credentials set in vault: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
[INFO] Restoring synchronous_commit on coordinator...
[OK]   Restored synchronous_commit to: on
[OK]   Password set and persisted
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: coordinator
[INFO] PostgreSQL IP: 10.100.1.189
[INFO] Primary hostname: db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com

[INFO] Adding /etc/hosts entries for coordinator...
[INFO]   1. db-iam-identity-universe-main-dev-postgresql.fastorder.com → 10.100.1.189 (primary/short)
[INFO]   2. db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.189 (compatibility)

[INFO]   ✅ db-iam-identity-universe-main-dev-postgresql.fastorder.com already exists with correct IP
[INFO]   ✅ db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.189    db-iam-identity-universe-main-dev-postgresql.fastorder.com
  10.100.1.189    db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com


[OK]   PostgreSQL 'iam-identity-universe-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key \
        host=db-iam-identity-universe-main-dev-postgresql-coordinator port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        iam-identity-universe-main-dev-postgresql-coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.189
[INFO]   Port:              5432
[INFO]   FQDN:              db-iam-identity-universe-main-dev-postgresql-coordinator
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: f0341e35-122d-498a-a5e5-cf38a434b79d
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 📦 03 role...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[2026-01-02 04:27:06 UTC] USER=www-data EUID=0 PID=1389596 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
Environment: iam-identity-universe-main-dev
Username:    debezium_user
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   debezium_user
  Hostname:    db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:27:27 UTC] USER=www-data EUID=0 PID=1389742 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-debezium_user
[2026-01-02 04:27:27 UTC] USER=www-data EUID=0 PID=1389751 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-01-02 04:27:27 UTC] USER=www-data EUID=0 PID=1389760 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-debezium_user/ra_root.key
[2026-01-02 04:27:27 UTC] USER=www-data EUID=0 PID=1389769 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389778 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389792 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389801 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389810 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.key
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389819 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.crt
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389828 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389837 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389846 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389855 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user_der.key
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389864 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.key
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389873 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389882 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389891 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.key
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389900 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389909 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user_der.key
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389918 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389927 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389953 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389962 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389971 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:27:28 UTC] USER=www-data EUID=0 PID=1389980 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1389989 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1389998 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.key
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390007 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.crt
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390016 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390025 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390034 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390043 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user_der.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user_der.key
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390053 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390063 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390072 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390081 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390090 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390099 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390108 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.key
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390117 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.crt
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390126 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390135 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390144 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390153 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user_der.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user_der.key
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390163 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390173 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390182 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390191 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390200 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390209 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390218 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.key
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390227 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.crt
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390236 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390245 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390254 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390263 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user_der.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user_der.key
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390273 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:29 UTC] USER=www-data EUID=0 PID=1390283 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:30 UTC] USER=www-data EUID=0 PID=1390292 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:27:30 UTC] USER=www-data EUID=0 PID=1390301 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:27:30 UTC] USER=www-data EUID=0 PID=1390310 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:30 UTC] USER=www-data EUID=0 PID=1390319 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:30 UTC] USER=www-data EUID=0 PID=1390328 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.key
[2026-01-02 04:27:30 UTC] USER=www-data EUID=0 PID=1390337 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.crt
[2026-01-02 04:27:30 UTC] USER=www-data EUID=0 PID=1390346 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:27:30 UTC] USER=www-data EUID=0 PID=1390355 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:27:30 UTC] USER=www-data EUID=0 PID=1390364 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-02 04:27:30 UTC] USER=www-data EUID=0 PID=1390373 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/debezium_user_der.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user_der.key
[2026-01-02 04:27:30 UTC] USER=www-data EUID=0 PID=1390383 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
🎉 All requested users processed.
ℹ️  Kafka certificates not found at /opt/kafka/secrets/iam-identity-universe-main-dev/coordinator/pem (Kafka may not be installed yet)

✅ Client certificate generated successfully!

Environment: iam-identity-universe-main-dev
User: debezium_user
Node: coordinator
FQDN: db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com -U debezium_user -d postgres

✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    iam-identity
  Zone:       universe
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/fastorder_admin_gd

Connection Info:
  Socket Dir: /var/run/postgresql-iam-identity-universe-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: fastorder_admin_gd
Storing password in AWS Secrets Manager: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Secret created: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: coordinator/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-iam-identity-universe-main-dev-postgresql.fastorder.com
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
Environment: iam-identity-universe-main-dev
Username:    fastorder_admin_gd
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   fastorder_admin_gd
  Hostname:    db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:27:43 UTC] USER=www-data EUID=0 PID=1390873 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-fastorder_admin_gd
[2026-01-02 04:27:43 UTC] USER=www-data EUID=0 PID=1390882 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-02 04:27:43 UTC] USER=www-data EUID=0 PID=1390892 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
[2026-01-02 04:27:43 UTC] USER=www-data EUID=0 PID=1390902 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-02 04:27:43 UTC] USER=www-data EUID=0 PID=1390913 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1390929 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1390938 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1390947 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1390956 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1390965 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1390974 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1390983 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1390992 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391001 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391010 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391019 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391028 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391037 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391046 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391055 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391064 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391073 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391082 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391108 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391117 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391126 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391135 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391144 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391153 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391162 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391171 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391180 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:27:44 UTC] USER=www-data EUID=0 PID=1391189 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391198 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391208 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391218 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391227 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391236 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391245 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391254 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391263 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391272 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391281 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391302 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391313 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391323 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391333 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391342 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391351 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391360 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391369 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391378 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391387 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391396 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391407 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391419 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391428 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391438 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391448 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391457 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:27:45 UTC] USER=www-data EUID=0 PID=1391466 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:27:46 UTC] USER=www-data EUID=0 PID=1391475 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:46 UTC] USER=www-data EUID=0 PID=1391484 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
[2026-01-02 04:27:46 UTC] USER=www-data EUID=0 PID=1391493 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-02 04:27:46 UTC] USER=www-data EUID=0 PID=1391502 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-02 04:27:46 UTC] USER=www-data EUID=0 PID=1391511 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt
[2026-01-02 04:27:46 UTC] USER=www-data EUID=0 PID=1391520 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
[2026-01-02 04:27:46 UTC] USER=www-data EUID=0 PID=1391530 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-02 04:27:46 UTC] USER=www-data EUID=0 PID=1391540 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-02 04:27:46 UTC] USER=www-data EUID=0 PID=1391550 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
🎉 All requested users processed.
ℹ️  Kafka certificates not found at /opt/kafka/secrets/iam-identity-universe-main-dev/coordinator/pem (Kafka may not be installed yet)

✅ Client certificate generated successfully!

Environment: iam-identity-universe-main-dev
User: fastorder_admin_gd
Node: coordinator
FQDN: db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-iam-identity-universe-main-dev-coordinator:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_iam_identity_universe_main_dev_db already exists, skipping creation
[2026-01-02 04:27:46 UTC] USER=www-data EUID=0 PID=1391609 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-iam-identity-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-02 04:27:46 UTC] USER=www-data EUID=0 PID=1391644 ACTION=passthru ARGS=systemctl reload postgresql@iam-identity-universe-main-dev-coordinator.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

=== Pre-flight Checks ===
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
✓ AWS Secrets Manager accessible

=== Retrieving Credentials from AWS ===
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
/opt/fastorder/bash/infra_core/cache.sh: line 145: /var/cache/secrets/fastorder_db_iam-identity_universe_main_dev_postgresql_coordinator_fastorder_admin_gd.cache.tmp.1391675: Permission denied
✅ Retrieved from secrets manager: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials loaded for coordinator/fastorder_admin_gd: fastorder_admin_gd@db-iam-identity-universe-main-dev-postgresql.fastorder.com:5432/fastorder_iam_identity_universe_main_dev_db
✓ Credentials retrieved: fastorder_admin_gd@db-iam-identity-universe-main-dev-postgresql.fastorder.com:5432/fastorder_iam_identity_universe_main_dev_db
╔════════════════════════════════════════════╗
║  PostgreSQL Test Suite (AWS Secrets MGR)  ║
╚════════════════════════════════════════════╝

=== PostgreSQL Authentication Test ===
✗ PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-iam-identity-universe-main-dev-postgresql.fastorder.com" (10.100.1.189), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[2026-01-02 04:27:54 UTC] USER=www-data EUID=0 PID=1391871 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : iam-identity-universe-main-dev
  IDENTIFIER  : coordinator
  PG HOST     : db-iam-identity-universe-main-dev-postgresql.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_iam_identity_universe_main_dev_db
  SCHEMA      : auth
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator
  DNS → 10.100.1.189
  CA         : /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    iam-identity
  Zone:       universe
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/debezium_user

Connection Info:
  Socket Dir: /var/run/postgresql-iam-identity-universe-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User debezium_user does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: debezium_user
Storing password in AWS Secrets Manager: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/debezium_user
✅ Secret created: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/debezium_user
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/debezium_user
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: coordinator/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-01-02 04:28:06 UTC] USER=www-data EUID=0 PID=1392256 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-iam-identity-universe-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_iam_identity_universe_main_dev_db already exists
[2026-01-02 04:28:07 UTC] USER=www-data EUID=0 PID=1392287 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-iam-identity-universe-main-dev-coordinator -p 5432 -d fastorder_iam_identity_universe_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (iam-identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=coordinator IDENTIFIER_PARENT=coordinator
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║                    IAM Database Schema Initialization                       ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] 🟢 Starting IAM schema provisioning...
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

[INFO] 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: core
  Core Identity Directory (tenants, realms, identities, devices, MFA)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [1/20]: core/01-tenant
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Identifier:  coordinator
  Database:    fastorder_iam_identity_universe_main_dev_db
  Host:        db-iam-identity-universe-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_iam_identity_universe_main_dev_db
ℹ️  Database fastorder_iam_identity_universe_main_dev_db already exists
✅ Connected to database: fastorder_iam_identity_universe_main_dev_db
🔧 Installing extensions...
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
   Creating reference table: core.tenant
 create_reference_table 
------------------------
 
(1 row)

✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[OK] Table core/01-tenant initialized

[INFO] 🔸 Table [2/20]: core/02-realm
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.realm$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_realm_updated" for relation "core.realm" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[OK] Table core/02-realm initialized

[INFO] 🔸 Table [3/20]: core/03-identity
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_identity_updated" for relation "core.identity" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[OK] Table core/03-identity initialized

[INFO] 🔸 Table [4/20]: core/04-device
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.device$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[OK] Table core/04-device initialized

[INFO] 🔸 Table [5/20]: core/05-identity_account
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity_account$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_identity_account_updated" for relation "core.identity_account" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[OK] Table core/05-identity_account initialized

[INFO] 🔸 Table [6/20]: core/06-identity_mfa
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity_mfa$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[OK] Table core/06-identity_mfa initialized

[INFO] 🔸 Table [7/20]: core/07-external_idp_link
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.external_idp_link$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[OK] Table core/07-external_idp_link initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: policy
  RBAC/ABAC Authorization (clients, roles, permissions, policies)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [8/20]: policy/01-client
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.client$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
NOTICE:  trigger "tr_client_updated" for relation "policy.client" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[OK] Table policy/01-client initialized

[INFO] 🔸 Table [9/20]: policy/02-resource
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.resource$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[OK] Table policy/02-resource initialized

[INFO] 🔸 Table [10/20]: policy/03-scope
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.scope$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_scope_updated" for relation "policy.scope" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[OK] Table policy/03-scope initialized

[INFO] 🔸 Table [11/20]: policy/04-permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.permission$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_permission_updated" for relation "policy.permission" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[OK] Table policy/04-permission initialized

[INFO] 🔸 Table [12/20]: policy/05-role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.role$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_role_updated" for relation "policy.role" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[OK] Table policy/05-role initialized

[INFO] 🔸 Table [13/20]: policy/06-role_permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.role_permission$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[OK] Table policy/06-role_permission initialized

[INFO] 🔸 Table [14/20]: policy/07-identity_role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.identity_role$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[OK] Table policy/07-identity_role initialized

[INFO] 🔸 Table [15/20]: policy/08-policy_rule
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.policy_rule$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_policy_rule_updated" for relation "policy.policy_rule" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[OK] Table policy/08-policy_rule initialized

[INFO] 🔸 Table [16/20]: policy/09-api_key
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.api_key$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[OK] Table policy/09-api_key initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: audit
  Audit & Risk Logging (auth events, admin actions, risk decisions)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [17/20]: audit/01-auth_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[OK] Table audit/01-auth_event initialized

[INFO] 🔸 Table [18/20]: audit/02-admin_action
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[OK] Table audit/02-admin_action initialized

[INFO] 🔸 Table [19/20]: audit/03-risk_decision
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[OK] Table audit/03-risk_decision initialized

[INFO] 🔸 Table [20/20]: audit/04-consent_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[OK] Table audit/04-consent_event initialized


════════════════════════════════════════════════════════════════════════════
[OK] ✅ IAM Schema Initialization Complete!
[OK] All 20 tables initialized successfully

Schemas created:
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

Design highlights:
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

════════════════════════════════════════════════════════════════════════════

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=coordinator IDENTIFIER_PARENT=coordinator
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for iam-identity in universe-dev...
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: iam-identity/run.sh started for SERVICE=iam-identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../iam-identity
🔍 DEBUG_CHECKPOINT_A3_FAIL: SERVICE_ROOT does not exist!
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: NONE
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
🔍 DEBUG_CHECKPOINT_A6_FAIL: No table folders found!
[WARN] No grouped table folders found under: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../iam-identity
End of 04-postgresql/steps/01-install/steps/iam-identity/run.sh

✓ ✅ Coordinator setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up 1 worker(s) (Citus data nodes)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up worker: worker-01
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-02 04:29:30 UTC] USER=unknown EUID=33 PID=1395420 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 04:29:30 UTC] USER=unknown EUID=33 PID=1395428 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 04:29:30 UTC] USER=unknown EUID=33 PID=1395435 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-02 04:29:30 UTC] USER=unknown EUID=33 PID=1395442 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 04:29:30 UTC] USER=unknown EUID=33 PID=1395449 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-02 04:29:30 UTC] USER=unknown EUID=33 PID=1395456 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for iam-identity in universe-dev...
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67 JOB_UUID=85222a65-779a-4529-84e6-94639bcc9bed

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[WARN] Could not find PostgreSQL IP for worker-01 in topology.json, allocating new VM IP...
[INFO] Allocated new VM IP: 10.100.1.190 for db-worker-01-postgresql
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.190
[INFO] Primary hostname: db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01...
[INFO]   db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.190

[INFO]   ➕ Adding db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.190
✅     ✅ Added: db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.190

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.190    db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  iam-identity-universe-main-dev.fastorder.com
  Alt CN:      iam-identity-universe-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 04:29:34 UTC] USER=www-data EUID=0 PID=1395918 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01 and /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:34 UTC] USER=www-data EUID=0 PID=1395927 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-02 04:29:34 UTC] USER=www-data EUID=0 PID=1395937 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1395884
[2026-01-02 04:29:34 UTC] USER=www-data EUID=0 PID=1395946 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1395884/ra_root.crt
[2026-01-02 04:29:34 UTC] USER=www-data EUID=0 PID=1395955 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1395884/ra_root.key
[2026-01-02 04:29:34 UTC] USER=www-data EUID=0 PID=1395965 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1395884/ra_root.crt
[2026-01-02 04:29:34 UTC] USER=www-data EUID=0 PID=1395974 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1395884/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = iam-identity-universe-main-dev.fastorder.com
[2026-01-02 04:29:35 UTC] USER=www-data EUID=0 PID=1396114 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1395884/server.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.key
[2026-01-02 04:29:36 UTC] USER=www-data EUID=0 PID=1396123 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1395884/server.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.crt
[2026-01-02 04:29:36 UTC] USER=www-data EUID=0 PID=1396132 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 04:29:36 UTC] USER=www-data EUID=0 PID=1396141 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1395884/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:36 UTC] USER=www-data EUID=0 PID=1396150 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:36 UTC] USER=www-data EUID=0 PID=1396164 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:36 UTC] USER=www-data EUID=0 PID=1396173 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 04:29:36 UTC] USER=www-data EUID=0 PID=1396184 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.key
[2026-01-02 04:29:36 UTC] USER=www-data EUID=0 PID=1396193 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.key
[2026-01-02 04:29:36 UTC] USER=www-data EUID=0 PID=1396202 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.crt
[2026-01-02 04:29:36 UTC] USER=www-data EUID=0 PID=1396211 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.crt
[2026-01-02 04:29:36 UTC] USER=www-data EUID=0 PID=1396220 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:36 UTC] USER=www-data EUID=0 PID=1396229 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = iam-identity-universe-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:iam-identity-universe-main-dev.fastorder.com, DNS:iam-identity-universe-main-dev.fastorder.com, DNS:db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:db-iam-identity-universe-main-dev-postgresql-worker-01, DNS:localhost, DNS:db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: iam-identity-universe-main-dev
Node:        worker-01
Primary CN:  iam-identity-universe-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@iam-identity-universe-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=iam-identity-universe-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
Environment: iam-identity-universe-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396290 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396299 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396308 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396317 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396326 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396340 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396349 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396358 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396367 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.crt
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396376 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396385 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396394 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396403 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres_der.key
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396412 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396421 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396430 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396439 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396448 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396457 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres_der.key
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396466 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396475 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396501 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396510 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396519 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:29:37 UTC] USER=www-data EUID=0 PID=1396528 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396537 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396546 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396555 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396564 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396574 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396587 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396596 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres_der.key
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396606 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396616 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396625 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396634 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396643 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396652 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396661 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396670 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396679 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396690 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396711 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres_der.key
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396721 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396731 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396740 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396749 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396758 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396767 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396776 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396785 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396794 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396803 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:29:38 UTC] USER=www-data EUID=0 PID=1396812 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 04:29:39 UTC] USER=www-data EUID=0 PID=1396821 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres_der.key
[2026-01-02 04:29:39 UTC] USER=www-data EUID=0 PID=1396831 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:39 UTC] USER=www-data EUID=0 PID=1396841 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:39 UTC] USER=www-data EUID=0 PID=1396850 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:29:39 UTC] USER=www-data EUID=0 PID=1396859 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:29:39 UTC] USER=www-data EUID=0 PID=1396869 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:39 UTC] USER=www-data EUID=0 PID=1396878 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:39 UTC] USER=www-data EUID=0 PID=1396887 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key
[2026-01-02 04:29:39 UTC] USER=www-data EUID=0 PID=1396896 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt
[2026-01-02 04:29:39 UTC] USER=www-data EUID=0 PID=1396905 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:39 UTC] USER=www-data EUID=0 PID=1396914 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:29:39 UTC] USER=www-data EUID=0 PID=1396923 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 04:29:39 UTC] USER=www-data EUID=0 PID=1396932 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres_der.key
[2026-01-02 04:29:39 UTC] USER=www-data EUID=0 PID=1396942 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
🎉 All requested users processed.
ℹ️  Kafka certificates not found at /opt/kafka/secrets/iam-identity-universe-main-dev/coordinator/pem (Kafka may not be installed yet)

✅ Client certificate generated successfully!

Environment: iam-identity-universe-main-dev
User: postgres
Node: worker-01
FQDN: db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
Environment: iam-identity-universe-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:29:39 UTC] USER=www-data EUID=0 PID=1396984 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1396993 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397002 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397011 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397020 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397034 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397043 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397052 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397062 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.crt
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397071 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397080 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397089 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397098 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres_der.key
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397107 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397116 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397125 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres_der.key
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397134 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397143 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397152 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397161 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397171 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres_der.key
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397180 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:40 UTC] USER=www-data EUID=0 PID=1397189 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397216 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397225 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397234 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397243 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397252 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397261 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397270 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397280 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397289 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397298 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397307 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres_der.key
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397317 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397327 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397336 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397345 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397354 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397363 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397373 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397382 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397391 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397400 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397409 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397418 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres_der.key
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397428 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397438 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397447 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397456 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397465 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397474 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397483 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key
[2026-01-02 04:29:41 UTC] USER=www-data EUID=0 PID=1397492 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt
[2026-01-02 04:29:42 UTC] USER=www-data EUID=0 PID=1397501 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:42 UTC] USER=www-data EUID=0 PID=1397511 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:29:42 UTC] USER=www-data EUID=0 PID=1397520 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 04:29:42 UTC] USER=www-data EUID=0 PID=1397529 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres_der.key
[2026-01-02 04:29:42 UTC] USER=www-data EUID=0 PID=1397539 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:42 UTC] USER=www-data EUID=0 PID=1397555 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:42 UTC] USER=www-data EUID=0 PID=1397564 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:29:42 UTC] USER=www-data EUID=0 PID=1397573 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:29:42 UTC] USER=www-data EUID=0 PID=1397582 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:42 UTC] USER=www-data EUID=0 PID=1397591 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:42 UTC] USER=www-data EUID=0 PID=1397600 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key
[2026-01-02 04:29:42 UTC] USER=www-data EUID=0 PID=1397609 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt
[2026-01-02 04:29:42 UTC] USER=www-data EUID=0 PID=1397618 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:42 UTC] USER=www-data EUID=0 PID=1397627 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:29:42 UTC] USER=www-data EUID=0 PID=1397636 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key.pkcs1
[2026-01-02 04:29:42 UTC] USER=www-data EUID=0 PID=1397645 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres_der.key
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
🎉 All requested users processed.
ℹ️  Kafka certificates not found at /opt/kafka/secrets/iam-identity-universe-main-dev/coordinator/pem (Kafka may not be installed yet)

✅ Client certificate generated successfully!

Environment: iam-identity-universe-main-dev
User: postgres
Node: worker-01
FQDN: db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-postgresql environment: db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com (10.100.1.190)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.190
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: worker-01
[INFO] Data dir:   /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01
[INFO] Port:       5432
[INFO] Hostname:   db-iam-identity-universe-main-dev-postgresql-worker-01
[2026-01-02 04:29:44 UTC] USER=www-data EUID=0 PID=1397764 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:44 UTC] USER=www-data EUID=0 PID=1397787 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:44 UTC] USER=www-data EUID=0 PID=1397811 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:44 UTC] USER=www-data EUID=0 PID=1397832 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  iam-identity-universe-main-dev.fastorder.com
  Alt CN:      iam-identity-universe-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 04:29:45 UTC] USER=www-data EUID=0 PID=1397873 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01 and /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:45 UTC] USER=www-data EUID=0 PID=1397882 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-02 04:29:45 UTC] USER=www-data EUID=0 PID=1397892 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1397839
[2026-01-02 04:29:45 UTC] USER=www-data EUID=0 PID=1397901 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1397839/ra_root.crt
[2026-01-02 04:29:45 UTC] USER=www-data EUID=0 PID=1397910 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1397839/ra_root.key
[2026-01-02 04:29:45 UTC] USER=www-data EUID=0 PID=1397919 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1397839/ra_root.crt
[2026-01-02 04:29:45 UTC] USER=www-data EUID=0 PID=1397928 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1397839/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = iam-identity-universe-main-dev.fastorder.com
[2026-01-02 04:29:46 UTC] USER=www-data EUID=0 PID=1397975 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1397839/server.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.key
[2026-01-02 04:29:46 UTC] USER=www-data EUID=0 PID=1397985 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1397839/server.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.crt
[2026-01-02 04:29:46 UTC] USER=www-data EUID=0 PID=1397994 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 04:29:47 UTC] USER=www-data EUID=0 PID=1398004 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1397839/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:47 UTC] USER=www-data EUID=0 PID=1398013 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:47 UTC] USER=www-data EUID=0 PID=1398022 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:29:47 UTC] USER=www-data EUID=0 PID=1398036 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 04:29:47 UTC] USER=www-data EUID=0 PID=1398047 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.key
[2026-01-02 04:29:47 UTC] USER=www-data EUID=0 PID=1398056 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.key
[2026-01-02 04:29:47 UTC] USER=www-data EUID=0 PID=1398083 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:29:47 UTC] USER=www-data EUID=0 PID=1398092 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = iam-identity-universe-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:iam-identity-universe-main-dev.fastorder.com, DNS:iam-identity-universe-main-dev.fastorder.com, DNS:db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com, DNS:db-iam-identity-universe-main-dev-postgresql-worker-01, DNS:localhost, DNS:db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: iam-identity-universe-main-dev
Node:        worker-01
Primary CN:  iam-identity-universe-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@iam-identity-universe-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=iam-identity-universe-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-02 04:29:47 UTC] USER=www-data EUID=0 PID=1398121 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.crt
[2026-01-02 04:29:47 UTC] USER=www-data EUID=0 PID=1398130 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.key
[2026-01-02 04:29:47 UTC] USER=www-data EUID=0 PID=1398140 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-02 04:29:47 UTC] USER=www-data EUID=0 PID=1398165 ACTION=passthru ARGS=systemctl stop postgresql@iam-identity-universe-main-dev-worker-01.service
[2026-01-02 04:29:47 UTC] USER=www-data EUID=0 PID=1398186 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-iam-identity-universe-main-dev-worker-01
[2026-01-02 04:29:47 UTC] USER=www-data EUID=0 PID=1398217 ACTION=fsop ARGS=rm -rf /var/run/postgresql-iam-identity-universe-main-dev-worker-01
[OK]   No conflicting Postgres left on port 5432
[OK]   Generated new postgres password for initdb
[2026-01-02 04:30:13 UTC] USER=www-data EUID=0 PID=1403504 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.Ig2fl5
[2026-01-02 04:30:13 UTC] USER=www-data EUID=0 PID=1403525 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.Ig2fl5
[2026-01-02 04:30:13 UTC] USER=www-data EUID=0 PID=1403547 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/iam-identity-universe-main-dev
[2026-01-02 04:30:13 UTC] USER=www-data EUID=0 PID=1403569 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/iam-identity-universe-main-dev
[2026-01-02 04:30:13 UTC] USER=www-data EUID=0 PID=1403591 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/iam-identity-universe-main-dev
[INFO] Initializing cluster in /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01 (SCRAM; pwfile)
[2026-01-02 04:30:13 UTC] USER=www-data EUID=0 PID=1403613 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:30:13 UTC] USER=www-data EUID=0 PID=1403634 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:30:13 UTC] USER=www-data EUID=0 PID=1403655 ACTION=fsop ARGS=chmod 700 /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:30:13 UTC] USER=www-data EUID=0 PID=1403676 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-iam-identity-universe-main-dev-worker-01
[2026-01-02 04:30:13 UTC] USER=www-data EUID=0 PID=1403697 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-iam-identity-universe-main-dev-worker-01
[2026-01-02 04:30:13 UTC] USER=www-data EUID=0 PID=1403718 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-iam-identity-universe-main-dev-worker-01
[2026-01-02 04:30:13 UTC] USER=www-data EUID=0 PID=1403729 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01 --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.Ig2fl5
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01 -l logfile start

[OK]   initdb complete
[2026-01-02 04:30:15 UTC] USER=www-data EUID=0 PID=1403764 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.Ig2fl5
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-02 04:30:15 UTC] USER=www-data EUID=0 PID=1403813 ACTION=fsop ARGS=cp /tmp/tmp.yPtCT83ZZp /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01/pg_hba.conf
[2026-01-02 04:30:15 UTC] USER=www-data EUID=0 PID=1403834 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01/pg_hba.conf
[2026-01-02 04:30:15 UTC] USER=www-data EUID=0 PID=1403855 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@iam-identity-universe-main-dev-worker-01.service
[2026-01-02 04:30:15 UTC] USER=www-data EUID=0 PID=1403880 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.DSOkL3 /etc/systemd/system/postgresql@iam-identity-universe-main-dev-worker-01.service
[2026-01-02 04:30:15 UTC] USER=www-data EUID=0 PID=1403901 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@iam-identity-universe-main-dev-worker-01.service
[OK]   systemd unit written
[2026-01-02 04:30:15 UTC] USER=www-data EUID=0 PID=1403923 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 04:30:15 UTC] USER=www-data EUID=0 PID=1403945 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 04:30:16 UTC] USER=www-data EUID=0 PID=1403966 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-02 04:30:17 UTC] USER=www-data EUID=0 PID=1404090 ACTION=passthru ARGS=systemctl start postgresql@iam-identity-universe-main-dev-worker-01.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-02 04:30:18 UTC] USER=www-data EUID=0 PID=1404139 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@iam-identity-universe-main-dev-worker-01.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Provisioning application database and Debezium role (if not exists)...
[INFO] Checking if database fastorder_iam_identity_universe_main_dev_db exists...
[INFO] DB check result: exit_code=0, output='[2026-01-02 04:30:19 UTC] USER=www-data EUID=0 PID=1404308 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_iam_identity_universe_main_dev_db''
[INFO] Creating database fastorder_iam_identity_universe_main_dev_db...
[2026-01-02 04:30:19 UTC] USER=www-data EUID=0 PID=1404333 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_iam_identity_universe_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[OK]   Database fastorder_iam_identity_universe_main_dev_db created
[INFO] Checking if role debezium_user exists...
[INFO] Role check result: exit_code=0, output='[2026-01-02 04:30:19 UTC] USER=www-data EUID=0 PID=1404357 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[INFO] Creating role debezium_user...
[2026-01-02 04:30:19 UTC] USER=www-data EUID=0 PID=1404390 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'N4I3CFrlol1AWzXbRG8isKGc';
CREATE ROLE
[OK]   Role debezium_user created
[2026-01-02 04:30:19 UTC] USER=www-data EUID=0 PID=1404413 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_iam_identity_universe_main_dev_db" TO debezium_user;
GRANT
[OK]   Application DB (fastorder_iam_identity_universe_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[INFO] Applying connection and memory optimizations...
[INFO] Current settings: max_connections=100, work_mem=4MB
[INFO] Target settings (worker): max_connections=100, work_mem=8MB
[2026-01-02 04:30:20 UTC] USER=www-data EUID=0 PID=1404495 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 100;
ALTER SYSTEM
[2026-01-02 04:30:20 UTC] USER=www-data EUID=0 PID=1404519 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-02 04:30:20 UTC] USER=www-data EUID=0 PID=1404543 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   Settings applied to postgresql.auto.conf
[2026-01-02 04:30:20 UTC] USER=www-data EUID=0 PID=1404558 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01/standby.signal
[INFO] Service recently started (3s ago) - restarting to apply max_connections...
[INFO] Stopping service...
[2026-01-02 04:30:21 UTC] USER=www-data EUID=0 PID=1404581 ACTION=passthru ARGS=systemctl stop postgresql@iam-identity-universe-main-dev-worker-01.service
[INFO] Waiting for port 5432 to be released...
[OK]   Port 5432 released
[INFO] Starting service...
[2026-01-02 04:30:24 UTC] USER=www-data EUID=0 PID=1404661 ACTION=passthru ARGS=systemctl start postgresql@iam-identity-universe-main-dev-worker-01.service
[OK]   ✅ Optimization complete: max_connections=100, work_mem=8MB
[OK]   Synchronous replication already configured (synchronous_commit: on)
[INFO] Setting postgres password via centralized script... for worker-01
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    iam-identity
  Zone:       universe
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01

Connection Info:
  Socket Dir: /var/run/postgresql-iam-identity-universe-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Initial setup: Using password from initdb
✓ PostgreSQL password already set during initdb
Storing password in AWS Secrets Manager: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01
✅ Secret created: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01
✅ PostgreSQL credentials set in vault: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
[OK]   Password set and persisted
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.190
[INFO] Primary hostname: db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01...
[INFO]   db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.190

[INFO]   ✅ db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.190    db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com


[OK]   PostgreSQL 'iam-identity-universe-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.key \
        host=db-iam-identity-universe-main-dev-postgresql-worker-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        iam-identity-universe-main-dev-postgresql-worker-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.190
[INFO]   Port:              5432
[INFO]   FQDN:              db-iam-identity-universe-main-dev-postgresql-worker-01
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 61988be3-ee40-48fd-9f9f-4fc4d252662a
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 📦 03 role...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[2026-01-02 04:30:41 UTC] USER=www-data EUID=0 PID=1405219 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
Environment: iam-identity-universe-main-dev
Username:    debezium_user
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   debezium_user
  Hostname:    db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:31:04 UTC] USER=www-data EUID=0 PID=1405424 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-debezium_user
[2026-01-02 04:31:04 UTC] USER=www-data EUID=0 PID=1405433 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-01-02 04:31:04 UTC] USER=www-data EUID=0 PID=1405442 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-debezium_user/ra_root.key
[2026-01-02 04:31:04 UTC] USER=www-data EUID=0 PID=1405451 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-01-02 04:31:04 UTC] USER=www-data EUID=0 PID=1405460 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405482 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405491 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405500 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.key
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405509 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.crt
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405518 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405527 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405536 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405551 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user_der.key
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405560 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.key
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405569 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405578 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405588 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.key
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405597 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405607 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user_der.key
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405616 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405625 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405651 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405660 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405669 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405678 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405687 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405697 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.key
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405706 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.crt
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405715 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405725 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:31:05 UTC] USER=www-data EUID=0 PID=1405734 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405743 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user_der.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user_der.key
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405753 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405763 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405772 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405781 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405790 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405799 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405808 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.key
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405817 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.crt
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405826 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405835 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405844 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405853 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user_der.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user_der.key
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405863 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405873 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405882 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405891 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405900 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405909 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405918 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.key
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405927 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.crt
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405936 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405945 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405954 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405963 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user_der.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user_der.key
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405973 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405983 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1405992 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1406001 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1406010 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1406019 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1406028 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.key
[2026-01-02 04:31:06 UTC] USER=www-data EUID=0 PID=1406037 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.crt
[2026-01-02 04:31:07 UTC] USER=www-data EUID=0 PID=1406046 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:07 UTC] USER=www-data EUID=0 PID=1406055 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:31:07 UTC] USER=www-data EUID=0 PID=1406064 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-02 04:31:07 UTC] USER=www-data EUID=0 PID=1406073 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/debezium_user_der.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user_der.key
[2026-01-02 04:31:07 UTC] USER=www-data EUID=0 PID=1406083 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
🎉 All requested users processed.
ℹ️  Kafka certificates not found at /opt/kafka/secrets/iam-identity-universe-main-dev/coordinator/pem (Kafka may not be installed yet)

✅ Client certificate generated successfully!

Environment: iam-identity-universe-main-dev
User: debezium_user
Node: worker-01
FQDN: db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com -U debezium_user -d postgres

🔐 Generating replicator client certificate for worker-01...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
Environment: iam-identity-universe-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:31:07 UTC] USER=www-data EUID=0 PID=1406124 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-02 04:31:07 UTC] USER=www-data EUID=0 PID=1406133 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 04:31:07 UTC] USER=www-data EUID=0 PID=1406142 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-02 04:31:07 UTC] USER=www-data EUID=0 PID=1406151 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 04:31:07 UTC] USER=www-data EUID=0 PID=1406160 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:08 UTC] USER=www-data EUID=0 PID=1406174 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:08 UTC] USER=www-data EUID=0 PID=1406183 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:08 UTC] USER=www-data EUID=0 PID=1406194 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:31:08 UTC] USER=www-data EUID=0 PID=1406203 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt
[2026-01-02 04:31:08 UTC] USER=www-data EUID=0 PID=1406212 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:08 UTC] USER=www-data EUID=0 PID=1406221 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406230 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406239 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406248 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406257 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406266 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406275 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406284 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406293 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406302 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406311 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406320 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406329 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406355 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406364 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406373 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406382 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406391 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406400 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406409 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406418 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406427 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406436 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406445 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406455 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406465 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406474 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406483 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406492 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406501 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406510 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:31:09 UTC] USER=www-data EUID=0 PID=1406519 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406528 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406537 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406546 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406555 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406565 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406575 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406584 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406593 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406602 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406611 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406620 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406629 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406638 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406647 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406656 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406665 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406675 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406685 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406694 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406703 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406712 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406721 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406730 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406739 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406748 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406757 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406766 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406775 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:31:10 UTC] USER=www-data EUID=0 PID=1406785 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
🎉 All requested users processed.
ℹ️  Kafka certificates not found at /opt/kafka/secrets/iam-identity-universe-main-dev/coordinator/pem (Kafka may not be installed yet)

✅ Client certificate generated successfully!

Environment: iam-identity-universe-main-dev
User: replicator
Node: worker-01
FQDN: db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

✅ Replicator certificate generated for worker-01
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    iam-identity
  Zone:       universe
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/fastorder_admin_gd

Connection Info:
  Socket Dir: /var/run/postgresql-iam-identity-universe-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: fastorder_admin_gd
Storing password in AWS Secrets Manager: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Secret created: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: worker-01/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
Environment: iam-identity-universe-main-dev
Username:    fastorder_admin_gd
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   fastorder_admin_gd
  Hostname:    db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:31:24 UTC] USER=www-data EUID=0 PID=1407144 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-fastorder_admin_gd
[2026-01-02 04:31:24 UTC] USER=www-data EUID=0 PID=1407153 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-01-02 04:31:24 UTC] USER=www-data EUID=0 PID=1407162 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
[2026-01-02 04:31:24 UTC] USER=www-data EUID=0 PID=1407171 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-01-02 04:31:24 UTC] USER=www-data EUID=0 PID=1407180 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407195 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407204 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407213 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407222 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407231 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407240 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407249 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407258 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407267 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407276 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407285 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407294 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407303 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407312 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407321 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407330 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407339 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407348 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407374 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407383 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:25 UTC] USER=www-data EUID=0 PID=1407392 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407401 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407410 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407419 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407428 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407437 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407446 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407455 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407464 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407474 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407484 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407493 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407502 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407511 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407520 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407529 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407538 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407547 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407556 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407565 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407574 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407584 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407594 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407603 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407612 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407621 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407630 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407639 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407648 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407657 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407666 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407675 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 04:31:26 UTC] USER=www-data EUID=0 PID=1407684 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 04:31:27 UTC] USER=www-data EUID=0 PID=1407694 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:27 UTC] USER=www-data EUID=0 PID=1407704 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:27 UTC] USER=www-data EUID=0 PID=1407713 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:27 UTC] USER=www-data EUID=0 PID=1407722 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:31:27 UTC] USER=www-data EUID=0 PID=1407731 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:27 UTC] USER=www-data EUID=0 PID=1407740 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:31:27 UTC] USER=www-data EUID=0 PID=1407749 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-02 04:31:27 UTC] USER=www-data EUID=0 PID=1407758 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-02 04:31:27 UTC] USER=www-data EUID=0 PID=1407767 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:31:27 UTC] USER=www-data EUID=0 PID=1407776 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:31:27 UTC] USER=www-data EUID=0 PID=1407785 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-02 04:31:27 UTC] USER=www-data EUID=0 PID=1407794 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-02 04:31:27 UTC] USER=www-data EUID=0 PID=1407804 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
🎉 All requested users processed.
ℹ️  Kafka certificates not found at /opt/kafka/secrets/iam-identity-universe-main-dev/coordinator/pem (Kafka may not be installed yet)

✅ Client certificate generated successfully!

Environment: iam-identity-universe-main-dev
User: fastorder_admin_gd
Node: worker-01
FQDN: db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-iam-identity-universe-main-dev-worker-01:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_iam_identity_universe_main_dev_db already exists, skipping creation
[2026-01-02 04:31:27 UTC] USER=www-data EUID=0 PID=1407862 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-iam-identity-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-02 04:31:27 UTC] USER=www-data EUID=0 PID=1407896 ACTION=passthru ARGS=systemctl reload postgresql@iam-identity-universe-main-dev-worker-01.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

=== Pre-flight Checks ===
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
✓ AWS Secrets Manager accessible

=== Retrieving Credentials from AWS ===
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
/opt/fastorder/bash/infra_core/cache.sh: line 145: /var/cache/secrets/fastorder_db_iam-identity_universe_main_dev_postgresql_worker-01_fastorder_admin_gd.cache.tmp.1407905: Permission denied
✅ Retrieved from secrets manager: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials loaded for worker-01/fastorder_admin_gd: fastorder_admin_gd@db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_iam_identity_universe_main_dev_db
✓ Credentials retrieved: fastorder_admin_gd@db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_iam_identity_universe_main_dev_db
╔════════════════════════════════════════════╗
║  PostgreSQL Test Suite (AWS Secrets MGR)  ║
╚════════════════════════════════════════════╝

=== PostgreSQL Authentication Test ===
✗ PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com" (10.100.1.190), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[2026-01-02 04:31:34 UTC] USER=www-data EUID=0 PID=1408068 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : iam-identity-universe-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_iam_identity_universe_main_dev_db
  SCHEMA      : auth
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
  DNS → 10.100.1.190
  CA         : /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    iam-identity
  Zone:       universe
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/debezium_user

Connection Info:
  Socket Dir: /var/run/postgresql-iam-identity-universe-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User debezium_user does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: debezium_user
Storing password in AWS Secrets Manager: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/debezium_user
✅ Secret created: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/debezium_user
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/iam-identity/universe/main/dev/postgresql/worker-01/debezium_user
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: worker-01/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-01-02 04:31:47 UTC] USER=www-data EUID=0 PID=1408449 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-iam-identity-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_iam_identity_universe_main_dev_db already exists
[2026-01-02 04:31:47 UTC] USER=www-data EUID=0 PID=1408475 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-iam-identity-universe-main-dev-worker-01 -p 5432 -d fastorder_iam_identity_universe_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
🔐 Creating replicator role for worker-01...
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : iam-identity-universe-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SSL DIR     : /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
  DNS → 10.100.1.190
  CA         : /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
🔑 Generating new secure replicator password...
💾 Storing replicator password in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/db/iam-identity/universe/main/dev/postgresql/replicator-eVKJSS",
    "Name": "fastorder/db/iam-identity/universe/main/dev/postgresql/replicator",
    "VersionId": "e7111cc1-9a4f-4708-84f6-b5a6983f2126"
}
✅ Password stored in AWS Secrets Manager: fastorder/db/iam-identity/universe/main/dev/postgresql/replicator
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Creating role: replicator with password
SET
CREATE ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/iam-identity/universe/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🎉 Done.
✅ Replicator role created for worker-01

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (iam-identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=worker-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║                    IAM Database Schema Initialization                       ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] 🟢 Starting IAM schema provisioning...
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

[INFO] 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: core
  Core Identity Directory (tenants, realms, identities, devices, MFA)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [1/20]: core/01-tenant
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Identifier:  coordinator
  Database:    fastorder_iam_identity_universe_main_dev_db
  Host:        db-iam-identity-universe-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_iam_identity_universe_main_dev_db
ℹ️  Database fastorder_iam_identity_universe_main_dev_db already exists
✅ Connected to database: fastorder_iam_identity_universe_main_dev_db
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "pgcrypto" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "citext" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
NOTICE:  schema "utils" already exists, skipping
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
NOTICE:  schema "core" already exists, skipping
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
NOTICE:  relation "tenant" already exists, skipping
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[OK] Table core/01-tenant initialized

[INFO] 🔸 Table [2/20]: core/02-realm
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  relation "realm" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_realm_keycloak_id" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_realm_tenant" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[OK] Table core/02-realm initialized

[INFO] 🔸 Table [3/20]: core/03-identity
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  relation "identity" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_unique_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_unique_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_type" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[OK] Table core/03-identity initialized

[INFO] 🔸 Table [4/20]: core/04-device
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  relation "device" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_device_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_fingerprint" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_trusted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_last_seen" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[OK] Table core/04-device initialized

[INFO] 🔸 Table [5/20]: core/05-identity_account
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  relation "identity_account" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_account_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_lockout" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_last_login" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[OK] Table core/05-identity_account initialized

[INFO] 🔸 Table [6/20]: core/06-identity_mfa
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  relation "identity_mfa" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_mfa_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_active" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[OK] Table core/06-identity_mfa initialized

[INFO] 🔸 Table [7/20]: core/07-external_idp_link
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  relation "external_idp_link" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_external_idp_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_provider" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_email" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[OK] Table core/07-external_idp_link initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: policy
  RBAC/ABAC Authorization (clients, roles, permissions, policies)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [8/20]: policy/01-client
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
NOTICE:  schema "policy" already exists, skipping
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  relation "client" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_client_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_key" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_status" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[OK] Table policy/01-client initialized

[INFO] 🔸 Table [9/20]: policy/02-resource
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  relation "resource" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_resource_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_external" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_owner" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[OK] Table policy/02-resource initialized

[INFO] 🔸 Table [10/20]: policy/03-scope
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  relation "scope" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_scope_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_scope_name" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[OK] Table policy/03-scope initialized

[INFO] 🔸 Table [11/20]: policy/04-permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  relation "permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_permission_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_resource" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[OK] Table policy/04-permission initialized

[INFO] 🔸 Table [12/20]: policy/05-role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  relation "role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_keycloak" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[OK] Table policy/05-role initialized

[INFO] 🔸 Table [13/20]: policy/06-role_permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  relation "role_permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_permission_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_permission_perm" already exists, skipping
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[OK] Table policy/06-role_permission initialized

[INFO] 🔸 Table [14/20]: policy/07-identity_role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  relation "identity_role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_role_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_active" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[OK] Table policy/07-identity_role initialized

[INFO] 🔸 Table [15/20]: policy/08-policy_rule
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  relation "policy_rule" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_policy_rule_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_enabled" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_priority" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[OK] Table policy/08-policy_rule initialized

[INFO] 🔸 Table [16/20]: policy/09-api_key
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  relation "api_key" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_api_key_prefix" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[OK] Table policy/09-api_key initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: audit
  Audit & Risk Logging (auth events, admin actions, risk decisions)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [17/20]: audit/01-auth_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
NOTICE:  schema "audit" already exists, skipping
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
NOTICE:  relation "auth_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_auth_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_result" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_ip" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_session" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_trace" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_risk" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[OK] Table audit/01-auth_event initialized

[INFO] 🔸 Table [18/20]: audit/02-admin_action
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
NOTICE:  relation "admin_action" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_admin_action_actor" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_target" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_trace" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[OK] Table audit/02-admin_action initialized

[INFO] 🔸 Table [19/20]: audit/03-risk_decision
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
NOTICE:  relation "risk_decision" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_risk_decision_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_level" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_decision" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_auth" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[OK] Table audit/03-risk_decision initialized

[INFO] 🔸 Table [20/20]: audit/04-consent_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
NOTICE:  relation "consent_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_consent_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_version" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_granted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  relation "audit.auth_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  relation "audit.auth_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  relation "audit.admin_action_2026_03" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  relation "audit.admin_action_2026_04" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  relation "audit.risk_decision_2026_03" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  relation "audit.risk_decision_2026_04" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  relation "audit.consent_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  relation "audit.consent_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[OK] Table audit/04-consent_event initialized


════════════════════════════════════════════════════════════════════════════
[OK] ✅ IAM Schema Initialization Complete!
[OK] All 20 tables initialized successfully

Schemas created:
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

Design highlights:
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

════════════════════════════════════════════════════════════════════════════

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=worker-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for iam-identity in universe-dev...
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: iam-identity/run.sh started for SERVICE=iam-identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../iam-identity
🔍 DEBUG_CHECKPOINT_A3_FAIL: SERVICE_ROOT does not exist!
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: NONE
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
🔍 DEBUG_CHECKPOINT_A6_FAIL: No table folders found!
[WARN] No grouped table folders found under: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../iam-identity
End of 04-postgresql/steps/01-install/steps/iam-identity/run.sh

✓ ✅ Worker worker-01 setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up standby replicas (1 per worker)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up standby: worker-01-standby-01 (replica of worker-01)
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-02 04:33:11 UTC] USER=unknown EUID=33 PID=1411203 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 04:33:11 UTC] USER=unknown EUID=33 PID=1411210 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 04:33:11 UTC] USER=unknown EUID=33 PID=1411217 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-02 04:33:11 UTC] USER=unknown EUID=33 PID=1411224 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 04:33:11 UTC] USER=unknown EUID=33 PID=1411231 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-02 04:33:11 UTC] USER=unknown EUID=33 PID=1411238 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for iam-identity in universe-dev...
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67 JOB_UUID=85222a65-779a-4529-84e6-94639bcc9bed

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[WARN] Could not find PostgreSQL IP for worker-01-standby-01 in topology.json, allocating new VM IP...
[INFO] Allocated new VM IP: 10.100.1.191 for db-worker-01-standby-01-postgresql
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] PostgreSQL IP: 10.100.1.191
[INFO] Primary hostname: db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01-standby-01...
[INFO]   db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.191

[INFO]   ➕ Adding db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.191
✅     ✅ Added: db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.191

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.191    db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  iam-identity-universe-main-dev.fastorder.com
  Alt CN:      iam-identity-universe-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 04:33:15 UTC] USER=www-data EUID=0 PID=1411677 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:15 UTC] USER=www-data EUID=0 PID=1411686 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-02 04:33:15 UTC] USER=www-data EUID=0 PID=1411696 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1411644
[2026-01-02 04:33:15 UTC] USER=www-data EUID=0 PID=1411705 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1411644/ra_root.crt
[2026-01-02 04:33:15 UTC] USER=www-data EUID=0 PID=1411714 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1411644/ra_root.key
[2026-01-02 04:33:15 UTC] USER=www-data EUID=0 PID=1411723 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1411644/ra_root.crt
[2026-01-02 04:33:15 UTC] USER=www-data EUID=0 PID=1411732 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1411644/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = iam-identity-universe-main-dev.fastorder.com
[2026-01-02 04:33:20 UTC] USER=www-data EUID=0 PID=1411769 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1411644/server.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.key
[2026-01-02 04:33:20 UTC] USER=www-data EUID=0 PID=1411778 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1411644/server.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt
[2026-01-02 04:33:20 UTC] USER=www-data EUID=0 PID=1411787 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 04:33:20 UTC] USER=www-data EUID=0 PID=1411796 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1411644/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:20 UTC] USER=www-data EUID=0 PID=1411805 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:20 UTC] USER=www-data EUID=0 PID=1411814 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:20 UTC] USER=www-data EUID=0 PID=1411823 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 04:33:20 UTC] USER=www-data EUID=0 PID=1411834 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.key
[2026-01-02 04:33:20 UTC] USER=www-data EUID=0 PID=1411843 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.key
[2026-01-02 04:33:20 UTC] USER=www-data EUID=0 PID=1411853 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt
[2026-01-02 04:33:20 UTC] USER=www-data EUID=0 PID=1411862 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt
[2026-01-02 04:33:20 UTC] USER=www-data EUID=0 PID=1411871 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:20 UTC] USER=www-data EUID=0 PID=1411880 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = iam-identity-universe-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:iam-identity-universe-main-dev.fastorder.com, DNS:iam-identity-universe-main-dev.fastorder.com, DNS:db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: iam-identity-universe-main-dev
Node:        worker-01-standby-01
Primary CN:  iam-identity-universe-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@iam-identity-universe-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=iam-identity-universe-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
Environment: iam-identity-universe-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:33:21 UTC] USER=www-data EUID=0 PID=1411934 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-01-02 04:33:21 UTC] USER=www-data EUID=0 PID=1411943 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 04:33:21 UTC] USER=www-data EUID=0 PID=1411952 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
[2026-01-02 04:33:21 UTC] USER=www-data EUID=0 PID=1411961 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 04:33:21 UTC] USER=www-data EUID=0 PID=1411970 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:21 UTC] USER=www-data EUID=0 PID=1411984 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1411993 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412002 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412011 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412020 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412029 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412038 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412047 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412056 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412065 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412074 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412083 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412092 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412101 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412110 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412119 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412145 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412154 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412163 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412172 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412181 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412190 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412199 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412208 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412217 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412226 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412242 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 04:33:22 UTC] USER=www-data EUID=0 PID=1412256 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412267 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412276 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412285 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412294 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412303 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412312 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412321 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412330 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412339 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412348 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412358 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412374 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412384 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412393 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412402 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412411 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412420 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412433 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412442 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412451 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412460 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412469 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412478 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412488 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412498 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412507 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412516 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412525 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412534 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412543 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 04:33:23 UTC] USER=www-data EUID=0 PID=1412552 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 04:33:24 UTC] USER=www-data EUID=0 PID=1412561 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:24 UTC] USER=www-data EUID=0 PID=1412570 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 04:33:24 UTC] USER=www-data EUID=0 PID=1412579 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 04:33:24 UTC] USER=www-data EUID=0 PID=1412588 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 04:33:24 UTC] USER=www-data EUID=0 PID=1412598 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
🎉 All requested users processed.
ℹ️  Kafka certificates not found at /opt/kafka/secrets/iam-identity-universe-main-dev/coordinator/pem (Kafka may not be installed yet)

✅ Client certificate generated successfully!

Environment: iam-identity-universe-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
Environment: iam-identity-universe-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:33:24 UTC] USER=www-data EUID=0 PID=1412639 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-01-02 04:33:24 UTC] USER=www-data EUID=0 PID=1412648 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 04:33:24 UTC] USER=www-data EUID=0 PID=1412657 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
[2026-01-02 04:33:24 UTC] USER=www-data EUID=0 PID=1412666 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-02 04:33:24 UTC] USER=www-data EUID=0 PID=1412675 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412689 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412698 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412707 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412716 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412725 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412734 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412743 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412752 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412761 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412770 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412779 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412788 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412799 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412810 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412819 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412828 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412837 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412846 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412872 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412881 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412890 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412899 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412908 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412917 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412926 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412936 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412945 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412955 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412964 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 04:33:25 UTC] USER=www-data EUID=0 PID=1412974 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1412984 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1412993 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413002 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413011 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413020 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413029 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413038 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413047 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413056 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413065 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413074 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413084 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413094 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413103 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413112 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413121 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413130 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413139 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413148 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413157 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413166 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413175 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413184 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413194 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413204 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413213 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413222 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413231 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413240 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413249 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413258 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413267 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413276 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413285 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-02 04:33:26 UTC] USER=www-data EUID=0 PID=1413294 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-02 04:33:27 UTC] USER=www-data EUID=0 PID=1413304 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
🎉 All requested users processed.
ℹ️  Kafka certificates not found at /opt/kafka/secrets/iam-identity-universe-main-dev/coordinator/pem (Kafka may not be installed yet)

✅ Client certificate generated successfully!

Environment: iam-identity-universe-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
Environment: iam-identity-universe-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:33:27 UTC] USER=www-data EUID=0 PID=1413345 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-02 04:33:27 UTC] USER=www-data EUID=0 PID=1413354 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 04:33:27 UTC] USER=www-data EUID=0 PID=1413363 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-02 04:33:27 UTC] USER=www-data EUID=0 PID=1413372 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 04:33:27 UTC] USER=www-data EUID=0 PID=1413381 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:27 UTC] USER=www-data EUID=0 PID=1413395 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:27 UTC] USER=www-data EUID=0 PID=1413404 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:27 UTC] USER=www-data EUID=0 PID=1413413 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:33:27 UTC] USER=www-data EUID=0 PID=1413422 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt
[2026-01-02 04:33:27 UTC] USER=www-data EUID=0 PID=1413431 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:33:27 UTC] USER=www-data EUID=0 PID=1413440 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413449 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413458 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413467 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413477 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413486 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413495 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413504 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413513 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413522 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413531 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413540 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413549 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413575 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413584 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413593 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413602 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413611 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413620 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413629 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413638 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413647 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413656 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413665 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413675 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413685 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413694 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413703 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413712 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413721 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413730 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413739 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413748 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413757 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:33:28 UTC] USER=www-data EUID=0 PID=1413766 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413775 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413785 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413795 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413804 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413813 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413822 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413831 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413840 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413849 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413858 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413867 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413876 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413885 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413895 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413905 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413914 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413923 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413932 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413941 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413950 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413959 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413968 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413977 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413986 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1413995 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:33:29 UTC] USER=www-data EUID=0 PID=1414005 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
🎉 All requested users processed.
ℹ️  Kafka certificates not found at /opt/kafka/secrets/iam-identity-universe-main-dev/coordinator/pem (Kafka may not be installed yet)

✅ Client certificate generated successfully!

Environment: iam-identity-universe-main-dev
User: replicator
Node: worker-01
FQDN: db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-standby-01-postgresql environment: db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com (10.100.1.191)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.191
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] Data dir:   /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01-standby-01
[INFO] Port:       5432
[INFO] Hostname:   db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01
[2026-01-02 04:33:31 UTC] USER=www-data EUID=0 PID=1414099 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:31 UTC] USER=www-data EUID=0 PID=1414120 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:31 UTC] USER=www-data EUID=0 PID=1414141 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:31 UTC] USER=www-data EUID=0 PID=1414162 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe (FastOrder Universe)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  iam-identity-universe-main-dev.fastorder.com
  Alt CN:      iam-identity-universe-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-02 04:33:32 UTC] USER=www-data EUID=0 PID=1414202 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:32 UTC] USER=www-data EUID=0 PID=1414211 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-02 04:33:32 UTC] USER=www-data EUID=0 PID=1414221 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-1414169
[2026-01-02 04:33:32 UTC] USER=www-data EUID=0 PID=1414230 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-1414169/ra_root.crt
[2026-01-02 04:33:32 UTC] USER=www-data EUID=0 PID=1414239 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-1414169/ra_root.key
[2026-01-02 04:33:32 UTC] USER=www-data EUID=0 PID=1414248 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1414169/ra_root.crt
[2026-01-02 04:33:32 UTC] USER=www-data EUID=0 PID=1414257 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-1414169/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = iam-identity-universe-main-dev.fastorder.com
[2026-01-02 04:33:34 UTC] USER=www-data EUID=0 PID=1414294 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1414169/server.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.key
[2026-01-02 04:33:34 UTC] USER=www-data EUID=0 PID=1414303 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1414169/server.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt
[2026-01-02 04:33:34 UTC] USER=www-data EUID=0 PID=1414312 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-02 04:33:34 UTC] USER=www-data EUID=0 PID=1414321 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-1414169/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:34 UTC] USER=www-data EUID=0 PID=1414330 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:35 UTC] USER=www-data EUID=0 PID=1414339 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt
[2026-01-02 04:33:35 UTC] USER=www-data EUID=0 PID=1414348 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-02 04:33:35 UTC] USER=www-data EUID=0 PID=1414359 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.key
[2026-01-02 04:33:35 UTC] USER=www-data EUID=0 PID=1414368 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.key
[2026-01-02 04:33:35 UTC] USER=www-data EUID=0 PID=1414377 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt
[2026-01-02 04:33:35 UTC] USER=www-data EUID=0 PID=1414386 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt
[2026-01-02 04:33:35 UTC] USER=www-data EUID=0 PID=1414395 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
[2026-01-02 04:33:35 UTC] USER=www-data EUID=0 PID=1414404 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = US, ST = Virginia, L = Ashburn, O = FastOrder, OU = PostgreSQL, CN = iam-identity-universe-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:iam-identity-universe-main-dev.fastorder.com, DNS:iam-identity-universe-main-dev.fastorder.com, DNS:db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: iam-identity-universe-main-dev
Node:        worker-01-standby-01
Primary CN:  iam-identity-universe-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@iam-identity-universe-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=iam-identity-universe-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-02 04:33:35 UTC] USER=www-data EUID=0 PID=1414433 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt
[2026-01-02 04:33:35 UTC] USER=www-data EUID=0 PID=1414442 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.key
[2026-01-02 04:33:35 UTC] USER=www-data EUID=0 PID=1414451 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-02 04:33:35 UTC] USER=www-data EUID=0 PID=1414472 ACTION=passthru ARGS=systemctl stop postgresql@iam-identity-universe-main-dev-worker-01-standby-01.service
[2026-01-02 04:33:35 UTC] USER=www-data EUID=0 PID=1414493 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-iam-identity-universe-main-dev-worker-01-standby-01
[2026-01-02 04:33:35 UTC] USER=www-data EUID=0 PID=1414524 ACTION=fsop ARGS=rm -rf /var/run/postgresql-iam-identity-universe-main-dev-worker-01-standby-01
[OK]   No conflicting Postgres left on port 5432
[OK]   Generated new postgres password for initdb
[2026-01-02 04:33:58 UTC] USER=www-data EUID=0 PID=1414721 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.g2A7LZ
[2026-01-02 04:33:59 UTC] USER=www-data EUID=0 PID=1414742 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.g2A7LZ
[2026-01-02 04:33:59 UTC] USER=www-data EUID=0 PID=1414764 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/iam-identity-universe-main-dev
[2026-01-02 04:33:59 UTC] USER=www-data EUID=0 PID=1414786 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/iam-identity-universe-main-dev
[2026-01-02 04:33:59 UTC] USER=www-data EUID=0 PID=1414808 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/iam-identity-universe-main-dev
[INFO] This is a standby. Using pg_basebackup from primary (worker-01)...
[INFO] Setting up replicator role and slot on primary (worker-01)...
ℹ️  Scanning primary for stuck queries from previous failed attempts...
ℹ️  Scanning for stuck queries (timeout: 30s)...
ℹ️  No stuck queries found
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : iam-identity-universe-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SLOT        : worker_01_standby_01
  SSL DIR     : /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
  DNS → 10.100.1.190
  CA         : /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Role replicator already exists, updating password and ensuring REPLICATION privilege
SET
ALTER ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/iam-identity/universe/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🔧 Ensuring replication slot: worker_01_standby_01…
🆕 Creating replication slot worker_01_standby_01
SET
 pg_create_physical_replication_slot 
-------------------------------------
 (worker_01_standby_01,)
(1 row)

✅ Replication slot worker_01_standby_01 created.
🎉 Done.
[OK]   Replicator role and slot created on primary
[INFO] Creating replicator client certificates for connecting to primary (worker-01)...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
Environment: iam-identity-universe-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:34:03 UTC] USER=www-data EUID=0 PID=1414955 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-02 04:34:03 UTC] USER=www-data EUID=0 PID=1414964 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 04:34:03 UTC] USER=www-data EUID=0 PID=1414973 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-02 04:34:03 UTC] USER=www-data EUID=0 PID=1414982 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-02 04:34:03 UTC] USER=www-data EUID=0 PID=1414991 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:03 UTC] USER=www-data EUID=0 PID=1415005 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:03 UTC] USER=www-data EUID=0 PID=1415014 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:03 UTC] USER=www-data EUID=0 PID=1415023 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:34:03 UTC] USER=www-data EUID=0 PID=1415032 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt
[2026-01-02 04:34:03 UTC] USER=www-data EUID=0 PID=1415041 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:34:03 UTC] USER=www-data EUID=0 PID=1415050 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:34:03 UTC] USER=www-data EUID=0 PID=1415059 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:34:03 UTC] USER=www-data EUID=0 PID=1415068 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:34:03 UTC] USER=www-data EUID=0 PID=1415077 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:34:03 UTC] USER=www-data EUID=0 PID=1415086 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:34:03 UTC] USER=www-data EUID=0 PID=1415095 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415104 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415113 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415122 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415131 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415140 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415149 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415158 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415184 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415193 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415202 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415211 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415220 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415229 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415238 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415247 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415256 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415265 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415274 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415284 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415294 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415303 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415312 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415321 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415336 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415345 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415354 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt
[2026-01-02 04:34:04 UTC] USER=www-data EUID=0 PID=1415364 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415374 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415383 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415394 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415404 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415414 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415423 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415433 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415443 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415454 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415465 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415474 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415484 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415493 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415503 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415512 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415522 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415532 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415541 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415550 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415559 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415568 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415577 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415586 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415595 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415604 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415614 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415624 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator_der.key
[2026-01-02 04:34:05 UTC] USER=www-data EUID=0 PID=1415634 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01 → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
🎉 All requested users processed.
ℹ️  Kafka certificates not found at /opt/kafka/secrets/iam-identity-universe-main-dev/coordinator/pem (Kafka may not be installed yet)

✅ Client certificate generated successfully!

Environment: iam-identity-universe-main-dev
User: replicator
Node: worker-01
FQDN: db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

[OK]   Replicator certificate created for worker-01 in /home/postgres/
[INFO] Using replicator certificates from primary worker-01...
[2026-01-02 04:34:06 UTC] USER=www-data EUID=0 PID=1415662 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key.pkcs1
[2026-01-02 04:34:06 UTC] USER=www-data EUID=0 PID=1415687 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.key
[2026-01-02 04:34:06 UTC] USER=www-data EUID=0 PID=1415708 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/replicator.crt
[OK]   Replicator certificates verified at /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[OK]   root.crt verified at /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01
[INFO] Updating primary pg_hba.conf to allow replication...
[INFO]   Standby IP: 10.100.1.191/32 (standby's source IP)
[INFO]   Primary application IP: 10.100.1.190/32 (for local pg_basebackup)
[INFO]   Primary DNS IP: 10.100.1.190/32 (DNS resolution of db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com)
[2026-01-02 04:34:06 UTC] USER=www-data EUID=0 PID=1415738 ACTION=passthru ARGS=grep -qxF # BEGIN standby-replication (managed) /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01/pg_hba.conf
[2026-01-02 04:34:06 UTC] USER=www-data EUID=0 PID=1415783 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.191/32  scram-sha-256 
      $0==begin {inside=1}
      inside && $0==rule {found=1}
      $0==end {inside=0}
      END {exit found?0:1}
     /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01/pg_hba.conf
[2026-01-02 04:34:06 UTC] USER=www-data EUID=0 PID=1415809 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl  replication  replicator  10.100.1.191/32  scram-sha-256 /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01/pg_hba.conf
[2026-01-02 04:34:06 UTC] USER=www-data EUID=0 PID=1415830 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.190/32  scram-sha-256 
        $0==begin {inside=1}
        inside && $0==rule {found=1}
        $0==end {inside=0}
        END {exit found?0:1}
       /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01/pg_hba.conf
[2026-01-02 04:34:06 UTC] USER=www-data EUID=0 PID=1415854 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl  replication  replicator  10.100.1.190/32  scram-sha-256 /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01/pg_hba.conf
[INFO] Reloading primary PostgreSQL service...
[2026-01-02 04:34:06 UTC] USER=www-data EUID=0 PID=1415875 ACTION=passthru ARGS=systemctl reload postgresql@iam-identity-universe-main-dev-worker-01.service
[OK]   Primary pg_hba.conf updated and service reloaded
[INFO] Primary host: db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com
[INFO] Using replicator cert: /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt
[INFO] Using replicator key: /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key (PKCS#8 format)
[INFO] Using CA cert: /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[INFO] Verifying postgres user can access certificates...
[ERR]  postgres user CANNOT read /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[INFO] File permissions:
lrwxrwxrwx 1 postgres ssl-cert 81 Jan  2 04:34 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt -> /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/root.crt
[INFO] Parent directory permissions:
drwx------ 2 postgres postgres 4096 Jan  2 04:34 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
drwx------ 6 postgres postgres 4096 Dec  8 11:34 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[WARN] Attempting to fix permissions (/usr/local/bin/fastorder-provisioning-wrapper.sh required)...
[INFO] Fixing /home/postgres/ directory...
[2026-01-02 04:34:07 UTC] USER=www-data EUID=0 PID=1415944 ACTION=fsop ARGS=chmod 755 /home/postgres/
[INFO] Fixing /home/postgres/ssl/.postgresql/...
[2026-01-02 04:34:07 UTC] USER=www-data EUID=0 PID=1415965 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/
[INFO] Fixing parent directory: /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:34:07 UTC] USER=www-data EUID=0 PID=1415988 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[INFO] Fixing certificate directory: /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[2026-01-02 04:34:07 UTC] USER=www-data EUID=0 PID=1416009 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[INFO] Fixing CA certificate: /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[2026-01-02 04:34:07 UTC] USER=www-data EUID=0 PID=1416030 ACTION=fsop ARGS=chmod 644 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[OK]   Permissions fixed
[OK]   postgres user can now read /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt after permission fix
[2026-01-02 04:34:07 UTC] USER=www-data EUID=0 PID=1416051 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-iam-identity-universe-main-dev-worker-01-standby-01
[2026-01-02 04:34:07 UTC] USER=www-data EUID=0 PID=1416072 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-iam-identity-universe-main-dev-worker-01-standby-01
[2026-01-02 04:34:07 UTC] USER=www-data EUID=0 PID=1416093 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-iam-identity-universe-main-dev-worker-01-standby-01
[INFO] Checking primary database size before pg_basebackup...
[INFO] Total primary database size: 29 MB
[INFO] Estimated transfer time: ~0 minutes (at 10MB/s with compression)
[INFO] Retrieving replicator password from AWS Secrets Manager: fastorder/db/iam-identity/universe/main/dev/postgresql/replicator
[OK]   Replicator password retrieved successfully
[INFO] Starting pg_basebackup...
[2026-01-02 04:34:09 UTC] USER=www-data EUID=0 PID=1416165 ACTION=passthru ARGS=sudo -u postgres env PGPASSWORD=7E50rwymkZFTuHNT6Tn141R9dLYRlmrJ PGSSLMODE=verify-full PGSSLCERT=/home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt PGSSLKEY=/home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key PGSSLROOTCERT=/home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt /usr/lib/postgresql/17/bin/pg_basebackup -h db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com -p 5432 -U replicator -D /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01-standby-01 -Fp -Xs -P -R --checkpoint=fast --wal-method=stream --verbose
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/2000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1416174"
30550/30550 kB (100%), 0/1 tablespace (...-01-standby-01/global/pg_control)
30550/30550 kB (100%), 1/1 tablespace                                         
pg_basebackup: write-ahead log end point: 0/2000120
pg_basebackup: waiting for background process to finish streaming ...
pg_basebackup: syncing data to disk ...
pg_basebackup: renaming backup_manifest.tmp to backup_manifest
pg_basebackup: base backup completed
[OK]   pg_basebackup complete
[INFO] Fixing postgresql.auto.conf to use IP-based primary_conninfo (matching golden backup)...
[2026-01-02 04:34:10 UTC] USER=www-data EUID=0 PID=1416178 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01-standby-01/standby.signal
[2026-01-02 04:34:10 UTC] USER=www-data EUID=0 PID=1416200 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01-standby-01/standby.signal
[2026-01-02 04:34:10 UTC] USER=www-data EUID=0 PID=1416221 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01-standby-01/standby.signal
[2026-01-02 04:34:10 UTC] USER=www-data EUID=0 PID=1416230 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01-standby-01/standby.signal
[OK]   standby.signal verified and permissions set
[INFO] Fixing postgresql.conf with standby-specific settings...
[WARN] postgresql.conf not found at /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01-standby-01/postgresql.conf
[INFO] Verifying postgresql.auto.conf...
[WARN] postgresql.auto.conf not found - pg_basebackup may have failed
[2026-01-02 04:34:10 UTC] USER=www-data EUID=0 PID=1416253 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.g2A7LZ
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-02 04:34:11 UTC] USER=www-data EUID=0 PID=1416302 ACTION=fsop ARGS=cp /tmp/tmp.5eIm2RMrWZ /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01-standby-01/pg_hba.conf
[2026-01-02 04:34:11 UTC] USER=www-data EUID=0 PID=1416323 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01-standby-01/pg_hba.conf
[2026-01-02 04:34:11 UTC] USER=www-data EUID=0 PID=1416344 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01-standby-01/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@iam-identity-universe-main-dev-worker-01-standby-01.service
[2026-01-02 04:34:11 UTC] USER=www-data EUID=0 PID=1416369 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.aYeRci /etc/systemd/system/postgresql@iam-identity-universe-main-dev-worker-01-standby-01.service
[2026-01-02 04:34:11 UTC] USER=www-data EUID=0 PID=1416390 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@iam-identity-universe-main-dev-worker-01-standby-01.service
[OK]   systemd unit written
[2026-01-02 04:34:11 UTC] USER=www-data EUID=0 PID=1416411 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 04:34:11 UTC] USER=www-data EUID=0 PID=1416432 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-02 04:34:11 UTC] USER=www-data EUID=0 PID=1416453 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-02 04:34:12 UTC] USER=www-data EUID=0 PID=1416567 ACTION=passthru ARGS=systemctl start postgresql@iam-identity-universe-main-dev-worker-01-standby-01.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-02 04:34:13 UTC] USER=www-data EUID=0 PID=1416608 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@iam-identity-universe-main-dev-worker-01-standby-01.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Configuring synchronous replication on primary worker-01...
[INFO] Current synchronous_standby_names: ''
[INFO] Initializing synchronous_standby_names with first standby
[INFO] New synchronous_standby_names: 'ANY 1 (worker_01_standby_01)'
[2026-01-02 04:34:13 UTC] USER=www-data EUID=0 PID=1416674 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_commit = on;
ALTER SYSTEM
[2026-01-02 04:34:14 UTC] USER=www-data EUID=0 PID=1416697 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_standby_names = 'ANY 1 (worker_01_standby_01)';
ALTER SYSTEM
[2026-01-02 04:34:14 UTC] USER=www-data EUID=0 PID=1416720 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-iam-identity-universe-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   ✅ Synchronous replication configured on primary
[OK]      Setting: ANY 1 (worker_01_standby_01)
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Skipping database/role provisioning on standby node (read-only)
[INFO]   Database/roles will be replicated from primary: worker-01
[INFO] Applying connection and memory optimizations...
[INFO] Standby will use primary's max_connections: 100
[INFO] Current settings: max_connections=100, work_mem=8MB
[INFO] Target settings (standby): max_connections=100, work_mem=8MB
[OK]   Connection settings already optimized
[INFO] Skipping password setting - this is a standby (read-only)
[INFO] Use primary's postgres password to connect to this standby
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] PostgreSQL IP: 10.100.1.191
[INFO] Primary hostname: db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01-standby-01...
[INFO]   db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.191

[INFO]   ✅ db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.191    db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com


[OK]   PostgreSQL 'iam-identity-universe-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.key \
        host=db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        iam-identity-universe-main-dev-postgresql-worker-01-standby-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.191
[INFO]   Port:              5432
[INFO]   FQDN:              db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 1a746118-a820-4708-bea9-85ec6c5e58dd
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 📦 03 role...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[2026-01-02 04:34:20 UTC] USER=www-data EUID=0 PID=1417054 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01-standby-01/standby.signal
⚠ This is a PostgreSQL STANDBY (read-only replica)
⚠ Skipping role creation - standby gets roles from primary via replication
⚠ Use the PRIMARY's credentials to connect to this standby


[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (iam-identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=worker-01-standby-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║                    IAM Database Schema Initialization                       ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] 🟢 Starting IAM schema provisioning...
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

[INFO] 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: core
  Core Identity Directory (tenants, realms, identities, devices, MFA)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [1/20]: core/01-tenant
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Identifier:  coordinator
  Database:    fastorder_iam_identity_universe_main_dev_db
  Host:        db-iam-identity-universe-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_iam_identity_universe_main_dev_db
ℹ️  Database fastorder_iam_identity_universe_main_dev_db already exists
✅ Connected to database: fastorder_iam_identity_universe_main_dev_db
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "pgcrypto" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "citext" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
NOTICE:  schema "utils" already exists, skipping
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
NOTICE:  schema "core" already exists, skipping
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
NOTICE:  relation "tenant" already exists, skipping
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[OK] Table core/01-tenant initialized

[INFO] 🔸 Table [2/20]: core/02-realm
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  relation "realm" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_realm_keycloak_id" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_realm_tenant" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[OK] Table core/02-realm initialized

[INFO] 🔸 Table [3/20]: core/03-identity
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  relation "identity" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_unique_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_unique_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_type" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[OK] Table core/03-identity initialized

[INFO] 🔸 Table [4/20]: core/04-device
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  relation "device" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_device_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_fingerprint" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_trusted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_last_seen" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[OK] Table core/04-device initialized

[INFO] 🔸 Table [5/20]: core/05-identity_account
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  relation "identity_account" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_account_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_lockout" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_last_login" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[OK] Table core/05-identity_account initialized

[INFO] 🔸 Table [6/20]: core/06-identity_mfa
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  relation "identity_mfa" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_mfa_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_active" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[OK] Table core/06-identity_mfa initialized

[INFO] 🔸 Table [7/20]: core/07-external_idp_link
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  relation "external_idp_link" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_external_idp_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_provider" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_email" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[OK] Table core/07-external_idp_link initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: policy
  RBAC/ABAC Authorization (clients, roles, permissions, policies)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [8/20]: policy/01-client
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
NOTICE:  schema "policy" already exists, skipping
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  relation "client" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_client_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_key" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_status" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[OK] Table policy/01-client initialized

[INFO] 🔸 Table [9/20]: policy/02-resource
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  relation "resource" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_resource_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_external" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_owner" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[OK] Table policy/02-resource initialized

[INFO] 🔸 Table [10/20]: policy/03-scope
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  relation "scope" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_scope_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_scope_name" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[OK] Table policy/03-scope initialized

[INFO] 🔸 Table [11/20]: policy/04-permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  relation "permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_permission_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_resource" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[OK] Table policy/04-permission initialized

[INFO] 🔸 Table [12/20]: policy/05-role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  relation "role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_keycloak" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[OK] Table policy/05-role initialized

[INFO] 🔸 Table [13/20]: policy/06-role_permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  relation "role_permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_permission_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_permission_perm" already exists, skipping
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[OK] Table policy/06-role_permission initialized

[INFO] 🔸 Table [14/20]: policy/07-identity_role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  relation "identity_role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_role_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_active" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[OK] Table policy/07-identity_role initialized

[INFO] 🔸 Table [15/20]: policy/08-policy_rule
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  relation "policy_rule" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_policy_rule_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_enabled" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_priority" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[OK] Table policy/08-policy_rule initialized

[INFO] 🔸 Table [16/20]: policy/09-api_key
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  relation "api_key" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_api_key_prefix" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[OK] Table policy/09-api_key initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: audit
  Audit & Risk Logging (auth events, admin actions, risk decisions)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [17/20]: audit/01-auth_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
NOTICE:  schema "audit" already exists, skipping
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
NOTICE:  relation "auth_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_auth_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_result" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_ip" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_session" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_trace" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_risk" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[OK] Table audit/01-auth_event initialized

[INFO] 🔸 Table [18/20]: audit/02-admin_action
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
NOTICE:  relation "admin_action" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_admin_action_actor" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_target" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_trace" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[OK] Table audit/02-admin_action initialized

[INFO] 🔸 Table [19/20]: audit/03-risk_decision
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
NOTICE:  relation "risk_decision" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_risk_decision_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_level" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_decision" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_auth" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[OK] Table audit/03-risk_decision initialized

[INFO] 🔸 Table [20/20]: audit/04-consent_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
NOTICE:  relation "consent_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_consent_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_version" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_granted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  relation "audit.auth_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  relation "audit.auth_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  relation "audit.admin_action_2026_03" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  relation "audit.admin_action_2026_04" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  relation "audit.risk_decision_2026_03" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  relation "audit.risk_decision_2026_04" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  relation "audit.consent_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  relation "audit.consent_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[OK] Table audit/04-consent_event initialized


════════════════════════════════════════════════════════════════════════════
[OK] ✅ IAM Schema Initialization Complete!
[OK] All 20 tables initialized successfully

Schemas created:
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

Design highlights:
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

════════════════════════════════════════════════════════════════════════════

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=f3523606-9518-407f-9770-e1c5d6b9db67)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=worker-01-standby-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for iam-identity in universe-dev...
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: iam-identity/run.sh started for SERVICE=iam-identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../iam-identity
🔍 DEBUG_CHECKPOINT_A3_FAIL: SERVICE_ROOT does not exist!
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: NONE
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
🔍 DEBUG_CHECKPOINT_A6_FAIL: No table folders found!
[WARN] No grouped table folders found under: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../iam-identity
End of 04-postgresql/steps/01-install/steps/iam-identity/run.sh

✓ ✅ Standby worker-01-standby-01 setup completed

✓ ✅ PostgreSQL installation completed
[INFO] Discovering additional setup steps...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 02-pg-bouncer.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up PgBouncer connection pooling...
[2026-01-02 04:35:42 UTC] USER=www-data EUID=0 PID=1419848 ACTION=fsop ARGS=rm -f /tmp/pgbouncer-ip.service /tmp/pgbouncer.service
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] Checking for existing PgBouncer application environment in topology …
[INFO] PgBouncer application not found in topology, creating new environment …
[INFO] 🎯 Custom Environment Creation (Example Wrapper)
[INFO] 📁 Orchestrator Library: /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] 💾 State Directory: /opt/fastorder/bash/scripts/env_app_setup/state

[INFO] 🚀 Calling centralized orchestrator: fo-env create-app
[INFO] 📋 Arguments: --service iam-identity --zone universe --branch main --env dev --domain db-iam-identity-universe-main-dev-postgresql-bouncer --app pgbouncer

[INFO] Creating application-specific environment configuration
[INFO] Environment ID: iam-identity-universe-main-dev
[INFO] Application: pgbouncer
[INFO] Base environment iam-identity-universe-main-dev already exists
[INFO] Allocated pgbouncer IP: 10.100.1.192
[INFO] Generated domain: db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO] Configuring network interface for pgbouncer IP: 10.100.1.192
[2026-01-02 04:35:44 UTC] USER=www-data EUID=0 PID=1420228 ACTION=passthru ARGS=ip addr add 10.100.1.192/32 dev eth0 label eth0:192
[ OK ] Configured pgbouncer IP 10.100.1.192 on interface eth0:192
[INFO] Creating systemd service for pgbouncer IP persistence...
[2026-01-02 04:35:44 UTC] USER=www-data EUID=0 PID=1420247 ACTION=passthru ARGS=systemctl daemon-reload
[ OK ] pgbouncer IP will persist across reboots
[INFO] Updating topology with application-specific configuration...
[ OK ] Topology updated with application-specific configuration
[INFO] Binding pgbouncer IP to domain: 10.100.1.192 -> db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com
[ OK ] Successfully bound db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com to 10.100.1.192
[ OK ] Domain correctly mapped
[ OK ] Application environment created successfully!
[INFO] 
[INFO] Application Details:
[INFO]   Environment ID: iam-identity-universe-main-dev
[INFO]   Application: pgbouncer
[INFO]   IP: 10.100.1.192
[INFO]   Domain: db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO] 
[INFO] To use this application:
[INFO]   source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO]   init_environment pgbouncer
[INFO]   echo $VM_IP  # Returns: 10.100.1.192

[ OK ] 🎉 Environment creation completed successfully!

[INFO] 📋 What happened:
[INFO]   ✅ Called centralized orchestrator at /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO]   ✅ All topology.json management handled centrally
[INFO]   ✅ Application-specific IP and domain configured
[INFO]   ✅ Network interface configured and made persistent
[INFO]   ✅ Domain binding added to /etc/hosts (if not skipped)

[INFO] 🔧 To use the centralized orchestrator directly:
[INFO]   # Add orchestrator to PATH
[INFO]   export PATH="/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/bin:$PATH"
[INFO]   # Then call directly
[INFO]   fo-env create-app --service auth --zone uae --env dev --app redis

[INFO] 📚 For more orchestrator commands:
[INFO]   fo-env --help
[OK]   Created new PgBouncer environment:
[INFO]   IP:     10.100.1.192
[INFO]   FQDN:   db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Domain: db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO] Final verification of /etc/hosts entry for db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com …
[OK]   /etc/hosts correctly maps db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com to 10.100.1.192
[WARN] IP 10.100.1.192 is already bound to other interface(s):
        inet 10.100.1.192/32 scope global eth0:192
[INFO] Attempting to also bind 10.100.1.192 to lo:pgbouncer ...
[2026-01-02 04:35:46 UTC] USER=www-data EUID=0 PID=1420394 ACTION=passthru ARGS=ip addr add 10.100.1.192/32 dev lo label lo:pgbouncer
[OK]   Successfully bound 10.100.1.192 to lo:pgbouncer
[2026-01-02 04:35:46 UTC] USER=www-data EUID=0 PID=1420412 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:35:47 UTC] USER=www-data EUID=0 PID=1420503 ACTION=passthru ARGS=systemctl restart pgbouncer-ip@iam-identity-universe-main-dev.service
Job for pgbouncer-ip@iam-identity-universe-main-dev.service failed because the control process exited with error code.
See "systemctl status pgbouncer-ip@iam-identity-universe-main-dev.service" and "journalctl -xeu pgbouncer-ip@iam-identity-universe-main-dev.service" for details.
[2026-01-02 04:35:47 UTC] USER=www-data EUID=0 PID=1420513 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer-ip@iam-identity-universe-main-dev.service
[WARN] pgbouncer-ip@iam-identity-universe-main-dev.service is not active
[WARN] Check status: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@iam-identity-universe-main-dev.service
[2026-01-02 04:35:47 UTC] USER=www-data EUID=0 PID=1420537 ACTION=fsop ARGS=mkdir -p /etc/pgbouncer/iam-identity-universe-main-dev
[2026-01-02 04:35:47 UTC] USER=www-data EUID=0 PID=1420547 ACTION=fsop ARGS=mkdir -p /run/pgbouncer/iam-identity-universe-main-dev
[2026-01-02 04:35:47 UTC] USER=www-data EUID=0 PID=1420556 ACTION=fsop ARGS=mkdir -p /var/log/pgbouncer/iam-identity-universe-main-dev
[2026-01-02 04:35:47 UTC] USER=www-data EUID=0 PID=1420565 ACTION=fsop ARGS=chmod 750 /etc/pgbouncer/iam-identity-universe-main-dev
[2026-01-02 04:35:47 UTC] USER=www-data EUID=0 PID=1420574 ACTION=fsop ARGS=chmod 750 /run/pgbouncer/iam-identity-universe-main-dev
[2026-01-02 04:35:47 UTC] USER=www-data EUID=0 PID=1420583 ACTION=fsop ARGS=chmod 750 /var/log/pgbouncer/iam-identity-universe-main-dev
[2026-01-02 04:35:47 UTC] USER=www-data EUID=0 PID=1420592 ACTION=fsop ARGS=chown root:postgres /etc/pgbouncer/iam-identity-universe-main-dev
[2026-01-02 04:35:47 UTC] USER=www-data EUID=0 PID=1420601 ACTION=fsop ARGS=chown postgres:postgres /run/pgbouncer/iam-identity-universe-main-dev
[2026-01-02 04:35:47 UTC] USER=www-data EUID=0 PID=1420610 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbouncer/iam-identity-universe-main-dev
[INFO] Generating pgbouncer_admin client certificates...
[INFO] ⏳ This may take 30-60 seconds...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
Environment: iam-identity-universe-main-dev
Username:    pgbouncer_admin
Identifier:  pgbouncer
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Service:     iam-identity
  Zone:        universe
  Branch:      main
  Env:         dev
  Node:        pgbouncer
  User (CN):   pgbouncer_admin
  Hostname:    db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:35:48 UTC] USER=www-data EUID=0 PID=1420644 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-pgbouncer-pgbouncer_admin
[2026-01-02 04:35:48 UTC] USER=www-data EUID=0 PID=1420653 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-02 04:35:48 UTC] USER=www-data EUID=0 PID=1420664 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
[2026-01-02 04:35:48 UTC] USER=www-data EUID=0 PID=1420673 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-02 04:35:48 UTC] USER=www-data EUID=0 PID=1420682 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = pgbouncer_admin
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:48 UTC] USER=www-data EUID=0 PID=1420696 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:48 UTC] USER=www-data EUID=0 PID=1420705 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:48 UTC] USER=www-data EUID=0 PID=1420714 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420723 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420732 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/root.crt
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420741 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/ca.crt
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420750 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key.pkcs1 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420759 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_der.key /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420768 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420777 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420786 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420795 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/root.crt
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420804 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420813 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420822 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420831 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420840 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/root.crt
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420849 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420875 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420884 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420893 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420902 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420911 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420920 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420929 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420938 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/root.crt
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420947 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/ca.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/ca.crt
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420956 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420965 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420975 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:49 UTC] USER=www-data EUID=0 PID=1420985 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1420994 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421003 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421012 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421021 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421030 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421039 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421048 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/root.crt
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421058 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/ca.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/ca.crt
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421067 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421076 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421086 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421096 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421105 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421114 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421123 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421132 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421141 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421150 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421159 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/root.crt
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421168 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/ca.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/ca.crt
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421177 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421186 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421196 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421206 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421215 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421224 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421233 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421242 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421251 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421260 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 04:35:50 UTC] USER=www-data EUID=0 PID=1421269 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/root.crt
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421278 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/ca.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/ca.crt
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421287 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421296 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin_der.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421306 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer
🎉 All requested users processed.
ℹ️  Kafka certificates not found at /opt/kafka/secrets/iam-identity-universe-main-dev/coordinator/pem (Kafka may not be installed yet)

✅ Client certificate generated successfully!

Environment: iam-identity-universe-main-dev
User: pgbouncer_admin
Node: pgbouncer
FQDN: db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/iam-identity-universe-main-dev/pgbouncer/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com -U pgbouncer_admin -d postgres

[OK]   mTLS client certificate present: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[INFO] Creating symlinks to canonical certificates in /etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend...
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421322 ACTION=fsop ARGS=mkdir -p /etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421331 ACTION=fsop ARGS=mkdir -p /etc/ssl/private/iam-identity-universe-main-dev/pg/pgbouncer-backend
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421340 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt /etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421349 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key /etc/ssl/private/iam-identity-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421358 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/root.crt /etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend/root.crt
[INFO] Creating coordinator CA symlink for PostgreSQL server verification...
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421367 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/root.crt /etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] Verifying canonical certificate permissions...
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421376 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421385 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421394 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/root.crt
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421403 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key
[OK]   Backend certificate symlinks created in /etc/ssl
[OK]   Coordinator CA symlink created for server verification
[OK]   Certificates already in canonical location - no symlinks needed
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421414 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/server.crt
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421423 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/server.key
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421432 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/ca.crt
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421441 ACTION=fsop ARGS=test -r /etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] PgBouncer will use PostgreSQL coordinator CA: /etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[OK]   PostgreSQL coordinator at db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com:5432 is reachable
[INFO] Dumping SCRAM secrets from coordinator for PgBouncer auth_file …
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421460 ACTION=fsop ARGS=cp /tmp/tmp.wAMmI7UU9W /etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421469 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt
[2026-01-02 04:35:51 UTC] USER=www-data EUID=0 PID=1421478 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt
[OK]   Auth file written: /etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt
[INFO] Generated new password for pgbouncer_admin
[INFO] Ensuring PgBouncer admin role 'pgbouncer_admin' exists in Postgres (coordinator) …
[OK]   Role pgbouncer_admin created/updated successfully
[SECRETS] Setting credentials in vault: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/pgbouncer_admin
✓ [SECRETS] Credentials created in vault: fastorder/db/iam-identity/universe/main/dev/postgresql/coordinator/pgbouncer_admin
[INFO] ✅ PgBouncer admin password stored in centralized secrets vault
[INFO] Re-fetching SCRAM secrets after role creation to ensure pgbouncer_admin is included …
[2026-01-02 04:35:59 UTC] USER=www-data EUID=0 PID=1421529 ACTION=fsop ARGS=cp /tmp/tmp.aOEFLw0BFD /etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt
[2026-01-02 04:35:59 UTC] USER=www-data EUID=0 PID=1421538 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt
[2026-01-02 04:35:59 UTC] USER=www-data EUID=0 PID=1421547 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt
[OK]   Auth file updated with pgbouncer_admin SCRAM hash
[INFO] Auth file contains [2026-01-02 04:35:59 UTC] USER=www-data EUID=0 PID=1421557 ACTION=passthru ARGS=bash -c wc -l < '/etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt'
4 user(s)
[OK]   Admin 'pgbouncer_admin' password generated and saved
[INFO] Configuring PostgreSQL to prevent Citus metadata sync hangs...
ALTER ROLE
[OK]   Disabled Citus metadata sync for pgbouncer_admin
[INFO] Verifying application database fastorder_iam_identity_universe_main_dev_db exists...
[OK]   ✓ Database fastorder_iam_identity_universe_main_dev_db exists
[INFO] Granting permissions to pgbouncer_admin on fastorder_iam_identity_universe_main_dev_db...
GRANT
[OK]   ✓ Granted CONNECT on fastorder_iam_identity_universe_main_dev_db to pgbouncer_admin
GRANT
[OK]   ✓ Granted USAGE on schema public to pgbouncer_admin
GRANT
[OK]   ✓ Granted SELECT on all tables to pgbouncer_admin
ALTER DATABASE
[OK]   Set synchronous_commit=local for fastorder_iam_identity_universe_main_dev_db
[INFO] Ensuring pg_hba.conf entry for pgbouncer_admin …
[INFO] Adding pg_hba.conf entries for pgbouncer_admin with cert auth …
[2026-01-02 04:36:00 UTC] USER=unknown EUID=33 PID=1421592 ACTION=-u ARGS=postgres bash
ERROR: Invalid or unauthorized action: -u
[OK]   pg_hba.conf updated and PostgreSQL configuration reloaded
[WARN] pg_hba.conf entry may not have loaded correctly
[INFO] Writing /etc/pgbouncer/iam-identity-universe-main-dev/pgbouncer.ini …
[2026-01-02 04:36:01 UTC] USER=www-data EUID=0 PID=1421617 ACTION=fsop ARGS=cp /tmp/tmp.Jd5jNVRv7K /etc/pgbouncer/iam-identity-universe-main-dev/pgbouncer.ini
[2026-01-02 04:36:01 UTC] USER=www-data EUID=0 PID=1421626 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/iam-identity-universe-main-dev/pgbouncer.ini
[2026-01-02 04:36:01 UTC] USER=www-data EUID=0 PID=1421635 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/iam-identity-universe-main-dev/pgbouncer.ini
[2026-01-02 04:36:01 UTC] USER=www-data EUID=0 PID=1421658 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbouncer/iam-identity-universe-main-dev /run/pgbouncer/iam-identity-universe-main-dev /var/log/pgbouncer/iam-identity-universe-main-dev
[2026-01-02 04:36:01 UTC] USER=www-data EUID=0 PID=1421671 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt
[OK]   pgbouncer.ini ready
[INFO] Verifying TLS settings in pgbouncer.ini:
[2026-01-02 04:36:01 UTC] USER=www-data EUID=0 PID=1421681 ACTION=fsop ARGS=grep -E (client_tls_sslmode|server_tls) /etc/pgbouncer/iam-identity-universe-main-dev/pgbouncer.ini
client_tls_sslmode = verify-full
server_tls_sslmode = verify-full
server_tls_ca_file = /etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
server_tls_cert_file = /etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
server_tls_key_file  = /etc/ssl/private/iam-identity-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[INFO] Verifying PgBouncer server certificate files:
[2026-01-02 04:36:01 UTC] USER=www-data EUID=0 PID=1421690 ACTION=fsop ARGS=test -r /etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[OK]   Server cert readable by postgres: /etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-01-02 04:36:01 UTC] USER=www-data EUID=0 PID=1421699 ACTION=fsop ARGS=test -r /etc/ssl/private/iam-identity-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[OK]   Server key readable by postgres: /etc/ssl/private/iam-identity-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[INFO] Verifying coordinator CA certificate:
[2026-01-02 04:36:01 UTC] USER=www-data EUID=0 PID=1421708 ACTION=fsop ARGS=test -r /etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[OK]   Coordinator CA readable by postgres: /etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] Preflight: stopping any conflicting PgBouncer on 6432 …
[2026-01-02 04:36:01 UTC] USER=www-data EUID=0 PID=1421717 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer.service
[2026-01-02 04:36:01 UTC] USER=www-data EUID=0 PID=1421726 ACTION=passthru ARGS=systemctl stop pgbouncer@iam-identity-universe-main-dev.service
Failed to stop pgbouncer@iam-identity-universe-main-dev.service: Unit pgbouncer@iam-identity-universe-main-dev.service not loaded.
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json?all=1": dial unix /var/run/docker.sock: connect: permission denied
[2026-01-02 04:36:04 UTC] USER=www-data EUID=0 PID=1421771 ACTION=passthru ARGS=systemctl daemon-reload
[OK]   systemd unit installed: pgbouncer@iam-identity-universe-main-dev.service
[INFO] Running pre-flight IP conflict check for 10.100.1.192:6432 …
[WARN] IP conflict checker not found at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/check-ip-conflicts.sh
[WARN] Skipping pre-flight check - conflicts may occur
[INFO] Starting PgBouncer (iam-identity-universe-main-dev) …
[2026-01-02 04:36:05 UTC] USER=www-data EUID=0 PID=1421874 ACTION=passthru ARGS=systemctl restart pgbouncer@iam-identity-universe-main-dev.service
[2026-01-02 04:36:05 UTC] USER=www-data EUID=0 PID=1421887 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer@iam-identity-universe-main-dev.service
[OK]   Service ACTIVE
[INFO] Verifying auth_file before probing …
[INFO] Auth file contains 4 user(s)
[WARN] Auth file does NOT contain pgbouncer_admin entry - authentication will fail
[INFO] Probing admin console via SSL (psql to database 'pgbouncer') …
[INFO] Retrieved password from vault for admin console probe
[WARN] Admin console probe failed (see error below)
psql: error: connection to server at "10.100.1.192", port 6432 failed: server certificate for "iam-identity-universe-main-dev.fastorder.com" (and 7 other names) does not match host name "10.100.1.192"
[WARN] Troubleshooting:
[WARN]   1. Check auth_file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt
[WARN]   2. Test with: PGPASSWORD='EHr7KMXOi9qm2aZPr8UX4p26' psql -h 10.100.1.192 -p 6432 -U pgbouncer_admin -d pgbouncer
[WARN]   3. Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@iam-identity-universe-main-dev.service -n 50

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Running Comprehensive PgBouncer Verification Tests
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Password extracted: EHr7KMXOi9... (using postgres user certificates)

[INFO] Test 1/7: Admin Console - SHOW POOLS
 database  |   user    | cl_active | cl_waiting | cl_active_cancel_req | cl_waiting_cancel_req | sv_active | sv_active_cancel | sv_being_canceled | sv_idle | sv_used | sv_tested | sv_login | maxwait | maxwait_us | pool_mode | load_balance_hosts 
-----------+-----------+-----------+------------+----------------------+-----------------------+-----------+------------------+-------------------+---------+---------+-----------+----------+---------+------------+-----------+--------------------
 pgbouncer | pgbouncer |         1 |          0 |                    0 |                     0 |         0 |                0 |                 0 |       0 |       0 |         0 |        0 |       0 |          0 | statement | 
(1 row)

[OK]   ✓ SHOW POOLS: SUCCESS

[INFO] Test 2/7: Admin Console - SHOW VERSION
[OK]   ✓ SHOW VERSION: PgBouncer 1.24.1

[INFO] Test 3/7: Admin Console - SHOW STATS
 database  | total_server_assignment_count | total_xact_count | total_query_count | total_received | total_sent | total_xact_time | total_query_time | total_wait_time | total_client_parse_count | total_server_parse_count | total_bind_count | avg_server_assignment_count | avg_xact_count | avg_query_count | avg_recv | avg_sent | avg_xact_time | avg_query_time | avg_wait_time | avg_client_parse_count | avg_server_parse_count | avg_bind_count 
-----------+-------------------------------+------------------+-------------------+----------------+------------+-----------------+------------------+-----------------+--------------------------+--------------------------+------------------+-----------------------------+----------------+-----------------+----------+----------+---------------+----------------+---------------+------------------------+------------------------+----------------
 pgbouncer |                             0 |                3 |                 3 |              0 |          0 |               0 |                0 |               0 |                        0 |                        0 |                0 |                           0 |              0 |               0 |        0 |        0 |             0 |              0 |             0 |                      0 |                      0 |              0
(1 row)

[OK]   ✓ SHOW STATS: SUCCESS

[INFO] Test 4/7: Admin Console - SHOW DATABASES
                    name                     |                                  host                                  | port |                  database                   | force_user | pool_size | min_pool_size | reserve_pool_size | server_lifetime | pool_mode | load_balance_hosts | max_connections | current_connections | max_client_connections | current_client_connections | paused | disabled 
---------------------------------------------+------------------------------------------------------------------------+------+---------------------------------------------+------------+-----------+---------------+-------------------+-----------------+-----------+--------------------+-----------------+---------------------+------------------------+----------------------------+--------+----------
 fastorder_iam_identity_universe_main_dev_db | db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com | 5432 | fastorder_iam_identity_universe_main_dev_db |            |       100 |             0 |                20 |            3600 |           |                    |               0 |                   0 |                      0 |                          0 |      0 |        0
 pgbouncer                                   |                                                                        | 6432 | pgbouncer                                   | pgbouncer  |         2 |             0 |                 0 |            3600 | statement |                    |               0 |                   0 |                      0 |                          1 |      0 |        0
(2 rows)

[OK]   ✓ SHOW DATABASES: SUCCESS

[INFO] Test 5/7: Admin Console - SHOW CONFIG
[OK]   ✓ SHOW CONFIG: SUCCESS
[INFO]   Key settings:
[INFO]     client_tls_sslmode = verify-full|disable|yes
[INFO]     max_client_conn = 2048|100|yes
[INFO]     pool_mode = transaction|session|yes
[INFO]     server_tls_sslmode = verify-full|prefer|yes
psql   "host=db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_iam_identity_universe_main_dev_db user=pgbouncer_admin password=EHr7KMXOi9qm2aZPr8UX4p26    connect_timeout=5 sslmode=verify-full    sslrootcert=/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/root.crt    sslcert=/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.crt    sslkey=/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/pgbouncer_admin.key"   --no-psqlrc -Atc 'SELECT version();'

[INFO] Test 6/7: Application Database - SELECT version()
[WARN] ✗ Application database query: FAILED (timeout or connection issue)
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[INFO] Test 7/8: Application Database - Connection Details
[WARN] ✗ Connection details: FAILED (timeout or connection issue)
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[INFO] Test 8/8: End-to-End Application Routing - Pool Verification
[INFO]   Running actual queries through PgBouncer to verify routing and pooling...
[WARN] ✗ End-to-end routing verification: FAILED - All 3 queries failed
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[WARN]    Otherwise check if database fastorder_iam_identity_universe_main_dev_db exists and user pgbouncer_admin has permissions

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Verification Complete - Tests 1-5 PASSED (Admin console verified)
[WARN]   Tests 6-8 FAILED - Application database not accessible
[WARN]   This is expected if Citus is not set up yet
[WARN]   Run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[OK]   PgBouncer is up for iam-identity-universe-main-dev

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Connection Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Password stored in: AWS Secrets Manager (fastorder/db/web/ksa/main/dev/postgresqliam-identity/universe/main/dev/coordinator-pgbouncer_admin)
Current password: EHr7KMXOi9qm2aZPr8UX4p26

1. Admin Console (using IP address to avoid DNS/SSL issues):
   psql "host=10.100.1.192 port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/iam-identity-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=EHr7KMXOi9qm2aZPr8UX4p26 sslmode=verify-full sslrootcert=/etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

2. Admin Console (using hostname):
   psql "host=db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/iam-identity-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=EHr7KMXOi9qm2aZPr8UX4p26 sslmode=verify-full sslrootcert=/etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

3. Application Database:
   psql "host=db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_iam_identity_universe_main_dev_db sslkey=/etc/ssl/private/iam-identity-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=EHr7KMXOi9qm2aZPr8UX4p26 sslmode=verify-full sslrootcert=/etc/ssl/certs/iam-identity-universe-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

4. Using .pgpass file:
   echo "db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com:6432:*:pgbouncer_admin:EHr7KMXOi9qm2aZPr8UX4p26" >> ~/.pgpass
   chmod 600 ~/.pgpass
   psql -h db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com -p 6432 -U pgbouncer_admin -d fastorder_iam_identity_universe_main_dev_db

5. Retrieve password from vault:
   source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
   PGPASSWORD="$(get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password')" \
     psql -h 10.100.1.192 -p 6432 -U pgbouncer_admin -d pgbouncer -c "SHOW POOLS;"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Architecture
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  • Default db 'fastorder_iam_identity_universe_main_dev_db' → Citus coordinator (db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com)
  • Worker access: 'fastorder_iam_identity_universe_main_dev_db_worker_1', 'fastorder_iam_identity_universe_main_dev_db_worker_2', … (if exist)
  • Client TLS: require (password auth) / verify-full (mTLS with certs)
  • Server TLS: verify-full (PgBouncer validates PostgreSQL certs)
  • Auth: SCRAM-SHA-256 via /etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt
  • Pool mode: transaction (stateless connections)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Management
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Service Status:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer@iam-identity-universe-main-dev.service
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@iam-identity-universe-main-dev.service

Logs:
  command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@iam-identity-universe-main-dev.service -f
  /usr/local/bin/fastorder-provisioning-wrapper.sh tail -f /var/log/pgbouncer/iam-identity-universe-main-dev/pgbouncer.log

Reload Config:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@iam-identity-universe-main-dev.service

Restart:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@iam-identity-universe-main-dev.service

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Files
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Config:        /etc/pgbouncer/iam-identity-universe-main-dev/pgbouncer.ini
Auth file:     /etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt
Server cert:   /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/server.crt
Server key:    /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/server.key
CA cert:       /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer/ca.crt
PG CA:         /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt
Logs:          /var/log/pgbouncer/iam-identity-universe-main-dev/pgbouncer.log

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Troubleshooting
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


If "SASL authentication failed":
  1. Check auth file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt
  2. Verify pgbouncer_admin is present with SCRAM hash
  3. Get password from vault:
     source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
     get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password'
  4. Reload PgBouncer: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@iam-identity-universe-main-dev.service

If "no pg_hba.conf entry":
  1. Check pg_hba.conf on coordinator
  2. Add rule: hostssl all pgbouncer_admin 10.100.1.192/32 cert clientcert=verify-full
  3. Reload PostgreSQL

To add users to PgBouncer:
  1. Create user in PostgreSQL with password
  2. Re-run SCRAM dump:
     psql "host=db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com port=5432 dbname=postgres user=postgres \
       sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/root.crt \
       sslcert=/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.crt sslkey=/etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key" \
       -Atc "SELECT '\"' || rolname || '\" \"' || rolpassword || '\"' \
             FROM pg_authid WHERE rolpassword LIKE 'SCRAM-SHA-256%' \
             AND rolcanlogin ORDER BY rolname;" | command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop tee /etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt
  3. Reload: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@iam-identity-universe-main-dev.service

[INFO] Registering PgBouncer node to observability API...
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        iam-identity-universe-main-dev-pgbouncer
[INFO]   Identifier Parent: postgresql
[INFO]   IP:                10.100.1.192
[INFO]   Port:              6432
[INFO]   FQDN:              db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 2337a090-f7f6-4571-84b2-d5854f13549c
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   PgBouncer node registered to observability API
✓ ✅ PgBouncer setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 03-citus-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] CITUS DISTRIBUTED CLUSTER SETUP
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Phase 1: Installing Citus extension on workers...
[INFO] Phase 2: Setting up coordinator and registering workers...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 📦 PHASE 1: Installing Citus extension on 1 worker(s)...

[INFO] → Worker 1/1: Installing Citus on worker-01...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS CLUSTER SETUP
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔧 Setting up Citus Worker...
[INFO] Temporarily disabling synchronous replication for extension installation...
t
[INFO] Installing Citus extension on worker...
[OK]   Citus extension installed on worker
[INFO] Restoring synchronous replication settings...
t
[INFO] Worker Citus extension installed - registration will happen when coordinator setup runs

[OK]   Citus setup complete for worker-01
[INFO] ═══════════════════════════════════════════════════════════════════════════════
✓   ✅ Citus extension installed on worker-01

✓ ✅ Phase 1 Complete: All 1 workers have Citus extension installed

[INFO] 🔧 PHASE 2: Setting up Citus coordinator and registering workers...

[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS CLUSTER SETUP
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔧 Setting up Citus Coordinator...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] DIAGNOSTIC: Configuration Variables
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PG_WORKERS_NUM: 1
[INFO] ENV_ID: iam-identity-universe-main-dev
[INFO] DOMAIN: fastorder.com
[INFO] PORT: 5432
[INFO] SOCKET_DIR: /var/run/postgresql-iam-identity-universe-main-dev-coordinator
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Ensuring postgres client certificates exist for coordinator...
[OK]   Postgres client certificates already exist for coordinator
[INFO] Adding citus_cert_map to coordinator pg_ident.conf...
[OK]   pg_ident.conf updated for coordinator
[INFO] Installing Citus extension on coordinator...
[OK]   Citus extension installed on coordinator (postgres database)
[INFO] Installing Citus extension on application database: fastorder_iam_identity_universe_main_dev_db...
[OK]   Citus extension installed on application database: fastorder_iam_identity_universe_main_dev_db
[INFO] Configuring Citus SSL connection parameters...
[2026-01-02 04:36:32 UTC] USER=www-data EUID=0 PID=1422212 ACTION=passthru ARGS=systemctl reload postgresql@iam-identity-universe-main-dev-coordinator.service
[OK]   ✅ Citus SSL connection parameters configured: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator
[WARN] Node not identified as coordinator, initializing...
[INFO] Checking coordinator configuration...
[INFO] Persisting citus.local_hostname to postgresql.conf...
[2026-01-02 04:36:35 UTC] USER=www-data EUID=0 PID=1422249 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator/postgresql.conf
[2026-01-02 04:36:35 UTC] USER=www-data EUID=0 PID=1422270 ACTION=passthru ARGS=systemctl reload postgresql@iam-identity-universe-main-dev-coordinator.service
[OK]   ✅ citus.local_hostname persisted to config and reloaded
[INFO] Configuring coordinator hostname in postgres database: db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com:5432

[OK]   ✅ Coordinator hostname set to db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com:5432 in postgres database
[INFO] Checking coordinator configuration in application database: fastorder_iam_identity_universe_main_dev_db...
[WARN] ⚠️  Coordinator registered as 'localhost' in application database, fixing...
[INFO] Configuring coordinator hostname in application database: db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com:5432
[OK]   ✅ Coordinator hostname set to db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com:5432 in application database
[INFO] Validating coordinator configuration before worker registration...
[OK]   ✅ Coordinator hostname validated: db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com
[OK]   ✅ citus_tables view is accessible
[INFO] Checking coordinator self-registration...
[OK]   ✅ Coordinator is already self-registered
[INFO] Configuring coordinator shard placement policy...
[OK]   ✅ Coordinator already configured in postgres database (shouldhaveshards = false)
[WARN] ⚠️  Coordinator has 16 shards in fastorder_iam_identity_universe_main_dev_db - cannot set shouldhaveshards=false
[WARN]    You must rebalance shards to workers first, then run this setup again
[WARN]    Skipping shouldhaveshards configuration for application database
[INFO] Registering 1 worker(s) to Citus cluster...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PRE-FLIGHT: Checking worker availability...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking worker worker-01...
[INFO]   FQDN: db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com
[OK]   ✅ Worker worker-01 is reachable via SSL
[OK]   All workers are reachable - proceeding with registration

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Adding Citus worker: db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com:5432
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Adding citus_cert_map to worker-01 pg_ident.conf...
[OK]   pg_ident.conf updated for worker-01
[INFO] Configuring worker worker-01 HBA for coordinator (10.100.1.189) access...
[OK]   Worker worker-01 HBA configured for coordinator (10.100.1.189)
[INFO] Adding replication rules for 3 standby(s)...
[OK]   Replication rules already exist for worker-01
[INFO] Reloading worker worker-01 to apply HBA changes...
[2026-01-02 04:36:38 UTC] USER=www-data EUID=0 PID=1422490 ACTION=passthru ARGS=systemctl reload postgresql@iam-identity-universe-main-dev-worker-01.service
[INFO] Configuring coordinator HBA for worker worker-01 (10.100.1.190) access...
[OK]   Coordinator HBA configured for worker worker-01 (10.100.1.190)
[INFO] Reloading coordinator to apply HBA changes...
[2026-01-02 04:36:39 UTC] USER=www-data EUID=0 PID=1422520 ACTION=passthru ARGS=systemctl reload postgresql@iam-identity-universe-main-dev-coordinator.service
[INFO] Ensuring postgres client certificates exist for worker-01...
[OK]   Postgres client certificates already exist for worker-01
[INFO] Configuring citus.node_conninfo on worker-01...
[2026-01-02 04:36:39 UTC] USER=www-data EUID=0 PID=1422538 ACTION=passthru ARGS=systemctl reload postgresql@iam-identity-universe-main-dev-worker-01.service
[OK]   citus.node_conninfo configured on worker-01
[INFO] Temporarily relaxing sync-rep on worker worker-01...
t
[OK]   Worker worker-01 sync-rep relaxed (was: sync_commit=on)
[INFO] Ensuring Citus extension on worker databases...
CREATE EXTENSION
CREATE EXTENSION
[INFO] Running citus_add_node with 180s timeout...
NOTICE:  shards are still on the coordinator after adding the new node
HINT:  Use SELECT rebalance_table_shards(); to balance shards data between workers and coordinator or SELECT citus_drain_node('db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com',5432); to permanently move shards away from the coordinator.
2
[INFO] Restoring worker worker-01 sync-rep settings...
t
[OK]   Worker worker-01 sync-rep restored
[OK]   ✅ Worker db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com successfully added to Citus cluster
[INFO]    Node ID: 2
[INFO]    Registered in: postgres, fastorder_iam_identity_universe_main_dev_db
[OK]   Worker worker-01 registration successful
[INFO] Configuring worker worker-01 shard placement policy...
[OK]   ✅ Worker worker-01 configured to hold shards in all databases


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] POST-REGISTRATION: Verifying cluster state...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Expected workers: 1
[INFO] Registered workers: 1
[OK]   ✅ All 1 workers successfully registered!

[INFO] Citus cluster configuration:
db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com  5432  0  t  primary  f
db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com    5432  1  t  primary  t

[INFO] Note: groupid=0 is the coordinator, groupid>0 are workers
[INFO]       shouldhaveshards: false=query router only, true=holds data shards

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] FINAL VALIDATION: Verifying configuration persistence...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:36:43 UTC] USER=www-data EUID=0 PID=1422696 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator/postgresql.conf
[OK]   ✅ citus.local_hostname persisted in postgresql.conf
[OK]   ✅ All 1 worker(s) successfully registered and verified

[OK]   ✅ All validation checks passed
[OK]   Citus coordinator setup complete

[OK]   Citus setup complete for coordinator
[INFO] ═══════════════════════════════════════════════════════════════════════════════

✓ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ CITUS CLUSTER SETUP COMPLETED SUCCESSFULLY
✓    Coordinator: Ready and accepting connections
✓    Workers registered: 1
✓ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 05-backup-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up coordinator backup...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for iam-identity-universe-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-01-02 04:36:45 UTC] USER=www-data EUID=0 PID=1422753 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/iam-identity-universe-main-dev
[2026-01-02 04:36:45 UTC] USER=www-data EUID=0 PID=1422762 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/iam-identity-universe-main-dev
[2026-01-02 04:36:45 UTC] USER=www-data EUID=0 PID=1422771 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-01-02 04:36:45 UTC] USER=www-data EUID=0 PID=1422780 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-02 04:36:45 UTC] USER=www-data EUID=0 PID=1422789 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-02 04:36:45 UTC] USER=www-data EUID=0 PID=1422798 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-02 04:36:53 UTC] USER=www-data EUID=0 PID=1422816 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-02 04:36:53 UTC] USER=www-data EUID=0 PID=1422825 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-02 04:36:53 UTC] USER=www-data EUID=0 PID=1422834 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-01-02 04:36:53 UTC] USER=www-data EUID=0 PID=1422843 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/iam-identity-universe-main-dev
[2026-01-02 04:36:53 UTC] USER=www-data EUID=0 PID=1422852 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/iam-identity-universe-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-iam-identity-universe-main-dev
[2026-01-02 04:36:53 UTC] USER=www-data EUID=0 PID=1422875 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-02 04:36:53 UTC] USER=www-data EUID=0 PID=1422884 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-01-02 04:36:53 UTC] USER=www-data EUID=0 PID=1422893 ACTION=fsop ARGS=find /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-01-02 04:36:53 UTC] USER=www-data EUID=0 PID=1422902 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-02 04:36:53 UTC] USER=www-data EUID=0 PID=1422911 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-01-02 04:36:53 UTC] USER=www-data EUID=0 PID=1422920 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-01-02 04:36:53 UTC] USER=www-data EUID=0 PID=1422929 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-02 04:36:53 UTC] USER=www-data EUID=0 PID=1422938 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator/PG_VERSION
[2026-01-02 04:36:53 UTC] USER=www-data EUID=0 PID=1422948 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@iam-identity-universe-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza iam-identity-universe-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-02 04:36:54 UTC] USER=www-data EUID=0 PID=1423002 ACTION=passthru ARGS=systemctl stop postgresql@iam-identity-universe-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-02 04:36:56 UTC] USER=www-data EUID=0 PID=1423015 ACTION=passthru ARGS=systemctl start postgresql@iam-identity-universe-main-dev-coordinator.service
[2026-01-02 04:37:00 UTC] USER=www-data EUID=0 PID=1423047 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@iam-identity-universe-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-02 04:37:00 UTC] USER=www-data EUID=0 PID=1423071 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=iam-identity-universe-main-dev-coordinator --log-level-console=info check
2026-01-02 04:37:00.631 P00   INFO: check command begin 2.56.0: --exec-id=1423079-c58c3dfc --log-level-console=info --log-level-file=debug --pg1-path=/var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-iam-identity-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/iam-identity-universe-main-dev --stanza=iam-identity-universe-main-dev-coordinator
2026-01-02 04:37:00.656 P00   INFO: check repo1 configuration (primary)
2026-01-02 04:37:00.676 P00  ERROR: [028]: backup and archive info files exist but do not match the database
                                    HINT: is this the correct stanza?
                                    HINT: did an error occur during stanza-upgrade?
2026-01-02 04:37:00.676 P00   INFO: check command end: aborted with exception [028]
[WARN] ⚠️  Stanza verification failed - this may be normal if WAL archiving hasn't started yet
[WARN]    The backup system is configured and will work once WAL segments are generated

[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-02 04:37:00 UTC] USER=www-data EUID=0 PID=1423092 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|iam-identity-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-iam-identity-universe-main-dev.sh
[2026-01-02 04:37:00 UTC] USER=www-data EUID=0 PID=1423101 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-iam-identity-universe-main-dev.sh
[2026-01-02 04:37:00 UTC] USER=www-data EUID=0 PID=1423119 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|iam-identity-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-iam-identity-universe-main-dev.sh
[2026-01-02 04:37:00 UTC] USER=www-data EUID=0 PID=1423128 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-iam-identity-universe-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-02 04:37:00 UTC] USER=www-data EUID=0 PID=1423146 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-iam-identity-universe-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-01-02 04:37:01 UTC] USER=www-data EUID=0 PID=1423164 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|iam-identity-universe-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md
[2026-01-02 04:37:01 UTC] USER=www-data EUID=0 PID=1423173 ACTION=fsop ARGS=sed -i s|__ENV_ID__|iam-identity-universe-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md
[2026-01-02 04:37:01 UTC] USER=www-data EUID=0 PID=1423182 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md
[2026-01-02 04:37:01 UTC] USER=www-data EUID=0 PID=1423191 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md
[2026-01-02 04:37:01 UTC] USER=www-data EUID=0 PID=1423200 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-02 04:37:01.294 P00   INFO: start command begin 2.56.0: --exec-id=1423221-3e4e0934 --log-level-console=info --log-level-file=debug --stanza=iam-identity-universe-main-dev-coordinator
2026-01-02 04:37:01.295 P00   WARN: stop file does not exist for stanza iam-identity-universe-main-dev-coordinator
2026-01-02 04:37:01.295 P00   INFO: start command end: completed successfully (6ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-02 04:37:01.365 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=1423232-229f3a72 --log-level-console=info --log-level-file=debug --no-online --pg1-path=/var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-iam-identity-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/iam-identity-universe-main-dev --stanza=iam-identity-universe-main-dev-coordinator
2026-01-02 04:37:01.371 P00   INFO: stanza-upgrade for stanza 'iam-identity-universe-main-dev-coordinator' on repo1
2026-01-02 04:37:01.402 P00   INFO: stanza-upgrade command end: completed successfully (42ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-02 04:37:01 UTC] USER=www-data EUID=0 PID=1423236 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260102-043701.log
[2026-01-02 04:37:01 UTC] USER=www-data EUID=0 PID=1423254 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260102-043701.log
[2026-01-02 04:37:01 UTC] USER=www-data EUID=0 PID=1423266 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260102-043701.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-01-02 04:37:10 UTC] USER=www-data EUID=0 PID=1423340 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-1422728.log /var/log/pgbackrest/initial-backup-20260102-043701.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260102-043701.log
   2026-01-02 04:37:10.694 P00   INFO: repo1: remove expired backup 20260102-025431F
   2026-01-02 04:37:10.735 P00   INFO: repo1: 17-82 remove archive, start = 000000010000000000000004, stop = 000000010000000000000006
   2026-01-02 04:37:10.736 P00   INFO: repo1: 17-83 no archive to remove
   2026-01-02 04:37:10.736 P00   INFO: repo1: 17-84 remove archive, start = 000000010000000000000003, stop = 000000010000000000000003
   2026-01-02 04:37:10.737 P00   INFO: expire command end: completed successfully (60ms)

[INFO] Current backups:
stanza: iam-identity-universe-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000007/00000001000000000000000C

        full backup: 20260102-025455F
            timestamp start/stop: 2026-01-02 02:54:55+00 / 2026-01-02 02:55:00+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.4MB, database backup size: 37.4MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

    db (prior)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000013

        full backup: 20260102-031925F
            timestamp start/stop: 2026-01-02 03:19:25+00 / 2026-01-02 03:19:35+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.4MB, database backup size: 37.4MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

        full backup: 20260102-031947F
            timestamp start/stop: 2026-01-02 03:19:47+00 / 2026-01-02 03:19:50+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.4MB, database backup size: 37.4MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

    db (current)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000004

        full backup: 20260102-043701F
            timestamp start/stop: 2026-01-02 04:37:01+00 / 2026-01-02 04:37:10+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.4MB, database backup size: 37.4MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         iam-identity-universe-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/iam-identity-universe-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/iam-identity-universe-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=iam-identity-universe-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up worker backups for 1 worker(s)...
[INFO] Setting up backup for: worker-01
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for iam-identity-universe-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-01-02 04:37:11 UTC] USER=www-data EUID=0 PID=1423392 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/iam-identity-universe-main-dev
[2026-01-02 04:37:11 UTC] USER=www-data EUID=0 PID=1423401 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/iam-identity-universe-main-dev
[2026-01-02 04:37:11 UTC] USER=www-data EUID=0 PID=1423410 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-01-02 04:37:11 UTC] USER=www-data EUID=0 PID=1423419 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-02 04:37:11 UTC] USER=www-data EUID=0 PID=1423428 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-02 04:37:11 UTC] USER=www-data EUID=0 PID=1423437 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-02 04:37:13 UTC] USER=www-data EUID=0 PID=1423451 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-02 04:37:13 UTC] USER=www-data EUID=0 PID=1423460 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-02 04:37:13 UTC] USER=www-data EUID=0 PID=1423469 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-01-02 04:37:13 UTC] USER=www-data EUID=0 PID=1423478 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/iam-identity-universe-main-dev
[2026-01-02 04:37:13 UTC] USER=www-data EUID=0 PID=1423487 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/iam-identity-universe-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-iam-identity-universe-main-dev
[2026-01-02 04:37:13 UTC] USER=www-data EUID=0 PID=1423508 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-02 04:37:13 UTC] USER=www-data EUID=0 PID=1423517 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-01-02 04:37:13 UTC] USER=www-data EUID=0 PID=1423526 ACTION=fsop ARGS=find /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-01-02 04:37:13 UTC] USER=www-data EUID=0 PID=1423535 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-02 04:37:13 UTC] USER=www-data EUID=0 PID=1423544 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-01-02 04:37:14 UTC] USER=www-data EUID=0 PID=1423553 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-01-02 04:37:14 UTC] USER=www-data EUID=0 PID=1423562 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-02 04:37:14 UTC] USER=www-data EUID=0 PID=1423571 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator/PG_VERSION
[2026-01-02 04:37:14 UTC] USER=www-data EUID=0 PID=1423581 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@iam-identity-universe-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza iam-identity-universe-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-02 04:37:14 UTC] USER=www-data EUID=0 PID=1423644 ACTION=passthru ARGS=systemctl stop postgresql@iam-identity-universe-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-02 04:37:17 UTC] USER=www-data EUID=0 PID=1423660 ACTION=passthru ARGS=systemctl start postgresql@iam-identity-universe-main-dev-coordinator.service
[2026-01-02 04:37:21 UTC] USER=www-data EUID=0 PID=1423696 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@iam-identity-universe-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-02 04:37:21 UTC] USER=www-data EUID=0 PID=1423720 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=iam-identity-universe-main-dev-coordinator --log-level-console=info check
2026-01-02 04:37:21.253 P00   INFO: check command begin 2.56.0: --exec-id=1423728-dc2d7efa --log-level-console=info --log-level-file=debug --pg1-path=/var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-iam-identity-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/iam-identity-universe-main-dev --stanza=iam-identity-universe-main-dev-coordinator
2026-01-02 04:37:21.277 P00   INFO: check repo1 configuration (primary)
2026-01-02 04:37:21.355 P00   INFO: check repo1 archive for WAL (primary)
2026-01-02 04:37:21.657 P00   INFO: WAL segment 000000010000000000000006 successfully archived to '/var/lib/pgbackrest/backup/iam-identity-universe-main-dev/archive/iam-identity-universe-main-dev-coordinator/17-84/0000000100000000/000000010000000000000006-ba0adfd28df1bcce0fc29b3a6872e7d0e087657f.lz4' on repo1
2026-01-02 04:37:21.657 P00   INFO: check command end: completed successfully (409ms)
[INFO] ✅ Stanza verification passed

[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-02 04:37:21 UTC] USER=www-data EUID=0 PID=1423750 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|iam-identity-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-iam-identity-universe-main-dev.sh
[2026-01-02 04:37:21 UTC] USER=www-data EUID=0 PID=1423759 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-iam-identity-universe-main-dev.sh
[2026-01-02 04:37:21 UTC] USER=www-data EUID=0 PID=1423777 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|iam-identity-universe-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-iam-identity-universe-main-dev.sh
[2026-01-02 04:37:21 UTC] USER=www-data EUID=0 PID=1423786 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-iam-identity-universe-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-02 04:37:21 UTC] USER=www-data EUID=0 PID=1423804 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-iam-identity-universe-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-01-02 04:37:21 UTC] USER=www-data EUID=0 PID=1423822 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|iam-identity-universe-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md
[2026-01-02 04:37:22 UTC] USER=www-data EUID=0 PID=1423831 ACTION=fsop ARGS=sed -i s|__ENV_ID__|iam-identity-universe-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md
[2026-01-02 04:37:22 UTC] USER=www-data EUID=0 PID=1423840 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md
[2026-01-02 04:37:22 UTC] USER=www-data EUID=0 PID=1423849 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md
[2026-01-02 04:37:22 UTC] USER=www-data EUID=0 PID=1423858 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-02 04:37:22.237 P00   INFO: start command begin 2.56.0: --exec-id=1423879-f0bf635f --log-level-console=info --log-level-file=debug --stanza=iam-identity-universe-main-dev-coordinator
2026-01-02 04:37:22.237 P00   WARN: stop file does not exist for stanza iam-identity-universe-main-dev-coordinator
2026-01-02 04:37:22.237 P00   INFO: start command end: completed successfully (5ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-02 04:37:22.303 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=1423890-053cd34c --log-level-console=info --log-level-file=debug --no-online --pg1-path=/var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-iam-identity-universe-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/iam-identity-universe-main-dev --stanza=iam-identity-universe-main-dev-coordinator
2026-01-02 04:37:22.304 P00   INFO: stanza-upgrade for stanza 'iam-identity-universe-main-dev-coordinator' on repo1
2026-01-02 04:37:22.308 P00   INFO: stanza 'iam-identity-universe-main-dev-coordinator' on repo1 is already up to date
2026-01-02 04:37:22.308 P00   INFO: stanza-upgrade command end: completed successfully (10ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-02 04:37:22 UTC] USER=www-data EUID=0 PID=1423894 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260102-043722.log
[2026-01-02 04:37:22 UTC] USER=www-data EUID=0 PID=1423903 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260102-043722.log
[2026-01-02 04:37:22 UTC] USER=www-data EUID=0 PID=1423912 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260102-043722.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-01-02 04:37:25 UTC] USER=www-data EUID=0 PID=1423953 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-1423365.log /var/log/pgbackrest/initial-backup-20260102-043722.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260102-043722.log
   2026-01-02 04:37:25.497 P00   INFO: repo1: remove expired backup 20260102-025455F
   2026-01-02 04:37:25.551 P00   INFO: repo1: remove archive path /var/lib/pgbackrest/backup/iam-identity-universe-main-dev/archive/iam-identity-universe-main-dev-coordinator/17-82
   2026-01-02 04:37:25.552 P00   INFO: repo1: 17-83 no archive to remove
   2026-01-02 04:37:25.553 P00   INFO: repo1: 17-84 no archive to remove
   2026-01-02 04:37:25.553 P00   INFO: expire command end: completed successfully (65ms)

[INFO] Current backups:
stanza: iam-identity-universe-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000013

        full backup: 20260102-031925F
            timestamp start/stop: 2026-01-02 03:19:25+00 / 2026-01-02 03:19:35+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.4MB, database backup size: 37.4MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

        full backup: 20260102-031947F
            timestamp start/stop: 2026-01-02 03:19:47+00 / 2026-01-02 03:19:50+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.4MB, database backup size: 37.4MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

    db (current)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000007

        full backup: 20260102-043701F
            timestamp start/stop: 2026-01-02 04:37:01+00 / 2026-01-02 04:37:10+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.4MB, database backup size: 37.4MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

        full backup: 20260102-043722F
            timestamp start/stop: 2026-01-02 04:37:22+00 / 2026-01-02 04:37:25+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.4MB, database backup size: 37.4MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         iam-identity-universe-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/iam-identity-universe-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/iam-identity-universe-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=iam-identity-universe-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ Backup setup completed for coordinator and all workers

[INFO] Skipping 06-distribute-tables-canary.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 07-distribute-tables.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:37:27 UTC] USER=unknown EUID=33 PID=1424000 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-02 04:37:27 UTC] USER=unknown EUID=33 PID=1424007 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-02 04:37:27 UTC] USER=unknown EUID=33 PID=1424014 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-02 04:37:27 UTC] USER=unknown EUID=33 PID=1424021 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS TABLE DISTRIBUTION
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔐 Secure connection established
[INFO]    Host: db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com:5432
[INFO]    Database: fastorder_iam_identity_universe_main_dev_db
[INFO]    SSL: verify-full (TLS 1.2+)
[INFO]    Timeouts: statement=120s, idle_tx=300s

[INFO] 🔍 Running preflight checks...
[INFO] Testing database connectivity...
[OK]   ✅ Database connection successful
[OK]   ✅ Connected to correct database: fastorder_iam_identity_universe_main_dev_db
[INFO] Checking Citus extension in database fastorder_iam_identity_universe_main_dev_db...
[OK]   Citus version: 13.2-1
[INFO] Checking worker registration...
[OK]   Registered workers: 1
[INFO] Worker nodes:
[INFO]                                  nodename                               | nodeport | isactive | noderole 
[INFO]   ----------------------------------------------------------------------+----------+----------+----------
[INFO]    db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com |     5432 | t        | primary
[INFO]   (1 row)
[INFO]   

[INFO] 📊 Starting table distribution...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Distributing: auth.login_account
[INFO] Description: User authentication table - distributed by region for tenant isolation
[INFO] Shard key: region_hint
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ⏭️  Table does not exist, skipping

[INFO] ═══════════════════════════════════════════════════════════════════════════════
[OK]   ✅ All tables distributed successfully!
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 📊 Citus Cluster Summary:

[INFO] Distributed tables:
[INFO]            table          |   type    | shard_key | shards | size  
[INFO]   ------------------------+-----------+-----------+--------+-------
[INFO]    core.tenant            | reference | <none>    |      1 | 24 kB
[INFO]    core.realm             | local     | <none>    |      1 | 40 kB
[INFO]    core.identity          | local     | <none>    |      1 | 72 kB
[INFO]    core.device            | local     | <none>    |      1 | 48 kB
[INFO]    core.identity_account  | local     | <none>    |      1 | 48 kB
[INFO]    core.identity_mfa      | local     | <none>    |      1 | 40 kB
[INFO]    core.external_idp_link | local     | <none>    |      1 | 48 kB
[INFO]    policy.client          | local     | <none>    |      1 | 56 kB
[INFO]    policy.resource        | local     | <none>    |      1 | 48 kB
[INFO]    policy.scope           | local     | <none>    |      1 | 40 kB
[INFO]    policy.permission      | local     | <none>    |      1 | 48 kB
[INFO]    policy.role            | local     | <none>    |      1 | 56 kB
[INFO]    policy.role_permission | local     | <none>    |      1 | 24 kB
[INFO]    policy.identity_role   | local     | <none>    |      1 | 40 kB
[INFO]    policy.policy_rule     | local     | <none>    |      1 | 48 kB
[INFO]    policy.api_key         | local     | <none>    |      1 | 56 kB
[INFO]   (16 rows)
[INFO]   

[INFO] Worker capacity:
[INFO]    worker | total_shards | total_size 
[INFO]   --------+--------------+------------
[INFO]   (0 rows)
[INFO]   

[OK]   Citus table distribution complete

[INFO] Skipping 08-distribute-tables-rollback.sh (rollback script - run manually only)
[INFO] Skipping 09-distribute-tables-test.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 10-setup-cdc.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] CDC PIPELINE SETUP (Debezium + Elasticsearch Sink)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Log file: /var/log/fastorder/cdc/10-setup-cdc-*.log

[INFO] Running CDC setup for identifier: coordinator
[2026-01-02 04:37:35] ==========================================
[2026-01-02 04:37:35] CDC SETUP SCRIPT STARTED
[2026-01-02 04:37:35] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260102_043735.log
[2026-01-02 04:37:35] ==========================================
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[2026-01-02 04:37:35] Applied SERVICE_OVERRIDE: iam-identity
[2026-01-02 04:37:35] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:37:35]   CDC Pipeline Setup (Debezium + ES Sink)
[2026-01-02 04:37:35] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:37:35]   Environment: iam-identity-universe-main-dev
[2026-01-02 04:37:35]   Identifier:  coordinator
[2026-01-02 04:37:35]   Service:     iam-identity
[2026-01-02 04:37:35] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-02 04:37:35] 📂 CDC_BASE_DIR exists: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc
[2026-01-02 04:37:35] Looking for service folder: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/iam-identity
[2026-01-02 04:37:35] ⚠️  No CDC configuration for service 'iam-identity' at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/iam-identity
[2026-01-02 04:37:35] ℹ️  Skipping CDC setup for this service
✓ ✅ CDC Pipeline setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 11-monitoring-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up monitoring for coordinator...
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 PostgreSQL Monitoring Integration for iam-identity-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-iam-identity-universe-main-dev
[OK]   Observability cell endpoints registered for iam-identity-universe-main-dev
[OK]   ✓ Observability cell is ready

[INFO] ✓ Using private IP for metrics: 10.100.1.189
[INFO] 2️⃣ Setting up postgres_exporter integration...
[INFO] Checking observability cell readiness: obs-iam-identity-universe-main-dev
[OK]   Observability cell endpoints registered for iam-identity-universe-main-dev
[INFO] Setting up postgres_exporter for iam-identity-universe-main-dev
[2026-01-02 04:37:38 UTC] USER=www-data EUID=0 PID=1424477 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-iam-identity-universe-main-dev.yaml /etc/prometheus/postgres_exporter_queries-iam-identity-universe-main-dev.yaml
[2026-01-02 04:37:38 UTC] USER=www-data EUID=0 PID=1424486 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-iam-identity-universe-main-dev.yaml
[2026-01-02 04:37:38 UTC] USER=www-data EUID=0 PID=1424495 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-iam-identity-universe-main-dev.yaml
[OK]   Custom queries file created at /etc/prometheus/postgres_exporter_queries-iam-identity-universe-main-dev.yaml
[2026-01-02 04:37:38 UTC] USER=www-data EUID=0 PID=1424505 ACTION=passthru ARGS=mv /tmp/postgres_exporter-iam-identity-universe-main-dev.service /etc/systemd/system/postgres_exporter-iam-identity-universe-main-dev.service
[2026-01-02 04:37:38 UTC] USER=www-data EUID=0 PID=1424514 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:37:38 UTC] USER=www-data EUID=0 PID=1424561 ACTION=passthru ARGS=systemctl enable postgres_exporter-iam-identity-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/postgres_exporter-iam-identity-universe-main-dev.service -> /etc/systemd/system/postgres_exporter-iam-identity-universe-main-dev.service.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  IP Conflict Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: iam-identity-universe-main-dev
IP Address:  10.100.1.189
Port:        9187
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔍 Checking IP conflict for iam-identity-universe-main-dev on 10.100.1.189:9187...
✅ IP 10.100.1.189:9187 is available - no conflicts detected

🔍 Checking for orphaned processes that might conflict...
✅ No orphaned processes detected

✅ All checks passed - safe to proceed with iam-identity-universe-main-dev setup
[2026-01-02 04:37:39 UTC] USER=www-data EUID=0 PID=1424639 ACTION=passthru ARGS=systemctl restart postgres_exporter-iam-identity-universe-main-dev.service
[OK]   postgres_exporter configured on db-iam-identity-universe-main-dev-postgresql.fastorder.com:9187
[INFO] Adding PostgreSQL scrape target to Prometheus config...
[OK]   PostgreSQL scrape target added
[INFO] Creating PostgreSQL alert rules...
[2026-01-02 04:37:41 UTC] USER=www-data EUID=0 PID=1424669 ACTION=fsop ARGS=mv /tmp/postgresql_alerts_iam-identity-universe-main-dev.yml /etc/prometheus/obs-iam-identity-universe-main-dev/rules/postgresql_alerts.yml
[OK]   PostgreSQL alert rules created: /etc/prometheus/obs-iam-identity-universe-main-dev/rules/postgresql_alerts.yml
[INFO] Adding PostgreSQL alerts to Prometheus config...
[2026-01-02 04:37:41 UTC] USER=www-data EUID=0 PID=1424679 ACTION=fsop ARGS=sed -i /rule_files:/a\  - "rules/postgresql_alerts.yml" /etc/prometheus/obs-iam-identity-universe-main-dev/prometheus.yml
[OK]   PostgreSQL alerts registered in Prometheus
[2026-01-02 04:37:41 UTC] USER=www-data EUID=0 PID=1424689 ACTION=passthru ARGS=systemctl reload prometheus-obs-iam-identity-universe-main-dev.service
Failed to reload prometheus-obs-iam-identity-universe-main-dev.service: Job type reload is not applicable for unit prometheus-obs-iam-identity-universe-main-dev.service.
[2026-01-02 04:37:41 UTC] USER=www-data EUID=0 PID=1424698 ACTION=passthru ARGS=systemctl restart prometheus-obs-iam-identity-universe-main-dev.service
[OK]   Prometheus reloaded with PostgreSQL monitoring
[OK]   ✓ postgres_exporter integration complete
[INFO] Registering postgres_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: postgres_exporter -> 10.100.1.189:9187
[OK]   ✓ Registered postgres_exporter scrape target: 10.100.1.189:9187
[INFO]   Target file: /etc/prometheus/obs-iam-identity-universe-main-dev/targets/postgres_exporter.yml
[OK]   ✓ postgres_exporter registered as Prometheus scrape target

[INFO] 3️⃣ Setting up pgbouncer_exporter integration...
[INFO] PgBouncer FQDN found in /etc/hosts: db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.192
[INFO] PgBouncer detected: db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com:6432
[OK]   ✓ pgbouncer_exporter already installed
[INFO] Getting pgbouncer_admin password (SERVICE=iam-identity, ZONE=universe)
[OK]   ✓ pgbouncer_admin password retrieved (24 chars)
[INFO] Using pgbouncer certs from: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer
[INFO] Creating pgbouncer_exporter systemd service...
[OK]   ✓ pgbouncer_exporter service file created
[INFO] Starting pgbouncer_exporter service...
[2026-01-02 04:37:44 UTC] USER=www-data EUID=0 PID=1424761 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:37:44 UTC] USER=www-data EUID=0 PID=1424806 ACTION=passthru ARGS=systemctl enable pgbouncer_exporter-iam-identity-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/pgbouncer_exporter-iam-identity-universe-main-dev.service -> /etc/systemd/system/pgbouncer_exporter-iam-identity-universe-main-dev.service.
[2026-01-02 04:37:45 UTC] USER=www-data EUID=0 PID=1424851 ACTION=passthru ARGS=systemctl restart pgbouncer_exporter-iam-identity-universe-main-dev.service
[OK]   ✓ pgbouncer_exporter service running
[INFO] Registering pgbouncer_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: pgbouncer_exporter -> 10.100.1.192:9127
[OK]   ✓ Registered pgbouncer_exporter scrape target: 10.100.1.192:9127
[INFO]   Target file: /etc/prometheus/obs-iam-identity-universe-main-dev/targets/pgbouncer_exporter.yml
[OK]   ✓ pgbouncer_exporter registered as Prometheus scrape target

[INFO] 4️⃣ Registering nodes to monitoring database...
[INFO] PostgreSQL key permissions set for www-data access: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/postgres.key
[INFO] Registering PostgreSQL coordinator to monitoring dashboard...
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        iam-identity-universe-main-dev-postgresql-coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.189
[INFO]   Port:              5432
[INFO]   FQDN:              db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: f0341e35-122d-498a-a5e5-cf38a434b79d
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✓ PostgreSQL coordinator registered
[INFO] Registering PgBouncer to monitoring dashboard...
[INFO]   FQDN: db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com, IP: 10.100.1.192, Port: 6432
[INFO]   Key permissions set for www-data access
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        iam-identity-universe-main-dev-pgbouncer
[INFO]   Identifier Parent: pooling
[INFO]   IP:                10.100.1.192
[INFO]   Port:              6432
[INFO]   FQDN:              db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 2337a090-f7f6-4571-84b2-d5854f13549c
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✓ PgBouncer registered

[INFO] 5️⃣ Creating PgBouncer professional monitoring rules...
[INFO] Creating PgBouncer recording rules...
[OK]   ✓ PgBouncer recording rules created
[INFO] Creating PgBouncer alert rules with runbook URLs...
[OK]   ✓ PgBouncer alert rules with runbook URLs created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ PostgreSQL & PgBouncer Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] postgres_exporter: http://localhost:9187/metrics
[INFO] pgbouncer_exporter: http://localhost:9127/metrics
[INFO] Prometheus: https://metrics-iam-identity-universe-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-iam-identity-universe-main-dev.fastorder.com
[INFO] 
[INFO] PgBouncer Monitoring:
[INFO]   • Recording rules: /etc/prometheus/obs-iam-identity-universe-main-dev/rules/pgbouncer_recording_rules.yml
[INFO]   • Alert rules: /etc/prometheus/obs-iam-identity-universe-main-dev/rules/pgbouncer_alerts.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up monitoring for 1 worker(s) and 1 standby(s) per worker...
[INFO] Setting up monitoring for: worker-01
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 PostgreSQL Monitoring Integration for iam-identity-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-iam-identity-universe-main-dev
[OK]   Observability cell endpoints registered for iam-identity-universe-main-dev
[OK]   ✓ Observability cell is ready

[INFO] ✓ Using private IP for metrics: 10.100.1.189
[INFO] 2️⃣ Setting up postgres_exporter integration...
[INFO] Checking observability cell readiness: obs-iam-identity-universe-main-dev
[OK]   Observability cell endpoints registered for iam-identity-universe-main-dev
[INFO] Setting up postgres_exporter for iam-identity-universe-main-dev
[2026-01-02 04:37:50 UTC] USER=www-data EUID=0 PID=1425023 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-iam-identity-universe-main-dev.yaml /etc/prometheus/postgres_exporter_queries-iam-identity-universe-main-dev.yaml
[2026-01-02 04:37:50 UTC] USER=www-data EUID=0 PID=1425032 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-iam-identity-universe-main-dev.yaml
[2026-01-02 04:37:50 UTC] USER=www-data EUID=0 PID=1425041 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-iam-identity-universe-main-dev.yaml
[OK]   Custom queries file created at /etc/prometheus/postgres_exporter_queries-iam-identity-universe-main-dev.yaml
[OK]   postgres_exporter already running with custom queries for iam-identity-universe-main-dev
[OK]   ✓ postgres_exporter integration complete
[INFO] Registering postgres_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: postgres_exporter -> 10.100.1.189:9187
[OK]   ✓ Registered postgres_exporter scrape target: 10.100.1.189:9187
[INFO]   Target file: /etc/prometheus/obs-iam-identity-universe-main-dev/targets/postgres_exporter.yml
[OK]   ✓ postgres_exporter registered as Prometheus scrape target

[INFO] 3️⃣ Setting up pgbouncer_exporter integration...
[INFO] PgBouncer FQDN found in /etc/hosts: db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.192
[INFO] PgBouncer detected: db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com:6432
[OK]   ✓ pgbouncer_exporter already installed
[INFO] Getting pgbouncer_admin password (SERVICE=iam-identity, ZONE=universe)
[OK]   ✓ pgbouncer_admin password retrieved (24 chars)
[INFO] Using pgbouncer certs from: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer
[INFO] Creating pgbouncer_exporter systemd service...
[OK]   ✓ pgbouncer_exporter service file created
[INFO] Starting pgbouncer_exporter service...
[2026-01-02 04:37:52 UTC] USER=www-data EUID=0 PID=1425087 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:37:52 UTC] USER=www-data EUID=0 PID=1425132 ACTION=passthru ARGS=systemctl enable pgbouncer_exporter-iam-identity-universe-main-dev.service
[2026-01-02 04:37:53 UTC] USER=www-data EUID=0 PID=1425177 ACTION=passthru ARGS=systemctl restart pgbouncer_exporter-iam-identity-universe-main-dev.service
[OK]   ✓ pgbouncer_exporter service running
[INFO] Registering pgbouncer_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: pgbouncer_exporter -> 10.100.1.192:9127
[OK]   ✓ Registered pgbouncer_exporter scrape target: 10.100.1.192:9127
[INFO]   Target file: /etc/prometheus/obs-iam-identity-universe-main-dev/targets/pgbouncer_exporter.yml
[OK]   ✓ pgbouncer_exporter registered as Prometheus scrape target

[INFO] 4️⃣ Registering nodes to monitoring database...
[INFO] PostgreSQL key permissions set for www-data access: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/postgres.key
[INFO] Registering PostgreSQL worker-01 to monitoring dashboard...
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        iam-identity-universe-main-dev-postgresql-worker-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.189
[INFO]   Port:              5432
[INFO]   FQDN:              db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 61988be3-ee40-48fd-9f9f-4fc4d252662a
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✓ PostgreSQL worker-01 registered
[INFO] Registering PgBouncer to monitoring dashboard...
[INFO]   FQDN: db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com, IP: 10.100.1.192, Port: 6432
[INFO]   Key permissions set for www-data access
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        iam-identity-universe-main-dev-pgbouncer
[INFO]   Identifier Parent: pooling
[INFO]   IP:                10.100.1.192
[INFO]   Port:              6432
[INFO]   FQDN:              db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 2337a090-f7f6-4571-84b2-d5854f13549c
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✓ PgBouncer registered

[INFO] 5️⃣ Creating PgBouncer professional monitoring rules...
[INFO] Creating PgBouncer recording rules...
[OK]   ✓ PgBouncer recording rules created
[INFO] Creating PgBouncer alert rules with runbook URLs...
[OK]   ✓ PgBouncer alert rules with runbook URLs created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ PostgreSQL & PgBouncer Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] postgres_exporter: http://localhost:9187/metrics
[INFO] pgbouncer_exporter: http://localhost:9127/metrics
[INFO] Prometheus: https://metrics-iam-identity-universe-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-iam-identity-universe-main-dev.fastorder.com
[INFO] 
[INFO] PgBouncer Monitoring:
[INFO]   • Recording rules: /etc/prometheus/obs-iam-identity-universe-main-dev/rules/pgbouncer_recording_rules.yml
[INFO]   • Alert rules: /etc/prometheus/obs-iam-identity-universe-main-dev/rules/pgbouncer_alerts.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up monitoring for standby: worker-01-standby-01
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 PostgreSQL Monitoring Integration for iam-identity-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-iam-identity-universe-main-dev
[OK]   Observability cell endpoints registered for iam-identity-universe-main-dev
[OK]   ✓ Observability cell is ready

[INFO] ✓ Using private IP for metrics: 10.100.1.189
[INFO] 2️⃣ Setting up postgres_exporter integration...
[INFO] Checking observability cell readiness: obs-iam-identity-universe-main-dev
[OK]   Observability cell endpoints registered for iam-identity-universe-main-dev
[INFO] Setting up postgres_exporter for iam-identity-universe-main-dev
[2026-01-02 04:37:58 UTC] USER=www-data EUID=0 PID=1425369 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-iam-identity-universe-main-dev.yaml /etc/prometheus/postgres_exporter_queries-iam-identity-universe-main-dev.yaml
[2026-01-02 04:37:58 UTC] USER=www-data EUID=0 PID=1425378 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-iam-identity-universe-main-dev.yaml
[2026-01-02 04:37:58 UTC] USER=www-data EUID=0 PID=1425387 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-iam-identity-universe-main-dev.yaml
[OK]   Custom queries file created at /etc/prometheus/postgres_exporter_queries-iam-identity-universe-main-dev.yaml
[OK]   postgres_exporter already running with custom queries for iam-identity-universe-main-dev
[OK]   ✓ postgres_exporter integration complete
[INFO] Registering postgres_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: postgres_exporter -> 10.100.1.189:9187
[OK]   ✓ Registered postgres_exporter scrape target: 10.100.1.189:9187
[INFO]   Target file: /etc/prometheus/obs-iam-identity-universe-main-dev/targets/postgres_exporter.yml
[OK]   ✓ postgres_exporter registered as Prometheus scrape target

[INFO] 3️⃣ Setting up pgbouncer_exporter integration...
[INFO] PgBouncer FQDN found in /etc/hosts: db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.192
[INFO] PgBouncer detected: db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com:6432
[OK]   ✓ pgbouncer_exporter already installed
[INFO] Getting pgbouncer_admin password (SERVICE=iam-identity, ZONE=universe)
[OK]   ✓ pgbouncer_admin password retrieved (24 chars)
[INFO] Using pgbouncer certs from: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/pgbouncer
[INFO] Creating pgbouncer_exporter systemd service...
[OK]   ✓ pgbouncer_exporter service file created
[INFO] Starting pgbouncer_exporter service...
[2026-01-02 04:38:00 UTC] USER=www-data EUID=0 PID=1425436 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:38:01 UTC] USER=www-data EUID=0 PID=1425481 ACTION=passthru ARGS=systemctl enable pgbouncer_exporter-iam-identity-universe-main-dev.service
[2026-01-02 04:38:01 UTC] USER=www-data EUID=0 PID=1425542 ACTION=passthru ARGS=systemctl restart pgbouncer_exporter-iam-identity-universe-main-dev.service
[OK]   ✓ pgbouncer_exporter service running
[INFO] Registering pgbouncer_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: pgbouncer_exporter -> 10.100.1.192:9127
[OK]   ✓ Registered pgbouncer_exporter scrape target: 10.100.1.192:9127
[INFO]   Target file: /etc/prometheus/obs-iam-identity-universe-main-dev/targets/pgbouncer_exporter.yml
[OK]   ✓ pgbouncer_exporter registered as Prometheus scrape target

[INFO] 4️⃣ Registering nodes to monitoring database...
[INFO] Registering PostgreSQL worker-01-standby-01 to monitoring dashboard...
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        iam-identity-universe-main-dev-postgresql-worker-01-standby-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.189
[INFO]   Port:              5432
[INFO]   FQDN:              db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 1a746118-a820-4708-bea9-85ec6c5e58dd
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✓ PostgreSQL worker-01-standby-01 registered
[INFO] Registering PgBouncer to monitoring dashboard...
[INFO]   FQDN: db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com, IP: 10.100.1.192, Port: 6432
[INFO]   Key permissions set for www-data access
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        iam-identity-universe-main-dev-pgbouncer
[INFO]   Identifier Parent: pooling
[INFO]   IP:                10.100.1.192
[INFO]   Port:              6432
[INFO]   FQDN:              db-iam-identity-universe-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 2337a090-f7f6-4571-84b2-d5854f13549c
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✓ PgBouncer registered

[INFO] 5️⃣ Creating PgBouncer professional monitoring rules...
[INFO] Creating PgBouncer recording rules...
[OK]   ✓ PgBouncer recording rules created
[INFO] Creating PgBouncer alert rules with runbook URLs...
[OK]   ✓ PgBouncer alert rules with runbook URLs created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ PostgreSQL & PgBouncer Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] postgres_exporter: http://localhost:9187/metrics
[INFO] pgbouncer_exporter: http://localhost:9127/metrics
[INFO] Prometheus: https://metrics-iam-identity-universe-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-iam-identity-universe-main-dev.fastorder.com
[INFO] 
[INFO] PgBouncer Monitoring:
[INFO]   • Recording rules: /etc/prometheus/obs-iam-identity-universe-main-dev/rules/pgbouncer_recording_rules.yml
[INFO]   • Alert rules: /etc/prometheus/obs-iam-identity-universe-main-dev/rules/pgbouncer_alerts.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ Monitoring setup completed for coordinator, workers, and standbys

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 12-setup-offsite-backup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Setting up offsite backup repository for iam-identity-universe-main-dev...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Offsite Backup Repository Setup (repo2)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📋 OFFSITE BACKUP INFORMATION
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Why Offsite Backups?
[INFO]   ✓ Disaster recovery resilience (datacenter loss, hardware failure)
[INFO]   ✓ Protection against local corruption or ransomware
[INFO]   ✓ Compliance requirements (geographic redundancy)
[INFO]   ✓ Long-term archival with cost-effective storage tiers

[WARN] ⚠️  Offsite backup (repo2) is NOT ENABLED
[WARN]    Using local backups only (repo1)

[INFO] Configuration Example Location:
[INFO]   📄 /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example

[INFO] Supported Storage Backends:
[INFO]   • AWS S3 (standard, multi-region)
[INFO]   • AWS S3 Glacier (low-cost archival)
[INFO]   • MinIO (self-hosted S3-compatible)
[INFO]   • Google Cloud Storage (via S3 compatibility)
[INFO]   • Azure Blob Storage (via S3 compatibility)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📝 SETUP INSTRUCTIONS
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Step 1: Review the example configuration
[INFO]   cat /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example

[INFO] Step 2: Prepare S3 bucket and credentials
[INFO]   • Create S3 bucket (or MinIO bucket)
[INFO]   • Create IAM user with S3 permissions (PutObject, GetObject, DeleteObject, ListBucket)
[INFO]   • Note: Access Key ID and Secret Access Key

[INFO] Step 3: Add repo2 configuration to /etc/pgbackrest/pgbackrest.conf
[INFO]   • Copy repo2-* settings from example to [global] section
[INFO]   • Replace placeholders (bucket name, access keys, region)
[INFO]   • Note: Use same cipher key as repo1, or generate separate key for repo2

[INFO] Step 4: Initialize repo2 stanzas
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=iam-identity-universe-main-dev-coordinator stanza-create --repo=2
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=iam-identity-universe-main-dev-worker-01 stanza-create --repo=2
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=iam-identity-universe-main-dev-worker-02 stanza-create --repo=2

[INFO] Step 5: Verify repo2 configuration
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=iam-identity-universe-main-dev-coordinator check --repo=2

[INFO] Step 6: Take initial full backup to repo2
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=iam-identity-universe-main-dev-coordinator --repo=2 --type=full backup

[INFO] Step 7: Update backup automation to include repo2
[INFO]   • Edit: /usr/local/bin/pgbackrest-full-backup-iam-identity-universe-main-dev.sh
[INFO]   • Change: pgbackrest backup to pgbackrest --repo=1,2 backup
[INFO]   • Or: Add separate cron for repo2 backups

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🧪 TESTING
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] After configuration, run:
[INFO]   ./08-setup-offsite-backup.sh test

[INFO] This will verify:
[INFO]   ✓ S3 connectivity
[INFO]   ✓ Stanza initialization
[INFO]   ✓ Test backup and restore from repo2

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 💡 COST OPTIMIZATION
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] AWS S3 Lifecycle Policies (transition to cheaper storage):
[INFO]   • 0-30 days:   S3 Standard (~$0.023/GB/month)
[INFO]   • 30-90 days:  S3 Standard-IA (~$0.0125/GB/month)
[INFO]   • 90+ days:    S3 Glacier (~$0.004/GB/month)

[INFO] Estimated costs for 100GB backups:
[INFO]   • All Standard:     ~$2.30/month
[INFO]   • With lifecycle:   ~$1.20/month (48% savings)


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 13-setup-monitoring-alerts.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Setting up backup monitoring and alerting for iam-identity-universe-main-dev...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Monitoring and Alerting Configuration
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] This will set up monitoring for:
  • Backup failures (cron job failures)
  • WAL archiving backlog (>100 files)
  • Repository disk space (<20% free)
  • Backup age (>25 hours)

[INFO] No alert email configured (set ALERT_EMAIL environment variable)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Creating monitoring directories...
[2026-01-02 04:38:10 UTC] USER=www-data EUID=0 PID=1425747 ACTION=fsop ARGS=mkdir -p /opt/pgbackrest-monitoring
[2026-01-02 04:38:10 UTC] USER=www-data EUID=0 PID=1425756 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest-monitoring
[2026-01-02 04:38:10 UTC] USER=www-data EUID=0 PID=1425765 ACTION=fsop ARGS=chmod 777 /opt/pgbackrest-monitoring
[2026-01-02 04:38:10 UTC] USER=www-data EUID=0 PID=1425774 ACTION=fsop ARGS=chmod 777 /var/log/pgbackrest-monitoring
[2026-01-02 04:38:10 UTC] USER=www-data EUID=0 PID=1425783 ACTION=fsop ARGS=chown postgres:postgres /opt/pgbackrest-monitoring
[2026-01-02 04:38:10 UTC] USER=www-data EUID=0 PID=1425792 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest-monitoring
[INFO] ✅ Directories created

[INFO] 2️⃣ Creating alert helper script...
[2026-01-02 04:38:10 UTC] USER=www-data EUID=0 PID=1425812 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/send-alert.sh
[INFO] ✅ Alert helper created

[INFO] 3️⃣ Creating WAL queue monitoring script...
[2026-01-02 04:38:10 UTC] USER=www-data EUID=0 PID=1425832 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-wal-queue.sh
[INFO] ✅ WAL queue monitor created

[INFO] 4️⃣ Creating backup age monitoring script...
[2026-01-02 04:38:10 UTC] USER=www-data EUID=0 PID=1425852 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-backup-age.sh
[INFO] ✅ Backup age monitor created

[INFO] 5️⃣ Creating repository disk space monitoring script...
[2026-01-02 04:38:10 UTC] USER=www-data EUID=0 PID=1425871 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-repo-space.sh
[INFO] ✅ Disk space monitor created

[INFO] 6️⃣ Creating backup failure detection script...
[2026-01-02 04:38:11 UTC] USER=www-data EUID=0 PID=1425891 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-backup-failures.sh
[INFO] ✅ Backup failure detector created

[INFO] 7️⃣ Creating master monitoring script...
[2026-01-02 04:38:11 UTC] USER=www-data EUID=0 PID=1425909 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO] ✅ Master monitoring script created

[INFO] 8️⃣ Installing mailutils for email alerts...
[INFO] ✅ mailutils already installed

[INFO] 9️⃣ Installing jq for JSON parsing...
[INFO] ✅ jq already installed

[INFO] 🔟 Setting up monitoring cron jobs...
[2026-01-02 04:38:11 UTC] USER=www-data EUID=0 PID=1425927 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-monitoring-iam-identity-universe-main-dev
[INFO] ✅ Monitoring cron jobs configured
[INFO]    Checks run every 15 minutes

[INFO] 1️⃣1️⃣ Creating monitoring dashboard...
[2026-01-02 04:38:11 UTC] USER=www-data EUID=0 PID=1425947 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/dashboard.sh
[INFO] ✅ Monitoring dashboard created

[INFO] 1️⃣2️⃣ Running initial monitoring check...

[2026-01-02 04:38:11 UTC] USER=www-data EUID=0 PID=1425956 ACTION=passthru ARGS=bash /opt/pgbackrest-monitoring/run-all-checks.sh
grep: write error: Broken pipe

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup monitoring setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Monitoring Configuration:
[INFO]   Alert Email:        
[INFO]   Slack Webhook:      Not configured

[INFO] Monitoring Checks:
[INFO]   • WAL Queue:        Every 15 minutes (threshold: >100 files)
[INFO]   • Backup Age:       Every 15 minutes (threshold: >25 hours)
[INFO]   • Disk Space:       Every 15 minutes (threshold: <20% free)
[INFO]   • Backup Failures:  Every 15 minutes (log analysis)

[INFO] Scripts Created:
[INFO]   Monitoring dir:     /opt/pgbackrest-monitoring
[INFO]   Log dir:            /var/log/pgbackrest-monitoring
[INFO]   Dashboard:          /opt/pgbackrest-monitoring/dashboard.sh
[INFO]   Master check:       /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO]   Alert sender:       /opt/pgbackrest-monitoring/send-alert.sh

[INFO] Useful Commands:
[INFO]   View dashboard:     /usr/local/bin/fastorder-provisioning-wrapper.sh /opt/pgbackrest-monitoring/dashboard.sh
[INFO]   Run checks now:     /usr/local/bin/fastorder-provisioning-wrapper.sh /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO]   View alerts:        tail -f /var/log/pgbackrest-monitoring/alerts.log
[INFO]   View monitoring:    tail -f /var/log/pgbackrest-monitoring/monitoring.log

[INFO] Cron Schedule:
[INFO]   All checks:         Every 15 minutes
[INFO]   Log rotation:       Weekly (keep 7 days)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 14-vault-cipher-key.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] ✅ Using permanent AWS credentials from /home/ab/.aws/credentials [default] profile
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔐 PostgreSQL Cipher Key Vaulting
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Environment:       iam-identity-universe-main-dev
[INFO]   AWS Region:        me-central-1
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣  Verifying AWS setup...
[INFO] ✅ AWS authentication successful

[INFO] 2️⃣  Verifying cipher key...
[INFO] ✅ Cipher key found
[INFO]    Location: /etc/pgbackrest/.cipher-key-iam-identity-universe-main-dev
[INFO]    Hash (MD5): 0389f8f5033e04e65c22142f67d75be6
[INFO]    Size: 203 bytes

[INFO] 3️⃣  Vaulting cipher key to AWS Secrets Manager...
[INFO]    Secret name: fastorder/db/iam-identity/universe/main/dev/postgresql/pgbackrest/cipher-key
[INFO]    Creating new secret...
[INFO] ✅ Cipher key stored in AWS Secrets Manager
[INFO]    Verifying storage...
[INFO] ✅ Verification successful - key matches

[INFO] 4️⃣  Creating local encrypted backup...
[2026-01-02 04:38:21 UTC] USER=www-data EUID=0 PID=1426173 ACTION=fsop ARGS=mv /tmp/cipher-key-backup-1426083.enc /root/.pgbackrest-cipher-key-iam-identity-universe-main-dev.enc
[2026-01-02 04:38:21 UTC] USER=www-data EUID=0 PID=1426182 ACTION=fsop ARGS=chmod 600 /root/.pgbackrest-cipher-key-iam-identity-universe-main-dev.enc
[2026-01-02 04:38:21 UTC] USER=www-data EUID=0 PID=1426201 ACTION=fsop ARGS=chmod 600 /root/.pgbackrest-cipher-key-passphrase-iam-identity-universe-main-dev.txt
[INFO] ✅ Local encrypted backup created
[INFO]    Backup file: /root/.pgbackrest-cipher-key-iam-identity-universe-main-dev.enc
[INFO]    Passphrase: /root/.pgbackrest-cipher-key-passphrase-iam-identity-universe-main-dev.txt

[INFO] 5️⃣  Vaulting backup passphrase...
[INFO] ✅ Backup passphrase vaulted

[INFO] 6️⃣  Creating recovery documentation...
[2026-01-02 04:38:25 UTC] USER=www-data EUID=0 PID=1426240 ACTION=fsop ARGS=chmod 640 /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_iam-identity-universe-main-dev.md
[2026-01-02 04:38:26 UTC] USER=www-data EUID=0 PID=1426251 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_iam-identity-universe-main-dev.md
[INFO] ✅ Recovery documentation: /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_iam-identity-universe-main-dev.md

[INFO] 7️⃣  Storing backup metadata...
[INFO] ✅ Backup metadata stored in AWS Secrets Manager
[INFO]    Secret: fastorder/db/iam-identity/universe/main/dev/postgresql/backup/metadata-20260102-043826

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Cipher Key Vaulting Complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO]   Environment:          iam-identity-universe-main-dev
[INFO]   Key Hash:             0389f8f5033e04e65c22142f67d75be6

[INFO] AWS Secrets:
[INFO]   Cipher Key:           fastorder/db/iam-identity/universe/main/dev/postgresql/pgbackrest/cipher-key
[INFO]   Passphrase:           fastorder/db/iam-identity/universe/main/dev/postgresql/pgbackrest/cipher-key-passphrase
[INFO]   Backup Metadata:      fastorder/db/iam-identity/universe/main/dev/postgresql/backup/metadata-20260102-043826

[INFO] Local Backups:
[INFO]   Encrypted File:       /root/.pgbackrest-cipher-key-iam-identity-universe-main-dev.enc
[INFO]   Passphrase File:      /root/.pgbackrest-cipher-key-passphrase-iam-identity-universe-main-dev.txt

[INFO] Recovery Doc:           /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_iam-identity-universe-main-dev.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Skipping 15-backup-restore-test.sh (test script - set RUN_TESTS=true to enable)
[INFO] Skipping 16-test-recovery.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 17-verification.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)

[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] PostgreSQL Production Readiness Verification
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] 
[INFO] This script verifies 3 CRITICAL checks for production readiness:
[INFO]   1. Citus Cluster Operational (coordinator + workers)
[INFO]   2. SSL/TLS Enforced (certificates valid, connections secure)
[INFO]   3. Coordinator Backups Configured (pgBackRest functional)
[INFO] 
[INFO] 📖 Documentation: /tmp/VERIFICATION_RUNBOOK.md
[INFO] 🔐 Security: Uses sudo for certificate checks (maintains strict permissions)
[INFO] 📊 Exit Code: 0 = production ready, 1 = critical checks failed
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🕐 Ensuring all PostgreSQL services are ready...
[ OK ] ✅ All PostgreSQL services are ready

[INFO] 🔍 Starting PostgreSQL verification...
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Citus: yes

[INFO] Citus mode ENABLED
[INFO] → Coordinator + 1 worker(s) + 3 HA node(s) per worker

[INFO] Verifying 1 worker(s)...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying: worker-01 (type: worker-01)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Starting PostgreSQL verification for iam-identity-universe-main-dev-worker-01...

[INFO] 1️⃣ Checking systemd service status...
[OK]   ✅ Service postgresql@iam-identity-universe-main-dev-worker-01.service is active

[INFO] 2️⃣ Checking PostgreSQL process...
[OK]   ✅ PostgreSQL process is running

[INFO] 3️⃣ Checking socket directory...
[OK]   ✅ Socket directory exists: /var/run/postgresql-iam-identity-universe-main-dev-worker-01
total 4
drwxrwsr-x  2 postgres postgres   80 Jan  2 04:30 .
drwxr-xr-x 49 root     root     1320 Jan  2 04:36 ..
srwxrwxrwx  1 postgres postgres    0 Jan  2 04:30 .s.PGSQL.5432
-rw-------  1 postgres postgres  149 Jan  2 04:30 .s.PGSQL.5432.lock

[INFO] 4️⃣ Testing connection via Unix socket...
[OK]   ✅ Socket connection successful
                                                              version                                                              
-----------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.6 (Ubuntu 17.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

[INFO] 5️⃣ Checking SSL certificates...
[2026-01-02 04:38:30 UTC] USER=www-data EUID=0 PID=1426379 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.crt
[OK]   ✅ Server certificate exists: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.crt
[2026-01-02 04:38:30 UTC] USER=www-data EUID=0 PID=1426388 ACTION=fsop ARGS=openssl x509 -in /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/server.crt -noout -checkend 86400
Certificate will not expire
[OK]   ✅ Server certificate is valid
[2026-01-02 04:38:31 UTC] USER=www-data EUID=0 PID=1426397 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt
[OK]   ✅ CA certificate exists: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01/ca.crt
[INFO] ℹ️  Client certificates not found at /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/postgres.crt
[INFO]     (This is OK if using password authentication)

[INFO] 6️⃣ Checking PostgreSQL settings...
[OK]   ✅ SSL is enabled worker-01 worker-01
[OK]   ✅ Max connections: 100
[OK]   ✅ Listen addresses: 10.100.1.190
[OK]   ✅ WAL level: logical
[OK]   ✅ Shared preload libraries: shared_preload_libraries

[INFO] 7️⃣ Checking replication configuration...
[INFO] ℹ️  No synchronous standbys configured (single node or async replication)
[INFO] Checking replication slots...
      slot_name       | slot_type | active | restart_lsn 
----------------------+-----------+--------+-------------
 worker_01_standby_01 | physical  | f      | 
(1 row)
[OK]   ✅ Replication slot naming uses underscores (correct)
[INFO] Checking active replication connections...
 application_name | client_addr | state | sync_state 
------------------+-------------+-------+------------
(0 rows)
[INFO] ℹ️  No active replication connections
[INFO] ℹ️  This is a PRIMARY node (no standby.signal)

[INFO] 8️⃣ Checking pg_hba.conf for replication rules...
[WARN] ⚠️ pg_hba.conf not found at /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01/pg_hba.conf

[INFO] 9️⃣ Checking Citus configuration...
[OK]   ✅ Citus extension is installed
[OK]   ✅ Citus version: Citus 13.2.0
[OK]   ✅ max_prepared_transactions: 100 (adequate for Citus)
[INFO] Citus active worker nodes:
                              node_name                               | node_port 
----------------------------------------------------------------------+-----------
 db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com |      5432
(1 row)



[INFO] 🔟 Checking data directory...
[OK]   ✅ Data directory exists: /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01
[OK]   ✅ Data directory size: 4.0K

[INFO] 1️⃣1️⃣ Checking PgBouncer configuration...
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/iam-identity-universe-main-dev/pgbouncer.ini' as root on web-03.
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/%i/pgbouncer.ini' as root on web-03.
[OK]   ✅ PgBouncer is installed
[INFO]    Version: 1.24.1
2.1.12-stable
c-ares
OpenSSL
yes
Failed to print table: Broken pipe
[INFO] ℹ️  PgBouncer service not configured for this environment

[INFO] 1️⃣2️⃣ Enhanced PgBouncer Admin Console Verification...
[INFO] ℹ️  PgBouncer password not found

[INFO] 1️⃣3️⃣ Replicator User Connection Verification...
[INFO] Found 1 replication slot(s) - verifying replicator connectivity...
[WARN] ⚠️ Replicator certificates not found at /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01
[INFO]    Expected files:
[INFO]    - /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/root.crt
[INFO]    - /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.crt
[INFO]    - /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01/replicator.key
[INFO] Checking pg_hba.conf replicator rules...
[OK]   ✅ Replicator HBA rules found:
 line_number |  type   |   database    |  user_name   |   address    |  auth_method  | options | error 
-------------+---------+---------------+--------------+--------------+---------------+---------+-------
          20 | hostssl | {replication} | {replicator} | 10.100.1.191 | scram-sha-256 |         | 
          21 | hostssl | {replication} | {replicator} | 10.100.1.190 | scram-sha-256 |         | 
(2 rows)
[INFO] Checking active replicator connections in pg_stat_activity...
[WARN] ⚠️ No active replicator connections in pg_stat_activity
[WARN]    This is expected if standbys are not currently connected

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ PostgreSQL verification completed successfully!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Instance:       iam-identity-universe-main-dev-worker-01
[INFO] Service:        postgresql@iam-identity-universe-main-dev-worker-01.service
[INFO] Socket:         /var/run/postgresql-iam-identity-universe-main-dev-worker-01
[INFO] Data Directory: /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01
[INFO] Hostname:       db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com
[INFO] Port:           5432
[INFO] SSL:            on
[INFO] WAL Level:      logical
[INFO] Citus:          yes
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 💡 OPTIMIZATION OPPORTUNITIES (Optional Enhancements)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1. Review connection limits for production workload
[INFO]    🔌 Current: max_connections = 100 (PostgreSQL default)
[INFO]    💡 Consider: Increasing to 200-500 for production applications
[INFO]    ⚙️  Alternative: Use PgBouncer connection pooling (lower PostgreSQL limit, higher client capacity)
[INFO]    🔧 Action: Adjust max_connections in postgresql.conf based on workload analysis
[INFO]    ⚠️  Note: Each connection consumes ~10MB RAM; tune based on available memory
[INFO]    📚 Docs: https://www.postgresql.org/docs/current/runtime-config-connection.html

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ℹ️  These are optional enhancements for production-scale deployments
[INFO] ℹ️  Current configuration is fully functional and ready for production
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] ✅ Verification passed for worker-01

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying: worker-01-standby-01 (type: worker-01)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Starting PostgreSQL verification for iam-identity-universe-main-dev-worker-01-standby-01...

[INFO] 1️⃣ Checking systemd service status...
[OK]   ✅ Service postgresql@iam-identity-universe-main-dev-worker-01-standby-01.service is active

[INFO] 2️⃣ Checking PostgreSQL process...
[OK]   ✅ PostgreSQL process is running

[INFO] 3️⃣ Checking socket directory...
[OK]   ✅ Socket directory exists: /var/run/postgresql-iam-identity-universe-main-dev-worker-01-standby-01
total 4
drwxrwsr-x  2 postgres postgres   80 Jan  2 04:34 .
drwxr-xr-x 49 root     root     1320 Jan  2 04:36 ..
srwxrwxrwx  1 postgres postgres    0 Jan  2 04:34 .s.PGSQL.5432
-rw-------  1 postgres postgres  171 Jan  2 04:34 .s.PGSQL.5432.lock

[INFO] 4️⃣ Testing connection via Unix socket...
[OK]   ✅ Socket connection successful
                                                              version                                                              
-----------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.6 (Ubuntu 17.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

[INFO] 5️⃣ Checking SSL certificates...
[2026-01-02 04:38:48 UTC] USER=www-data EUID=0 PID=1426837 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt
[OK]   ✅ Server certificate exists: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt
[2026-01-02 04:38:48 UTC] USER=www-data EUID=0 PID=1426846 ACTION=fsop ARGS=openssl x509 -in /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/server.crt -noout -checkend 86400
Certificate will not expire
[OK]   ✅ Server certificate is valid
[2026-01-02 04:38:48 UTC] USER=www-data EUID=0 PID=1426855 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
[OK]   ✅ CA certificate exists: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/worker-01-standby-01/ca.crt
[INFO] ℹ️  Client certificates not found at /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/worker-01-standby-01/postgres.crt
[INFO]     (This is OK if using password authentication)

[INFO] 6️⃣ Checking PostgreSQL settings...
[OK]   ✅ SSL is enabled worker-01 worker-01-standby-01
[OK]   ✅ Max connections: 100
[OK]   ✅ Listen addresses: 10.100.1.191
[OK]   ✅ WAL level: logical
[OK]   ✅ Shared preload libraries: shared_preload_libraries

[INFO] 7️⃣ Checking replication configuration...
[INFO] ℹ️  No synchronous standbys configured (single node or async replication)
[INFO] Checking replication slots...
 slot_name | slot_type | active | restart_lsn 
-----------+-----------+--------+-------------
(0 rows)
[OK]   ✅ Replication slot naming uses underscores (correct)
[INFO] Checking active replication connections...
 application_name | client_addr | state | sync_state 
------------------+-------------+-------+------------
(0 rows)
[INFO] ℹ️  No active replication connections
[INFO] ℹ️  This is a PRIMARY node (no standby.signal)

[INFO] 8️⃣ Checking pg_hba.conf for replication rules...
[WARN] ⚠️ pg_hba.conf not found at /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01-standby-01/pg_hba.conf

[INFO] 9️⃣ Checking Citus configuration...
[INFO] ℹ️  Citus extension not needed on standby (will inherit from primary via replication)

[INFO] 🔟 Checking data directory...
[OK]   ✅ Data directory exists: /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01-standby-01
[OK]   ✅ Data directory size: 4.0K

[INFO] 1️⃣1️⃣ Checking PgBouncer configuration...
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/iam-identity-universe-main-dev/pgbouncer.ini' as root on web-03.
Failed to print table: Broken pipe
[OK]   ✅ PgBouncer is installed
[INFO]    Version: 1.24.1
2.1.12-stable
c-ares
OpenSSL
yes
[OK]   ✅ PgBouncer service is active: pgbouncer@iam-identity-universe-main-dev.service
[WARN] ⚠️ PgBouncer IP service is not active: pgbouncer-ip@iam-identity-universe-main-dev.service
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/iam-identity-universe-main-dev/pgbouncer.ini' as root on web-03.
[WARN] ⚠️ PgBouncer config not found: /etc/pgbouncer/iam-identity-universe-main-dev/pgbouncer.ini
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt' as root on web-03.
[WARN] ⚠️ PgBouncer auth file not found: /etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt
[OK]   ✅ PgBouncer is listening on port 6432

[INFO] 1️⃣2️⃣ Enhanced PgBouncer Admin Console Verification...
[INFO] ℹ️  PgBouncer password not found

[INFO] 1️⃣3️⃣ Replicator User Connection Verification...
[INFO] ℹ️  No replication slots configured - skipping replicator verification

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ PostgreSQL verification completed successfully!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Instance:       iam-identity-universe-main-dev-worker-01-standby-01
[INFO] Service:        postgresql@iam-identity-universe-main-dev-worker-01-standby-01.service
[INFO] Socket:         /var/run/postgresql-iam-identity-universe-main-dev-worker-01-standby-01
[INFO] Data Directory: /var/lib/postgresql/17/iam-identity-universe-main-dev/worker-01-standby-01
[INFO] Hostname:       db-iam-identity-universe-main-dev-postgresql-worker-01-standby-01.fastorder.com
[INFO] Port:           5432
[INFO] SSL:            on
[INFO] WAL Level:      logical
[INFO] Citus:          yes
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] ✅ Verification passed for worker-01-standby-01

[INFO] Skipping worker-01-standby-02 - service not configured
[INFO] Skipping worker-01-standby-03 - service not configured
[INFO] Verifying coordinator...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying: coordinator (type: coordinator)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] 🔍 Starting PostgreSQL verification for iam-identity-universe-main-dev-coordinator...

[INFO] 1️⃣ Checking systemd service status...
[OK]   ✅ Service postgresql@iam-identity-universe-main-dev-coordinator.service is active

[INFO] 2️⃣ Checking PostgreSQL process...
[OK]   ✅ PostgreSQL process is running

[INFO] 3️⃣ Checking socket directory...
[OK]   ✅ Socket directory exists: /var/run/postgresql-iam-identity-universe-main-dev-coordinator
total 4
drwxrwsr-x  2 postgres postgres   80 Jan  2 04:37 .
drwxr-xr-x 49 root     root     1320 Jan  2 04:36 ..
srwxrwxrwx  1 postgres postgres    0 Jan  2 04:37 .s.PGSQL.5432
-rw-------  1 postgres postgres  153 Jan  2 04:37 .s.PGSQL.5432.lock

[INFO] 4️⃣ Testing connection via Unix socket...
[OK]   ✅ Socket connection successful
                                                              version                                                              
-----------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.6 (Ubuntu 17.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

[INFO] 5️⃣ Checking SSL certificates...
[2026-01-02 04:39:06 UTC] USER=www-data EUID=0 PID=1427298 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt
[OK]   ✅ Server certificate exists: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt
[2026-01-02 04:39:06 UTC] USER=www-data EUID=0 PID=1427307 ACTION=fsop ARGS=openssl x509 -in /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/server.crt -noout -checkend 86400
Certificate will not expire
[OK]   ✅ Server certificate is valid
[2026-01-02 04:39:06 UTC] USER=www-data EUID=0 PID=1427316 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt
[OK]   ✅ CA certificate exists: /etc/fastorder/postgresql/certs/iam-identity-universe-main-dev/coordinator/ca.crt
[INFO] ℹ️  Client certificates not found at /home/postgres/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt
[INFO]     (This is OK if using password authentication)

[INFO] 6️⃣ Checking PostgreSQL settings...
[OK]   ✅ SSL is enabled coordinator coordinator
[OK]   ✅ Max connections: 150
[OK]   ✅ Listen addresses: 10.100.1.189
[OK]   ✅ WAL level: logical
[OK]   ✅ Shared preload libraries: shared_preload_libraries

[INFO] 7️⃣ Checking replication configuration...
[INFO] ℹ️  No synchronous standbys configured (single node or async replication)
[INFO] Checking replication slots...
 slot_name | slot_type | active | restart_lsn 
-----------+-----------+--------+-------------
(0 rows)
[OK]   ✅ Replication slot naming uses underscores (correct)
[INFO] Checking active replication connections...
 application_name | client_addr | state | sync_state 
------------------+-------------+-------+------------
(0 rows)
[INFO] ℹ️  No active replication connections
[INFO] ℹ️  This is a PRIMARY node (no standby.signal)

[INFO] 8️⃣ Checking pg_hba.conf for replication rules...
[WARN] ⚠️ pg_hba.conf not found at /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator/pg_hba.conf

[INFO] 9️⃣ Checking Citus configuration...
[OK]   ✅ Citus extension is installed
[OK]   ✅ Citus version: Citus 13.2.0
[OK]   ✅ max_prepared_transactions: 100 (adequate for Citus)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] COORDINATOR-SPECIFIC CHECKS
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking registered workers...
[INFO] ℹ️  Coordinator role verified via pg_dist_node (1 workers registered)
[INFO] Checking coordinator hostname configuration...
[OK]   ✅ Coordinator hostname: ------------------------------------------------------------------------:----------
[INFO] Checking for stuck prepared transactions...
[OK]   ✅ No stuck Citus prepared transactions
[INFO] Expected workers: 1
[INFO] Registered workers: 1
[OK]   ✅ All 1 worker(s) successfully registered
[INFO] Registered worker nodes:
                                nodename                                | nodeport | groupid | isactive | noderole | shouldhaveshards 
------------------------------------------------------------------------+----------+---------+----------+----------+------------------
 db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com |     5432 |       0 | t        | primary  | f
 db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com   |     5432 |       1 | t        | primary  | t
(2 rows)

[INFO] Note: groupid=0 is the coordinator, groupid>0 are workers

[INFO] Citus active worker nodes:
                              node_name                               | node_port 
----------------------------------------------------------------------+-----------
 db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com |      5432
(1 row)


[INFO] Verifying Citus workers...
[INFO] Checking worker: db-iam-identity-universe-main-dev-postgresql-worker-01.fastorder.com
 citus_add_node 
----------------
              2
(1 row)


[INFO] Testing Citus distributed table setup...
[INFO] Checking for blocking locks...
    SELECT pg_terminate_backend(pid)
    FROM pg_stat_activity
    WHERE pid <> pg_backend_pid()
      AND state = 'idle in transaction'
      AND query_start < NOW() - INTERVAL '30 seconds'
      AND datname = current_database();
  
 pg_terminate_backend 
----------------------
(0 rows)

[INFO] Creating demo schema (if needed)...
CREATE SCHEMA
[OK]   ✅ Demo schema ready
[INFO] Creating distributed table 'demo.events'...
CREATE TABLE
[OK]   ✅ Table is already distributed
[INFO] Inserting test data...
INSERT 0 1
[OK]   ✅ Distributed table contains 1 row(s)
[INFO] Checking shard distribution...
[OK]   ✅ Table has 1 shard(s)
[INFO] Shard placement across workers (first 10 shards):
 shardid | nodename | nodeport | shardstate 
---------+----------+----------+------------
(0 rows)
[OK]   ✅ Verified 3 shard placement(s)
[INFO] Testing query routing (EXPLAIN for user_id=42)...
[INFO]    Query plan:         QUERY PLAN        
--------------------------
 Seq Scan on events
   Filter: (user_id = 42)
(2 rows)


[INFO] 🔟 Checking data directory...
[OK]   ✅ Data directory exists: /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator
[OK]   ✅ Data directory size: 4.0K

[INFO] 1️⃣1️⃣ Checking PgBouncer configuration...
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/iam-identity-universe-main-dev/pgbouncer.ini' as root on web-03.
Failed to print table: Broken pipe
[OK]   ✅ PgBouncer is installed
[INFO]    Version: 1.24.1
2.1.12-stable
c-ares
OpenSSL
yes
[OK]   ✅ PgBouncer service is active: pgbouncer@iam-identity-universe-main-dev.service
[WARN] ⚠️ PgBouncer IP service is not active: pgbouncer-ip@iam-identity-universe-main-dev.service
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/iam-identity-universe-main-dev/pgbouncer.ini' as root on web-03.
[WARN] ⚠️ PgBouncer config not found: /etc/pgbouncer/iam-identity-universe-main-dev/pgbouncer.ini
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt' as root on web-03.
[WARN] ⚠️ PgBouncer auth file not found: /etc/pgbouncer/iam-identity-universe-main-dev/userlist.txt
[OK]   ✅ PgBouncer is listening on port 6432

[INFO] 1️⃣2️⃣ Enhanced PgBouncer Admin Console Verification...
[INFO] ℹ️  PgBouncer password not found

[INFO] 1️⃣3️⃣ Replicator User Connection Verification...
[INFO] ℹ️  No replication slots configured - skipping replicator verification

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ PostgreSQL verification completed successfully!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Instance:       iam-identity-universe-main-dev-coordinator
[INFO] Service:        postgresql@iam-identity-universe-main-dev-coordinator.service
[INFO] Socket:         /var/run/postgresql-iam-identity-universe-main-dev-coordinator
[INFO] Data Directory: /var/lib/postgresql/17/iam-identity-universe-main-dev/coordinator
[INFO] Hostname:       db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com
[INFO] Port:           5432
[INFO] SSL:            on
[INFO] WAL Level:      logical
[INFO] Citus:          yes
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Failed to print table: Broken pipe

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 💡 OPTIMIZATION OPPORTUNITIES (Optional Enhancements)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1. Enable PgBouncer connection pooling
[INFO]    📦 Benefit: Reduces connection overhead for high-concurrency workloads
[INFO]    ⚡ Use case: When facing connection exhaustion or frequent connect/disconnect cycles
[INFO]    🔧 Action: Enable and configure pgbouncer@iam-identity-universe-main-dev.service
[INFO]    📚 Docs: https://www.pgbouncer.org/config.html

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ℹ️  These are optional enhancements for production-scale deployments
[INFO] ℹ️  Current configuration is fully functional and ready for production
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] ✅ Verification passed for coordinator


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📊 PRODUCTION READINESS CHECKS (Step 04 & 05)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔍 Checking Monitoring Setup (postgres_exporter or observability cell)...

[INFO] ℹ️  Monitoring can be configured via:
[INFO]    • Local postgres_exporter (step 04-monitoring-setup.sh)
[INFO]    • Observability Cell integration (step 02-observability-cell)

[ OK ] ✅ postgres_exporter is installed
[INFO]    Version: 0.10.1-1ubuntu0.22.04.3
[ OK ] ✅ postgres_exporter-iam-identity-universe-main-dev.service is running
[WARN] ⚠️  Metrics endpoint not responding
[INFO] ℹ️  Monitoring user 'postgres_exporter' not found in PostgreSQL
[INFO]    This is expected if using observability cell remote monitoring
[INFO] ℹ️  Monitoring check passed (local or observability cell)

[INFO] 🔍 Checking Backup Setup (pgBackRest + WAL archiving)...

[ OK ] ✅ pgBackRest is installed
[INFO]    Version: pgBackRest 2.56.0
[ OK ] ✅ WAL archiving is enabled (archive_mode=on)
[ OK ] ✅ archive_command is configured for pgBackRest
[INFO]    Command: timeout 30 /usr/bin/pgbackrest --stanza=iam-identity-universe-main-dev-coordinator archive-push %p
[ OK ] ✅ pgBackRest configuration exists
[ OK ] ✅ pgBackRest stanza 'iam-identity-universe-main-dev-coordinator' is initialized
[ OK ] ✅ Backups exist (4 full backup(s))
[INFO]    Latest backup info:
                 timestamp start/stop: 2026-01-02 04:37:01+00 / 2026-01-02 04:37:10+00
                 wal start/stop: 000000010000000000000004 / 000000010000000000000004
                 database size: 37.4MB, database backup size: 37.4MB
                 repo1: backup set size: 5.7MB, backup size: 5.7MB
     
             full backup: 20260102-043722F
                 timestamp start/stop: 2026-01-02 04:37:22+00 / 2026-01-02 04:37:25+00
                 wal start/stop: 000000010000000000000007 / 000000010000000000000007
                 database size: 37.4MB, database backup size: 37.4MB
                 repo1: backup set size: 5.7MB, backup size: 5.7MB
[ OK ] ✅ Automated backup cron jobs are configured
[INFO]    Schedule:
     0 2 * * 0 root /usr/local/bin/pgbackrest-full-backup-iam-identity-universe-main-dev.sh
     0 2 * * 1-6 root /usr/local/bin/pgbackrest-diff-backup-iam-identity-universe-main-dev.sh
[ OK ] ✅ Backup directory exists: /var/lib/pgbackrest
[INFO]    Total backup size: 2.2G

[INFO] 🔍 Checking Worker Backup Coverage...

[INFO] ℹ️  Worker backups are optional for development environments
[INFO]    For production, ensure all workers have backup coverage

[INFO] Checking worker 1/1: worker-01...
[WARN] ⚠️  Worker worker-01 stanza exists but status unknown
[INFO] ℹ️  Incomplete worker backup coverage (0/1) - OK for dev

[INFO] 🔍 Checking Synchronous Replication (RPO=0)...

[INFO] ℹ️  Synchronous replication (RPO=0) is optional for development
[INFO]    For production with zero data loss requirement, enable sync replication

[INFO] ℹ️  Worker worker-01 synchronous replication NOT configured
[INFO]    └─ synchronous_commit: on
[INFO]    └─ synchronous_standby_names: 
[ OK ] ✅ All workers have synchronous replication (RPO=0)

[INFO] 🔍 Checking Connection and Memory Optimization...

[ OK ] ✅ Coordinator max_connections optimized: 150
[ OK ] ✅ Coordinator work_mem optimized: 8MB
[ OK ] ✅ Worker worker-01 max_connections optimized: 100
[ OK ] ✅ Worker worker-01 work_mem optimized: 8MB
[ OK ] ✅ All instances have optimized connection and memory settings

[INFO] 🔍 Checking Optimizations...

[ OK ] ✅ Citus coordinator host configured: db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com
[ OK ] ✅ Periodic integrity checks configured
[INFO]    └─ Daily checks: 3, Weekly verify: 3
[WARN] ⚠️  Backup schedule NOT staggered (all at :00)
[INFO]    Optimize with: ./setup/04-postgresql/steps/04-production-optimizations.sh
[2026-01-02 04:39:26 UTC] USER=www-data EUID=0 PID=1428094 ACTION=fsop ARGS=test -f /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md
[2026-01-02 04:39:27 UTC] USER=www-data EUID=0 PID=1428103 ACTION=fsop ARGS=grep -q ## Cipher Key Management /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md
[WARN] ⚠️  Cipher key management documentation missing
[INFO]    Add with: ./setup/04-postgresql/steps/04-production-optimizations.sh
[INFO] ℹ️  Offsite backup (repo2) not configured (optional for production)
[INFO]    Setup guide: ./setup/05-db/engine/postgresql/steps/14-setup-offsite-backup.sh
[WARN] ⚠️  Some production optimizations incomplete

[INFO] 🔍 Checking Citus Maintenance Daemon Health...

[INFO] Checking for stuck Citus Maintenance Daemons...
[ OK ] ✅ Citus Maintenance Daemons are healthy
[INFO] Checking for stuck distributed table operations...
[ OK ] ✅ No stuck distributed table operations
[INFO] Testing distributed table operations (10s timeout)...
[WARN] ⚠️  CRITICAL: Distributed table test TIMED OUT (10s)
[WARN]    Citus cluster is NOT operational - distributed tables cannot be created
[WARN]    This confirms maintenance daemons are stuck
[WARN]    
[WARN]    🔧 ACTION REQUIRED: Restart coordinator before using Citus
[WARN]       sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@iam-identity-universe-main-dev-coordinator.service
[INFO] Checking for lock contention...
[ OK ] ✅ No lock contention detected
[INFO] Checking for lingering prepared transactions...
[ OK ] ✅ No lingering prepared transactions

[WARN] ⚠️  Citus cluster has health issues - see warnings above
[WARN]    
[WARN]    ⚡ IMMEDIATE ACTION: Restart coordinator to restore Citus functionality
[WARN]       sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@iam-identity-universe-main-dev.service

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📋 PRODUCTION READINESS SUMMARY
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Component                 Status          Production Ready?   
───────────────────────── ─────────────── ────────────────────
Citus Cluster             ✅ Operational YES                 
High Availability         ✅ Configured  YES                 
SSL/TLS Security          ✅ Enabled     YES                 
PgBouncer                 ✅ Running     YES                 
Monitoring                ✅ Operational YES                 
Backups (Coordinator)     ✅ Configured  YES                 
Backups (Workers)         ✅ Configured  YES                 
Sync Replication (RPO=0)  ✅ Enabled     YES                 
Connection Optimization   ✅ Configured  YES                 
Optimizations             ⚠️  Incomplete OPTIONAL            

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[ OK ] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] 🎉 PRODUCTION READY: 100% (3/3 critical checks passed)
[ OK ] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ All critical components are operational and production-ready

[INFO] Next steps:
[INFO]   1. Configure Prometheus to scrape metrics: http://localhost:9197/metrics
[INFO]   2. Import Grafana dashboards for PostgreSQL + Citus monitoring
[INFO]   3. Setup alerting rules for critical metrics
[INFO]   4. Schedule regular restore drills (monthly)
[INFO]   5. Review /var/www/html/skeleton.dev.fastorder.com/fixing/scripts/PRODUCTION_READINESS.md

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✓ Verification process completed successfully
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 18-production-optimizations.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: iam-identity-universe-main-dev (svc=iam-identity zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PostgreSQL Production Optimizations
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Enable Sync Replication: --auto

[INFO] 1️⃣ Configuring Citus coordinator hostname...
[ OK ] ✅ Coordinator hostname already configured: db-iam-identity-universe-main-dev-postgresql-coordinator.fastorder.com

[INFO] 2️⃣ Configuring synchronous replication for RPO=0...
[INFO] Synchronous replication NOT enabled (use './04-production-optimizations.sh yes' to enable)
[INFO] Current configuration: async replication (RPO > 0)
[INFO] 
[INFO] To enable safely after deployment:
[INFO]   /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/enable_sync_replication_safe.sh \
[INFO]     /var/run/postgresql-iam-identity-universe-main-dev-worker-01 worker_01_standby_01

[INFO] 3️⃣ Adding periodic integrity check cron jobs...
[2026-01-02 04:39:29 UTC] USER=www-data EUID=0 PID=1428239 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-integrity-iam-identity-universe-main-dev
[ OK ] ✅ Integrity check cron jobs configured
[INFO]    Daily checks: 02:15, 03:30, 04:45 (coordinator, worker-01, worker-02)
[INFO]    Weekly verify: Sundays at same times

[INFO] 4️⃣ Updating backup schedule with staggered timing...
[ OK ] ✅ Backup schedule staggered:
[INFO]    Coordinator: 02:05 (full: Sun, diff: Mon-Sat)
[INFO]    Worker-01:   03:10 (full: Sun, diff: Mon-Sat)
[INFO]    Worker-02:   04:15 (full: Sun, diff: Mon-Sat)

[INFO] 5️⃣ Documenting cipher key backup procedures...
[2026-01-02 04:39:29 UTC] USER=www-data EUID=0 PID=1428266 ACTION=fsop ARGS=test -f /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md
Sorry, user www-data is not allowed to execute '/usr/bin/grep -q ## Cipher Key Management /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md' as root on web-03.
[ OK ] ✅ Cipher key documentation added to /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_iam-identity-universe-main-dev.md

[INFO] 6️⃣ Checking offsite backup configuration...
[INFO] ℹ️  Offsite backup (repo2) is NOT configured
[INFO]    Configuration example: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[INFO]    Setup instructions: ./setup/04-postgresql/steps/08-setup-offsite-backup.sh
[ OK ] ✅ Offsite backup example available: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Production Optimizations Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[ OK ] Applied optimizations:
[ OK ]   1. ✅ Citus coordinator hostname
[ OK ]   2. ⏭️  Synchronous replication (RPO=0)
[ OK ]   3. ✅ Periodic integrity checks (daily + weekly)
[ OK ]   4. ✅ Staggered backup schedule (reduced load spikes)
[ OK ]   5. ✅ Cipher key backup documentation
[ OK ]   6. ✅ Offsite backup (repo2) example configuration

[INFO] Next steps:
[INFO]   1. Backup cipher keys to secure vault immediately
[INFO]   2. Set up S3/MinIO for offsite backups:
[INFO]      - Instructions: ./setup/04-postgresql/steps/08-setup-offsite-backup.sh
[INFO]      - Example config: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[INFO]   3. Configure alerting for backup failures (cron exit codes)
[INFO]   4. Test restore drill from offsite repository
[INFO]   5. Enable RPO=0 if needed: ./04-production-optimizations.sh yes

[ OK ] System is now production-grade! 🎉

✓ ✔ PostgreSQL creation completed
[INFO] Cleaning up temporary files...
[INFO] Starting cleanup of temporary files...
[INFO] Cleaning up SSL temp files for iam-identity-universe-main-dev...
[INFO] Cleaning up old provisioning logs...
[SUCCESS] Removed 21 old log files
[INFO] Cleaning up old configuration backups...
✓ ✔ Cleanup completed
[OK] PostgreSQL setup completed for iam-identity

[INFO] Creating IAM database...
[INFO]   Host: db-iam-identity-universe-main-dev-postgresql.fastorder.com
[INFO]   Database: fastorder_iam_identity_universe_main_dev_db

✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Checking if database fastorder_iam_identity_universe_main_dev_db exists...
[OK] Database fastorder_iam_identity_universe_main_dev_db already exists
[INFO] Running IAM schema initialization...
[INFO] Executing IAM schema scripts...
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║                    IAM Database Schema Initialization                       ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] 🟢 Starting IAM schema provisioning...
[INFO] Environment: iam-identity-universe-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

[INFO] 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: core
  Core Identity Directory (tenants, realms, identities, devices, MFA)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [1/20]: core/01-tenant
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Identifier:  coordinator
  Database:    fastorder_iam_identity_universe_main_dev_db
  Host:        db-iam-identity-universe-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_iam_identity_universe_main_dev_db
ℹ️  Database fastorder_iam_identity_universe_main_dev_db already exists
✅ Connected to database: fastorder_iam_identity_universe_main_dev_db
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "pgcrypto" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "citext" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
NOTICE:  schema "utils" already exists, skipping
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
NOTICE:  schema "core" already exists, skipping
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
NOTICE:  relation "tenant" already exists, skipping
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[OK] Table core/01-tenant initialized

[INFO] 🔸 Table [2/20]: core/02-realm
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  relation "realm" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_realm_keycloak_id" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_realm_tenant" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[OK] Table core/02-realm initialized

[INFO] 🔸 Table [3/20]: core/03-identity
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  relation "identity" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_unique_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_unique_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_type" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[OK] Table core/03-identity initialized

[INFO] 🔸 Table [4/20]: core/04-device
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  relation "device" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_device_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_fingerprint" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_trusted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_last_seen" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[OK] Table core/04-device initialized

[INFO] 🔸 Table [5/20]: core/05-identity_account
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  relation "identity_account" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_account_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_lockout" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_last_login" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[OK] Table core/05-identity_account initialized

[INFO] 🔸 Table [6/20]: core/06-identity_mfa
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  relation "identity_mfa" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_mfa_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_active" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[OK] Table core/06-identity_mfa initialized

[INFO] 🔸 Table [7/20]: core/07-external_idp_link
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  relation "external_idp_link" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_external_idp_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_provider" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_email" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[OK] Table core/07-external_idp_link initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: policy
  RBAC/ABAC Authorization (clients, roles, permissions, policies)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [8/20]: policy/01-client
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
NOTICE:  schema "policy" already exists, skipping
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  relation "client" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_client_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_key" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_status" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[OK] Table policy/01-client initialized

[INFO] 🔸 Table [9/20]: policy/02-resource
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  relation "resource" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_resource_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_external" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_owner" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[OK] Table policy/02-resource initialized

[INFO] 🔸 Table [10/20]: policy/03-scope
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  relation "scope" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_scope_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_scope_name" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[OK] Table policy/03-scope initialized

[INFO] 🔸 Table [11/20]: policy/04-permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  relation "permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_permission_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_resource" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[OK] Table policy/04-permission initialized

[INFO] 🔸 Table [12/20]: policy/05-role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  relation "role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_keycloak" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[OK] Table policy/05-role initialized

[INFO] 🔸 Table [13/20]: policy/06-role_permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  relation "role_permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_permission_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_permission_perm" already exists, skipping
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[OK] Table policy/06-role_permission initialized

[INFO] 🔸 Table [14/20]: policy/07-identity_role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  relation "identity_role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_role_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_active" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[OK] Table policy/07-identity_role initialized

[INFO] 🔸 Table [15/20]: policy/08-policy_rule
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  relation "policy_rule" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_policy_rule_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_enabled" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_priority" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[OK] Table policy/08-policy_rule initialized

[INFO] 🔸 Table [16/20]: policy/09-api_key
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  relation "api_key" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_api_key_prefix" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[OK] Table policy/09-api_key initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: audit
  Audit & Risk Logging (auth events, admin actions, risk decisions)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [17/20]: audit/01-auth_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: iam-identity-universe-main-dev
  Database:    fastorder_iam_identity_universe_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
NOTICE:  schema "audit" already exists, skipping
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
NOTICE:  relation "auth_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_auth_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_result" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_ip" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_session" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_trace" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_risk" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[OK] Table audit/01-auth_event initialized

[INFO] 🔸 Table [18/20]: audit/02-admin_action
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
NOTICE:  relation "admin_action" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_admin_action_actor" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_target" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_trace" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[OK] Table audit/02-admin_action initialized

[INFO] 🔸 Table [19/20]: audit/03-risk_decision
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
NOTICE:  relation "risk_decision" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_risk_decision_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_level" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_decision" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_auth" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[OK] Table audit/03-risk_decision initialized

[INFO] 🔸 Table [20/20]: audit/04-consent_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
NOTICE:  relation "consent_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_consent_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_version" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_granted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  relation "audit.auth_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  relation "audit.auth_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  relation "audit.admin_action_2026_03" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  relation "audit.admin_action_2026_04" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  relation "audit.risk_decision_2026_03" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  relation "audit.risk_decision_2026_04" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  relation "audit.consent_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  relation "audit.consent_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[OK] Table audit/04-consent_event initialized


════════════════════════════════════════════════════════════════════════════
[OK] ✅ IAM Schema Initialization Complete!
[OK] All 20 tables initialized successfully

Schemas created:
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

Design highlights:
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

════════════════════════════════════════════════════════════════════════════
[OK] IAM schema initialization completed successfully

════════════════════════════════════════════════════════════════════════════
[OK] IAM Database Provisioning Complete

[INFO] Schemas created:
[INFO]   • core   - Identity directory (tenant, realm, identity, devices, MFA)
[INFO]   • policy - Authorization (clients, roles, permissions, policies)
[INFO]   • audit  - Logging (auth events, admin actions, risk decisions)

[INFO] Next step: 04-configure-database.sh
[INFO]   - Configure Keycloak JDBC connection
[INFO]   - Update keycloak.conf with database settings
════════════════════════════════════════════════════════════════════════════

[OK] ✅ Step 4 completed: 04-database-setup.sh

[INFO] 🔐 Step 5/12: configure jdbc...
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[CREDS] Using AWS credentials from: /var/www/.aws/credentials
[CREDS] Credential management library loaded (region: me-central-1)

╔════════════════════════════════════════════════════════════════════════════╗
║                    Keycloak JDBC Configuration                              ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] Configuring Keycloak JDBC: iam-identity-universe-main-dev
[INFO]   Storage Backend: domain-db
[INFO] Determining database connection...
[INFO] Using domain database: db-iam-identity-universe-main-dev-postgresql.fastorder.com
[INFO]   Database: fastorder_iam_identity_universe_main_dev_db
[INFO] Retrieving database credentials from Secrets Manager...
[INFO] Using credentials from vault
[OK] Database credentials retrieved
[INFO] Setting up SSL certificates for database connection...
[2026-01-02 04:41:05 UTC] USER=www-data EUID=0 PID=1431186 ACTION=fsop ARGS=mkdir -p /etc/keycloak/iam-identity-universe-main-dev/db-ssl
[2026-01-02 04:41:05 UTC] USER=www-data EUID=0 PID=1431195 ACTION=fsop ARGS=cp /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/root.crt /etc/keycloak/iam-identity-universe-main-dev/db-ssl/root.crt
[2026-01-02 04:41:05 UTC] USER=www-data EUID=0 PID=1431204 ACTION=fsop ARGS=cp /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.crt /etc/keycloak/iam-identity-universe-main-dev/db-ssl/client.crt
[2026-01-02 04:41:05 UTC] USER=www-data EUID=0 PID=1431214 ACTION=fsop ARGS=cp /home/www-data/ssl/.postgresql/iam-identity-universe-main-dev/coordinator/postgres.key /etc/keycloak/iam-identity-universe-main-dev/db-ssl/client.key
[2026-01-02 04:41:05 UTC] USER=www-data EUID=0 PID=1431225 ACTION=fsop ARGS=openssl pkcs8 -topk8 -inform PEM -in /etc/keycloak/iam-identity-universe-main-dev/db-ssl/client.key -outform DER -out /etc/keycloak/iam-identity-universe-main-dev/db-ssl/client.pk8 -nocrypt
[2026-01-02 04:41:05 UTC] USER=www-data EUID=0 PID=1431237 ACTION=fsop ARGS=chown -R keycloak:keycloak /etc/keycloak/iam-identity-universe-main-dev/db-ssl
[2026-01-02 04:41:05 UTC] USER=www-data EUID=0 PID=1431246 ACTION=fsop ARGS=chmod 700 /etc/keycloak/iam-identity-universe-main-dev/db-ssl
[2026-01-02 04:41:05 UTC] USER=www-data EUID=0 PID=1431255 ACTION=fsop ARGS=chmod 600 /etc/keycloak/iam-identity-universe-main-dev/db-ssl/client.key /etc/keycloak/iam-identity-universe-main-dev/db-ssl/client.pk8
[2026-01-02 04:41:05 UTC] USER=www-data EUID=0 PID=1431266 ACTION=fsop ARGS=chmod 644 /etc/keycloak/iam-identity-universe-main-dev/db-ssl/root.crt /etc/keycloak/iam-identity-universe-main-dev/db-ssl/client.crt
[OK] SSL certificates copied to /etc/keycloak/iam-identity-universe-main-dev/db-ssl
[INFO] Updating Keycloak configuration with JDBC settings...
[2026-01-02 04:41:05 UTC] USER=www-data EUID=0 PID=1431275 ACTION=fsop ARGS=test -f /etc/keycloak/iam-identity-universe-main-dev/keycloak.conf
[2026-01-02 04:41:05 UTC] USER=www-data EUID=0 PID=1431296 ACTION=fsop ARGS=sed -i /^# db-url=/d /etc/keycloak/iam-identity-universe-main-dev/keycloak.conf
[2026-01-02 04:41:05 UTC] USER=www-data EUID=0 PID=1431305 ACTION=fsop ARGS=sed -i /^# db-username=/d /etc/keycloak/iam-identity-universe-main-dev/keycloak.conf
[2026-01-02 04:41:05 UTC] USER=www-data EUID=0 PID=1431314 ACTION=fsop ARGS=sed -i /^# db-password=/d /etc/keycloak/iam-identity-universe-main-dev/keycloak.conf
[2026-01-02 04:41:05 UTC] USER=www-data EUID=0 PID=1431335 ACTION=fsop ARGS=chown keycloak:keycloak /etc/keycloak/iam-identity-universe-main-dev/keycloak.conf
[2026-01-02 04:41:05 UTC] USER=www-data EUID=0 PID=1431344 ACTION=fsop ARGS=chmod 640 /etc/keycloak/iam-identity-universe-main-dev/keycloak.conf
[OK] Keycloak configuration updated
[INFO] Testing database connection...
[OK] Database connection successful
[INFO] Restarting Keycloak to apply new JDBC configuration...
[INFO] Keycloak service not running, will be started by later steps

════════════════════════════════════════════════════════════════════════════
[OK] Keycloak JDBC Configuration Complete

[INFO] Database Connection:
[INFO]   Host: db-iam-identity-universe-main-dev-postgresql.fastorder.com:5432
[INFO]   Database: fastorder_iam_identity_universe_main_dev_db
[INFO]   User: postgres
[INFO]   Schema: public
[INFO]   JDBC URL: jdbc:postgresql://db-iam-identity-universe-main-dev-postgresql.fastorder.com:5432/fastorder_iam_identity_universe_main_dev_db?sslmode=verify-full&sslrootcert=/etc/keycloak/iam-identity-universe-main-dev/db-ssl/root.crt&sslcert=/etc/keycloak/iam-identity-universe-main-dev/db-ssl/client.crt&sslkey=/etc/keycloak/iam-identity-universe-main-dev/db-ssl/client.pk8

[INFO] Configuration file: /etc/keycloak/iam-identity-universe-main-dev/keycloak.conf
════════════════════════════════════════════════════════════════════════════

[OK] ✅ Step 5 completed: 05-configure-jdbc.sh

[INFO] 🔐 Step 6/12: create admin...
[CREDS] Using AWS credentials from: /var/www/.aws/credentials
[CREDS] Credential management library loaded (region: me-central-1)
[CREDS] Using AWS credentials from: /var/www/.aws/credentials
[CREDS] Credential management library loaded (region: me-central-1)
[INFO] Creating Keycloak admin user for: iam-identity-universe-main-dev
[INFO] AWS credentials loaded from: /var/www/.aws/credentials
[INFO] Setting up admin credentials...
[INFO] Secret path: fastorder/iam/identity/universe/main/dev/keycloak/admin
[INFO] No existing credentials found, generating new ones...
[INFO] Generated new admin password
[INFO] Storing admin credentials in Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/iam/identity/universe/main/dev/keycloak/admin-2NrEv9",
    "Name": "fastorder/iam/identity/universe/main/dev/keycloak/admin",
    "VersionId": "d40adf7a-7e07-43a2-87ce-452de4bd488b"
}
[OK] Created admin credentials in Secrets Manager: fastorder/iam/identity/universe/main/dev/keycloak/admin
[INFO] Setting up admin credentials for Keycloak...
[2026-01-02 04:41:11 UTC] USER=www-data EUID=0 PID=1431438 ACTION=fsop ARGS=mv /tmp/keycloak-iam-identity-universe-main-dev.env /etc/keycloak/iam-identity-universe-main-dev/keycloak.env
[2026-01-02 04:41:11 UTC] USER=www-data EUID=0 PID=1431459 ACTION=fsop ARGS=chown keycloak:keycloak /etc/keycloak/iam-identity-universe-main-dev/keycloak.env
[2026-01-02 04:41:11 UTC] USER=www-data EUID=0 PID=1431480 ACTION=fsop ARGS=chmod 600 /etc/keycloak/iam-identity-universe-main-dev/keycloak.env
[OK] Environment file created: /etc/keycloak/iam-identity-universe-main-dev/keycloak.env
[INFO] Updating systemd service...
[2026-01-02 04:41:12 UTC] USER=www-data EUID=0 PID=1431502 ACTION=fsop ARGS=sed -i /\[Service\]/a EnvironmentFile=/etc/keycloak/iam-identity-universe-main-dev/keycloak.env /etc/systemd/system/keycloak-iam-identity-universe-main-dev.service
[OK] Added environment file to systemd service
[2026-01-02 04:41:12 UTC] USER=www-data EUID=0 PID=1431523 ACTION=passthru ARGS=systemctl daemon-reload
[INFO]   Keycloak URL: https://10.100.60.2:8443
[INFO]   Health URL: http://10.100.60.2:8080
[INFO] Restarting Keycloak to apply admin credentials...
[2026-01-02 04:41:12 UTC] USER=www-data EUID=0 PID=1431588 ACTION=passthru ARGS=systemctl start keycloak-iam-identity-universe-main-dev
[INFO] Waiting for Keycloak to become ready...
...........[OK] Keycloak is ready

[INFO] Verifying admin user credentials...
[2026-01-02 04:41:40 UTC] USER=www-data EUID=0 PID=1431875 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "config" "credentials" "--server" "http://10.100.60.2:8080" "--realm" "master" "--user" "admin" "--password" "eGBWBe03mgWRqxNxnJ5Mgs1Y" "--config" "/tmp/kcadm.config"
Logging into http://10.100.60.2:8080 as user admin of realm master
[OK] Admin user verified successfully
[2026-01-02 04:41:43 UTC] USER=www-data EUID=0 PID=1431939 ACTION=fsop ARGS=rm -f /tmp/kcadm.config

[OK] Admin user setup completed
[INFO]   Username: admin
[INFO]   Email: admin@iam-identity.fastorder.com
[INFO]   Credentials stored in: fastorder/iam/identity/universe/main/dev/keycloak/admin

[WARN] SECURITY: Admin password is stored in AWS Secrets Manager
[WARN] To retrieve: aws secretsmanager get-secret-value --secret-id fastorder/iam/identity/universe/main/dev/keycloak/admin

[OK] ✅ Step 6 completed: 06-create-admin.sh

[INFO] 🔐 Step 7/12: session cache setup...
[INFO] Configuring session cache for Keycloak: iam-identity-universe-main-dev
[INFO]   Session Backend: redis
[INFO]   HA Cluster: false
[INFO] Configuring Redis session store...
[2026-01-02 04:41:44 UTC] USER=www-data EUID=0 PID=1431964 ACTION=fsop ARGS=mkdir -p /etc/keycloak/iam-identity-universe-main-dev/cache
[2026-01-02 04:41:44 UTC] USER=www-data EUID=0 PID=1431974 ACTION=fsop ARGS=mv /tmp/cache-ispn-redis-iam-identity-universe-main-dev.xml /etc/keycloak/iam-identity-universe-main-dev/cache/cache-ispn.xml
[2026-01-02 04:41:44 UTC] USER=www-data EUID=0 PID=1431983 ACTION=fsop ARGS=chown keycloak:keycloak /etc/keycloak/iam-identity-universe-main-dev/cache/cache-ispn.xml
[2026-01-02 04:41:44 UTC] USER=www-data EUID=0 PID=1431992 ACTION=fsop ARGS=chmod 640 /etc/keycloak/iam-identity-universe-main-dev/cache/cache-ispn.xml
[OK] Redis session cache configured
[INFO]   Redis Host: cache-iam-identity-universe-main-dev-redis.fastorder.com:6379
[INFO]   Session TTL: 3600s

[OK] Session cache setup completed
[INFO]   Backend: redis
[INFO]   HA Mode: false
[INFO]   Redis TTL: 3600s

[OK] ✅ Step 7 completed: 07-session-cache-setup.sh

[INFO] 🔐 Step 8/12: realm setup...
[CREDS] Using AWS credentials from: /var/www/.aws/credentials
[CREDS] Credential management library loaded (region: me-central-1)
[INFO] Setting up realm for Keycloak: iam-identity-universe-main-dev
[INFO]   Realm Name: identity
[INFO]   Keycloak IP: 10.100.60.2
[INFO]   Keycloak FQDN: keycloak-iam-identity-universe-main-dev-controller.fastorder.com
[INFO] Waiting for Keycloak to be ready...
[INFO]   Admin URL: http://10.100.60.2:8080
[INFO]   Health URL: http://10.100.60.2:8080
[OK] Keycloak is ready

[INFO] Authenticating with Keycloak admin...
[INFO] Credentials loaded using shared function
[INFO]   Using admin user: admin
[2026-01-02 04:41:46 UTC] USER=www-data EUID=0 PID=1432066 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "config" "credentials" "--server" "http://10.100.60.2:8080" "--realm" "master" "--user" "admin" "--password" "eGBWBe03mgWRqxNxnJ5Mgs1Y" "--config" "/tmp/kcadm.config"
Logging into http://10.100.60.2:8080 as user admin of realm master
[OK] Authenticated with Keycloak admin
[INFO] Creating realm: identity...
[2026-01-02 04:41:50 UTC] USER=www-data EUID=0 PID=1432199 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "create" "realms" "--config" "/tmp/kcadm.config" "-s" "realm=identity" "-s" "displayName=FastOrder Iam-identity Service" "-s" "enabled=true" "-s" "registrationAllowed=false" "-s" "loginWithEmailAllowed=true" "-s" "duplicateEmailsAllowed=false" "-s" "resetPasswordAllowed=true" "-s" "editUsernameAllowed=false" "-s" "bruteForceProtected=true" "-s" "permanentLockout=false" "-s" "maxFailureWaitSeconds=900" "-s" "minimumQuickLoginWaitSeconds=60" "-s" "waitIncrementSeconds=60" "-s" "quickLoginCheckMilliSeconds=1000" "-s" "maxDeltaTimeSeconds=43200" "-s" "failureFactor=5" "-s" "sslRequired=external" "-s" "accessTokenLifespan=900" "-s" "ssoSessionIdleTimeout=1800" "-s" "ssoSessionMaxLifespan=36000" "-s" "offlineSessionIdleTimeout=2592000" "-s" "accessCodeLifespan=60" "-s" "accessCodeLifespanUserAction=300" "-s" "accessCodeLifespanLogin=1800"
Created new realm with id 'identity'
[OK] Realm created: identity
[INFO] Configuring authentication flows...
[INFO]   Enabling TOTP authentication...
[2026-01-02 04:41:54 UTC] USER=www-data EUID=0 PID=1432263 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "update" "realms/identity" "--config" "/tmp/kcadm.config" "-s" "otpPolicyType=totp" "-s" "otpPolicyAlgorithm=HmacSHA1" "-s" "otpPolicyDigits=6" "-s" "otpPolicyPeriod=30"
[OK] Authentication flows configured
[INFO] Configuring password policy...
[2026-01-02 04:41:56 UTC] USER=www-data EUID=0 PID=1432328 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "update" "realms/identity" "--config" "/tmp/kcadm.config" "-s" "passwordPolicy=length(12) and digits(1) and upperCase(1) and lowerCase(1) and specialChars(1) and notUsername and passwordHistory(5)"
[OK] Password policy configured
[INFO] Creating default roles...
[2026-01-02 04:42:00 UTC] USER=www-data EUID=0 PID=1432457 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "create" "roles" "-r" "identity" "--config" "/tmp/kcadm.config" "-s" "name=user" "-s" "description=Default user role"
Created new role with id 'user'
[INFO]   Created role: user
[2026-01-02 04:42:04 UTC] USER=www-data EUID=0 PID=1432608 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "create" "roles" "-r" "identity" "--config" "/tmp/kcadm.config" "-s" "name=admin" "-s" "description=Default admin role"
Created new role with id 'admin'
[INFO]   Created role: admin
[2026-01-02 04:42:08 UTC] USER=www-data EUID=0 PID=1432748 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "create" "roles" "-r" "identity" "--config" "/tmp/kcadm.config" "-s" "name=service" "-s" "description=Default service role"
Created new role with id 'service'
[INFO]   Created role: service
[2026-01-02 04:42:12 UTC] USER=www-data EUID=0 PID=1432873 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "create" "roles" "-r" "identity" "--config" "/tmp/kcadm.config" "-s" "name=api-client" "-s" "description=Default api-client role"
Created new role with id 'api-client'
[INFO]   Created role: api-client
[2026-01-02 04:42:14 UTC] USER=www-data EUID=0 PID=1432935 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "update" "realms/identity" "--config" "/tmp/kcadm.config" "-s" "defaultRole={\"name\":\"user\"}"
[OK] Default roles created
[INFO] Exporting realm configuration...
[2026-01-02 04:42:16 UTC] USER=www-data EUID=0 PID=1432996 ACTION=fsop ARGS=mkdir -p /etc/keycloak/iam-identity-universe-main-dev/realms
[2026-01-02 04:42:18 UTC] USER=www-data EUID=0 PID=1433079 ACTION=fsop ARGS=mv /tmp/identity-realm.json /etc/keycloak/iam-identity-universe-main-dev/realms/
[2026-01-02 04:42:18 UTC] USER=www-data EUID=0 PID=1433100 ACTION=fsop ARGS=chown keycloak:keycloak /etc/keycloak/iam-identity-universe-main-dev/realms/identity-realm.json
[OK] Realm configuration exported to /etc/keycloak/iam-identity-universe-main-dev/realms/identity-realm.json
[2026-01-02 04:42:18 UTC] USER=www-data EUID=0 PID=1433121 ACTION=fsop ARGS=rm -f /tmp/kcadm.config

[OK] Realm setup completed
[INFO]   Realm: identity
[INFO]   Display Name: FastOrder Iam-identity Service
[INFO]   Access Token Lifespan: 15m
[INFO]   Refresh Token Lifespan: 30d
[INFO]   Brute Force Protection: Enabled
[INFO]   Password Policy: true

[INFO] OIDC Discovery URL:
[INFO]   https://10.100.60.2:8443/realms/identity/.well-known/openid-configuration

[OK] ✅ Step 8 completed: 08-realm-setup.sh

[INFO] 🔐 Step 9/12: client setup...
[CREDS] Using AWS credentials from: /var/www/.aws/credentials
[CREDS] Credential management library loaded (region: me-central-1)
[INFO] Setting up Keycloak clients for: iam-identity-universe-main-dev
[INFO]   Realm: identity
[INFO] Authenticating with Keycloak...
[INFO]   Keycloak IP: 10.100.60.2
[INFO]   Admin URL: http://10.100.60.2:8080
[INFO]   Using admin user: admin
[2026-01-02 04:42:20 UTC] USER=www-data EUID=0 PID=1433181 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "config" "credentials" "--server" "http://10.100.60.2:8080" "--realm" "master" "--user" "admin" "--password" "eGBWBe03mgWRqxNxnJ5Mgs1Y" "--config" "/tmp/kcadm.config"
Logging into http://10.100.60.2:8080 as user admin of realm master
[OK] Authenticated with Keycloak
[INFO] Creating backend API client...
[2026-01-02 04:42:24 UTC] USER=www-data EUID=0 PID=1433312 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "create" "clients" "-r" "identity" "--config" "/tmp/kcadm.config" "-s" "clientId=identity-api" "-s" "name=Backend API Service" "-s" "description=Service account for backend API" "-s" "enabled=true" "-s" "protocol=openid-connect" "-s" "publicClient=false" "-s" "serviceAccountsEnabled=true" "-s" "directAccessGrantsEnabled=false" "-s" "standardFlowEnabled=false" "-s" "secret=LCHBoTIH5VcRuHJvnHUINo6OwaoiyV7r"
Created new client with id '613dca55-e374-4d3a-9a50-8e85f113fa6e'
[OK]   Created client: identity-api
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/iam/identity/universe/main/dev/keycloak/client-api-TXin2H",
    "Name": "fastorder/iam/identity/universe/main/dev/keycloak/client-api",
    "VersionId": "6a2789ab-799d-4634-a56f-8f50b642ef34"
}
[OK] Backend API client configured
[INFO] Creating frontend web client...
[2026-01-02 04:42:32 UTC] USER=www-data EUID=0 PID=1433478 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "create" "clients" "-r" "identity" "--config" "/tmp/kcadm.config" "-s" "clientId=identity-web" "-s" "name=Frontend Web Application" "-s" "description=Public client for web application (PKCE)" "-s" "enabled=true" "-s" "protocol=openid-connect" "-s" "publicClient=true" "-s" "standardFlowEnabled=true" "-s" "directAccessGrantsEnabled=false" "-s" "redirectUris=[\"https://app.identity.universe.fastorder.com/*\", \"http://localhost:3000/*\", \"http://localhost:8080/*\"]" "-s" "webOrigins=[\"https://app.identity.universe.fastorder.com\", \"http://localhost:3000\", \"http://localhost:8080\"]" "-s" "attributes={\"pkce.code.challenge.method\":\"S256\"}"
Created new client with id 'ec12b1b9-8a6e-49ff-822a-ecc563556f8f'
[OK]   Created client: identity-web
[INFO] Creating mobile client...
[2026-01-02 04:42:36 UTC] USER=www-data EUID=0 PID=1433604 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "create" "clients" "-r" "identity" "--config" "/tmp/kcadm.config" "-s" "clientId=identity-mobile" "-s" "name=Mobile Application" "-s" "description=Public client for mobile apps (PKCE)" "-s" "enabled=true" "-s" "protocol=openid-connect" "-s" "publicClient=true" "-s" "standardFlowEnabled=true" "-s" "directAccessGrantsEnabled=false" "-s" "redirectUris=[\"com.fastorder.identity:/oauth2redirect\", \"fastorder://callback\"]" "-s" "attributes={\"pkce.code.challenge.method\":\"S256\"}"
Created new client with id 'efff45f4-1c3e-4103-a93a-493f92130cc1'
[OK]   Created client: identity-mobile
[INFO] Creating admin console client...
[2026-01-02 04:42:40 UTC] USER=www-data EUID=0 PID=1433831 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "create" "clients" "-r" "identity" "--config" "/tmp/kcadm.config" "-s" "clientId=identity-admin" "-s" "name=Admin Console" "-s" "description=Confidential client for admin console" "-s" "enabled=true" "-s" "protocol=openid-connect" "-s" "publicClient=false" "-s" "standardFlowEnabled=true" "-s" "directAccessGrantsEnabled=false" "-s" "secret=eNK78YXVksn51PtEj21AhGTkRmbf0cpv" "-s" "redirectUris=[\"https://app.identity.universe.fastorder.com/admin/*\"]" "-s" "webOrigins=[\"https://app.identity.universe.fastorder.com\"]"
Created new client with id 'd3d22303-1fdf-4a03-a4e3-003f8517134a'
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/iam/identity/universe/main/dev/keycloak/client-admin-mWKgvI",
    "Name": "fastorder/iam/identity/universe/main/dev/keycloak/client-admin",
    "VersionId": "90a0d25e-9e72-4c75-88e3-ef241502b752"
}
[OK]   Created client: identity-admin
[INFO] Creating custom client scopes...
[2026-01-02 04:42:47 UTC] USER=www-data EUID=0 PID=1433969 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "create" "client-scopes" "-r" "identity" "--config" "/tmp/kcadm.config" "-s" "name=api" "-s" "description=API access scope" "-s" "protocol=openid-connect" "-s" "attributes={\"include.in.token.scope\":\"true\",\"display.on.consent.screen\":\"true\"}"
Created new client-scope with id 'cba91f08-99e3-48f0-b6d1-b05b8bf83e9e'
[OK]   Created scope: api
[2026-01-02 04:42:51 UTC] USER=www-data EUID=0 PID=1434096 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "create" "client-scopes" "-r" "identity" "--config" "/tmp/kcadm.config" "-s" "name=admin" "-s" "description=Admin access scope" "-s" "protocol=openid-connect" "-s" "attributes={\"include.in.token.scope\":\"true\",\"display.on.consent.screen\":\"true\"}"
Created new client-scope with id 'd2631d0b-4304-44f7-9f58-cba785c1e28e'
[OK]   Created scope: admin
[OK] Client scopes configured
[2026-01-02 04:42:53 UTC] USER=www-data EUID=0 PID=1434156 ACTION=fsop ARGS=rm -f /tmp/kcadm.config

[OK] Client setup completed
[INFO]   Clients created:
[INFO]     - identity-api (confidential, service account)
[INFO]     - identity-web (public, PKCE)
[INFO]     - identity-mobile (public, PKCE)
[INFO]     - identity-admin (confidential)

[INFO]   Client credentials stored in AWS Secrets Manager

[OK] ✅ Step 9 completed: 09-client-setup.sh

[INFO] 🔐 Step 10/12: rate limiting setup...
[CREDS] Using AWS credentials from: /var/www/.aws/credentials
[CREDS] Credential management library loaded (region: me-central-1)
[INFO] Setting up rate limiting and anti-abuse for: iam-identity-universe-main-dev
[INFO]   Rate Limiting: true
[INFO]   Login Rate Limit: 10/min
[INFO]   Risk Engine: builtin
[INFO] Configuring brute force protection...
[INFO]   Keycloak IP: 10.100.60.2
[INFO]   Using admin user: admin
[2026-01-02 04:42:55 UTC] USER=www-data EUID=0 PID=1434216 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "config" "credentials" "--server" "http://10.100.60.2:8080" "--realm" "master" "--user" "admin" "--password" "eGBWBe03mgWRqxNxnJ5Mgs1Y" "--config" "/tmp/kcadm.config"
Logging into http://10.100.60.2:8080 as user admin of realm master
[2026-01-02 04:42:57 UTC] USER=www-data EUID=0 PID=1434276 ACTION=passthru ARGS=bash -c cd /opt/keycloak && sudo -n -u keycloak /opt/keycloak/bin/kcadm.sh "update" "realms/identity" "--config" "/tmp/kcadm.config" "-s" "bruteForceProtected=true" "-s" "permanentLockout=false" "-s" "failureFactor=5" "-s" "maxFailureWaitSeconds=900" "-s" "waitIncrementSeconds=180" "-s" "minimumQuickLoginWaitSeconds=60" "-s" "quickLoginCheckMilliSeconds=1000" "-s" "maxDeltaTimeSeconds=43200"
[OK] Brute force protection configured
rm: cannot remove '/tmp/kcadm.config': Operation not permitted
[INFO] Configuring Nginx rate limiting...
[WARN] Nginx conf.d directory not found, skipping Nginx rate limiting
[WARN] fail2ban not installed, skipping IP-based blocking
[INFO] Creating risk scoring configuration...
[2026-01-02 04:42:59 UTC] USER=www-data EUID=0 PID=1434341 ACTION=fsop ARGS=mkdir -p /etc/keycloak/iam-identity-universe-main-dev/risk
[2026-01-02 04:42:59 UTC] USER=www-data EUID=0 PID=1434364 ACTION=fsop ARGS=mv /tmp/risk-scoring-iam-identity-universe-main-dev.json /etc/keycloak/iam-identity-universe-main-dev/risk/config.json
[2026-01-02 04:42:59 UTC] USER=www-data EUID=0 PID=1434385 ACTION=fsop ARGS=chown -R keycloak:keycloak /etc/keycloak/iam-identity-universe-main-dev/risk
[2026-01-02 04:42:59 UTC] USER=www-data EUID=0 PID=1434406 ACTION=fsop ARGS=chmod 640 /etc/keycloak/iam-identity-universe-main-dev/risk/config.json
[OK] Risk scoring configuration created
[INFO] Creating IP blocklist management script...
[2026-01-02 04:43:00 UTC] USER=www-data EUID=0 PID=1434427 ACTION=fsop ARGS=mkdir -p /etc/keycloak/iam-identity-universe-main-dev/scripts
[2026-01-02 04:43:00 UTC] USER=www-data EUID=0 PID=1434450 ACTION=fsop ARGS=mv /tmp/manage-blocklist-iam-identity-universe-main-dev.sh /etc/keycloak/iam-identity-universe-main-dev/scripts/manage-blocklist.sh
[2026-01-02 04:43:00 UTC] USER=www-data EUID=0 PID=1434471 ACTION=fsop ARGS=chmod +x /etc/keycloak/iam-identity-universe-main-dev/scripts/manage-blocklist.sh
[2026-01-02 04:43:00 UTC] USER=www-data EUID=0 PID=1434492 ACTION=fsop ARGS=mkdir -p /etc/keycloak/iam-identity-universe-main-dev/blocklist
[2026-01-02 04:43:00 UTC] USER=www-data EUID=0 PID=1434513 ACTION=fsop ARGS=touch /etc/keycloak/iam-identity-universe-main-dev/blocklist/ip-blocklist.txt
[2026-01-02 04:43:00 UTC] USER=www-data EUID=0 PID=1434534 ACTION=fsop ARGS=chown -R keycloak:keycloak /etc/keycloak/iam-identity-universe-main-dev/blocklist
[OK] IP blocklist management script created

[OK] Rate limiting and anti-abuse setup completed
[INFO]   Brute Force Protection: Enabled
[INFO]   Max Login Failures: 5
[INFO]   Lockout Duration: 15 minutes
[INFO]   Login Rate Limit: 10/min
[INFO]   Token Rate Limit: 60/min

[INFO]   Risk Scoring:
[INFO]     Engine: builtin
[INFO]     Low Threshold: 30
[INFO]     Medium Threshold: 70
[INFO]     High Threshold: 90

[INFO]   Config files:
[INFO]     Risk config: /etc/keycloak/iam-identity-universe-main-dev/risk/config.json
[INFO]     Blocklist: /etc/keycloak/iam-identity-universe-main-dev/blocklist/ip-blocklist.txt

[OK] ✅ Step 10 completed: 10-rate-limiting-setup.sh

[INFO] 🔐 Step 11/12: identity federation setup...
[CREDS] Using AWS credentials from: /var/www/.aws/credentials
[CREDS] Credential management library loaded (region: me-central-1)
[INFO] Setting up identity federation for: iam-identity-universe-main-dev
[INFO]   Global Directory Sync: false
[INFO]   LDAP Enabled: false
[INFO] No identity federation configured
[OK] Identity federation setup skipped
[OK] ✅ Step 11 completed: 11-identity-federation-setup.sh

[INFO] 🔐 Step 12/12: monitoring setup...
[INFO] Setting up monitoring for Keycloak: iam-identity-universe-main-dev
[INFO]   Prometheus Metrics: true
[INFO]   Log Level: INFO
[INFO] Configuring Keycloak metrics...
[2026-01-02 04:43:00 UTC] USER=www-data EUID=0 PID=1434608 ACTION=fsop ARGS=chown keycloak:keycloak /etc/keycloak/iam-identity-universe-main-dev/keycloak.conf
[OK] Metrics configuration added to keycloak.conf
[INFO] Setting up log directory and rotation...
[2026-01-02 04:43:00 UTC] USER=www-data EUID=0 PID=1434629 ACTION=fsop ARGS=mkdir -p /var/log/keycloak/iam-identity-universe-main-dev
[2026-01-02 04:43:01 UTC] USER=www-data EUID=0 PID=1434650 ACTION=fsop ARGS=chown keycloak:keycloak /var/log/keycloak/iam-identity-universe-main-dev
[2026-01-02 04:43:01 UTC] USER=www-data EUID=0 PID=1434671 ACTION=fsop ARGS=chmod 750 /var/log/keycloak/iam-identity-universe-main-dev
[2026-01-02 04:43:01 UTC] USER=www-data EUID=0 PID=1434693 ACTION=fsop ARGS=mv /tmp/logrotate-keycloak-iam-identity-universe-main-dev /etc/logrotate.d/keycloak-iam-identity-universe-main-dev
[2026-01-02 04:43:01 UTC] USER=www-data EUID=0 PID=1434714 ACTION=fsop ARGS=chmod 644 /etc/logrotate.d/keycloak-iam-identity-universe-main-dev
[OK] Log rotation configured
[INFO] Creating Prometheus scrape configuration...
[WARN] Prometheus targets directory not found, skipping scrape config
[INFO] Creating Grafana dashboard...
[WARN] Grafana dashboards directory not found, skipping dashboard creation
[INFO] Creating health check script...
[2026-01-02 04:43:01 UTC] USER=www-data EUID=0 PID=1434735 ACTION=fsop ARGS=mkdir -p /etc/keycloak/iam-identity-universe-main-dev/scripts
[2026-01-02 04:43:01 UTC] USER=www-data EUID=0 PID=1434757 ACTION=fsop ARGS=mv /tmp/health-check-iam-identity-universe-main-dev.sh /etc/keycloak/iam-identity-universe-main-dev/scripts/health-check.sh
[2026-01-02 04:43:01 UTC] USER=www-data EUID=0 PID=1434778 ACTION=fsop ARGS=chmod +x /etc/keycloak/iam-identity-universe-main-dev/scripts/health-check.sh
[OK] Health check script created
[INFO] Creating systemd health check timer...
[2026-01-02 04:43:01 UTC] USER=www-data EUID=0 PID=1434817 ACTION=fsop ARGS=mv /tmp/keycloak-iam-identity-universe-main-dev-health.service /etc/systemd/system/keycloak-iam-identity-universe-main-dev-health.service
[2026-01-02 04:43:01 UTC] USER=www-data EUID=0 PID=1434839 ACTION=fsop ARGS=mv /tmp/keycloak-iam-identity-universe-main-dev-health.timer /etc/systemd/system/keycloak-iam-identity-universe-main-dev-health.timer
[2026-01-02 04:43:02 UTC] USER=www-data EUID=0 PID=1434860 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:43:02 UTC] USER=www-data EUID=0 PID=1434917 ACTION=passthru ARGS=systemctl enable keycloak-iam-identity-universe-main-dev-health.timer
Created symlink /etc/systemd/system/timers.target.wants/keycloak-iam-identity-universe-main-dev-health.timer -> /etc/systemd/system/keycloak-iam-identity-universe-main-dev-health.timer.
[2026-01-02 04:43:03 UTC] USER=www-data EUID=0 PID=1434974 ACTION=passthru ARGS=systemctl start keycloak-iam-identity-universe-main-dev-health.timer
[OK] Health check timer created and started
[INFO] Registering Keycloak node to skeleton observability system...
[INFO] IAM Environment: iam-identity-universe-main-dev
[INFO]   Registering to IAM environment: iam-identity-universe-main-dev
[INFO]   Application: Keycloak
[INFO]   Identifier: iam-identity-universe-main-dev-keycloak-controller
[INFO]   Identifier Parent: controller
[INFO]   IP: 10.100.60.2
[INFO]   Port: 8443
[INFO]   FQDN: keycloak-iam-identity-universe-main-dev.fastorder.com
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Keycloak
[INFO]   Identifier:        iam-identity-universe-main-dev-keycloak-controller
[INFO]   Identifier Parent: controller
[INFO]   IP:                10.100.60.2
[INFO]   Port:              8443
[INFO]   FQDN:              keycloak-iam-identity-universe-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: d441f4c0-4040-4d7b-a86b-a41ff03cb256
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK] Keycloak node registered successfully to IAM tab
[INFO] Checking IAM database for monitoring integration...
[INFO]   IAM database found: db-iam-identity-universe-main-dev-postgresql.fastorder.com
[INFO]   Setting up postgres_exporter for IAM database...
[INFO] Checking observability cell readiness: obs-iam-identity-universe-main-dev
[OK] Observability cell endpoints registered for iam-identity-universe-main-dev
[INFO] Setting up postgres_exporter for iam-identity-universe-main-dev
[2026-01-02 04:43:03 UTC] USER=www-data EUID=0 PID=1435062 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-iam-identity-universe-main-dev.yaml /etc/prometheus/postgres_exporter_queries-iam-identity-universe-main-dev.yaml
[2026-01-02 04:43:03 UTC] USER=www-data EUID=0 PID=1435071 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-iam-identity-universe-main-dev.yaml
[2026-01-02 04:43:04 UTC] USER=www-data EUID=0 PID=1435080 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-iam-identity-universe-main-dev.yaml
[OK] Custom queries file created at /etc/prometheus/postgres_exporter_queries-iam-identity-universe-main-dev.yaml
[OK] postgres_exporter already running with custom queries for iam-identity-universe-main-dev
[OK]   PostgreSQL exporter configured for IAM database
[INFO]   Registering PostgreSQL to PARENT environment: 
[WARN]   Failed to register IAM PostgreSQL node

[OK] Monitoring setup completed
[INFO]   Log Directory: /var/log/keycloak/iam-identity-universe-main-dev
[INFO]   Log Rotation: Configured
[INFO]   Prometheus Metrics: true
[INFO]   Health Check Script: /etc/keycloak/iam-identity-universe-main-dev/scripts/health-check.sh
[INFO]   Skeleton Registration: Completed

[INFO]   Endpoints:
[INFO]     Health Ready: https://keycloak-iam-identity-universe-main-dev.fastorder.com:8443/health/ready
[INFO]     Health Live: https://keycloak-iam-identity-universe-main-dev.fastorder.com:8443/health/live
[INFO]     Metrics: https://keycloak-iam-identity-universe-main-dev.fastorder.com:8443/metrics

[INFO]   Dashboard:
[INFO]     https://skeleton.dev.fastorder.com/dashboard/monitoring/environment/<env-uuid>#iam

[INFO] Running initial metrics collection...
[WARN] Health check script not executable, skipping initial collection
[INFO]   Live Metrics: Timer runs every 1 minute and updates skeleton database

[OK] ✅ Step 12 completed: 12-monitoring-setup.sh


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Keycloak IAM setup completed successfully!
[OK] Executed all 12 steps
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Environment: iam-identity-universe-main-dev
[INFO] Service: iam-identity
[INFO] Zone: universe
[INFO] Branch: main
[INFO] Env: dev
[INFO] Registering Keycloak nodes via API...
[INFO] Detected IAM 5-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Keycloak
[INFO]   Identifier:        iam-identity-universe-main-dev-keycloak-main
[INFO]   Identifier Parent: main
[INFO]   IP:                10.100.60.2
[INFO]   Port:              8443
[INFO]   FQDN:              keycloak-iam-identity-universe-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       iam-identity-universe-main-dev (service=iam-identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 7958398c-38be-4044-97c3-f0cdbe1efb30
[SUCCESS] Environment UUID: 707f86c8-d23b-4ed1-8b04-53453f57623f
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK] ✔ Keycloak node registration completed
[INFO] Setting up Keycloak observability integration...
[INFO] Checking observability cell readiness: obs-iam-identity-universe-main-dev
[OK] Observability cell endpoints registered for iam-identity-universe-main-dev
[INFO] Observability cell verified for iam-identity-universe-main-dev
[INFO] Monitoring will be configured in step 10-monitoring-setup.sh

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
               Keycloak Access Information           
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Admin Console: https://keycloak-iam-identity-universe-main-dev.fastorder.com:8443/admin
[INFO] Account Console: https://keycloak-iam-identity-universe-main-dev.fastorder.com:8443/realms/master/account
[INFO] OIDC Discovery: https://keycloak-iam-identity-universe-main-dev.fastorder.com:8443/realms/master/.well-known/openid-configuration

[WARN] Default admin credentials are stored in AWS Secrets Manager:
[INFO]   Secret: fastorder/iam/identity/universe/main/dev/keycloak/admin


[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] Verifying IAM observability services are running...
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[OK]   ✓ tempo-obs-iam-identity-universe-main-dev.service is running
[OK]   ✓ otelcol-obs-iam-identity-universe-main-dev.service is running
[OK]   ✓ clickhouse-server-obs-iam-identity-universe-main-dev.service is running
[OK] ✓ All IAM observability services verified running

[INFO] Cleaning up temporary files...
[INFO] Starting cleanup of temporary files...
[INFO] Cleaning up SSL temp files for iam-identity-universe-main-dev...
[INFO] Cleaning up old provisioning logs...
[SUCCESS] Removed 1 old log files
[INFO] Cleaning up old configuration backups...
[OK] ✔ Cleanup completed

✓ ✅ IAM infrastructure (keycloak) setup completed successfully

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🚀 OBSERVABILITY CELL PROVISIONING STARTED
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Script: 02-observability-cell/run.sh
[INFO] Timestamp: 2026-01-02 04:43:23 UTC
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Ensuring correct permissions for observability deployment...
[2026-01-02 04:43:23 UTC] USER=www-data EUID=0 PID=1435216 ACTION=fsop ARGS=chmod 775 /var/log/fastorder
[2026-01-02 04:43:23 UTC] USER=www-data EUID=0 PID=1435225 ACTION=fsop ARGS=chown www-data:www-data /var/log/fastorder
[2026-01-02 04:43:23 UTC] USER=www-data EUID=0 PID=1435236 ACTION=fsop ARGS=touch /var/log/fastorder/provisioning-elevated.log
[2026-01-02 04:43:23 UTC] USER=www-data EUID=0 PID=1435245 ACTION=fsop ARGS=chmod 666 /var/log/fastorder/provisioning-elevated.log
[2026-01-02 04:43:23 UTC] USER=www-data EUID=0 PID=1435254 ACTION=fsop ARGS=chown www-data:www-data /var/log/fastorder/provisioning-elevated.log
[OK]   Log directory: /var/log/fastorder (775)
[OK]   Log file: provisioning-elevated.log (666)
[2026-01-02 04:43:23 UTC] USER=www-data EUID=0 PID=1435263 ACTION=fsop ARGS=chmod 775 /opt/fastorder/bash/scripts/env_app_setup/state
[OK]   State directory: 775
[2026-01-02 04:43:23 UTC] USER=www-data EUID=0 PID=1435272 ACTION=fsop ARGS=mkdir -p /etc/fastorder/observability/certs
[2026-01-02 04:43:23 UTC] USER=www-data EUID=0 PID=1435281 ACTION=fsop ARGS=chmod 750 /etc/fastorder/observability/certs
[OK]   Cert directory: /etc/fastorder/observability/certs (750 - secure)
[OK]   Lib scripts: executable (755)
[OK]   All deployment scripts: executable (755)
[OK]   All directories: accessible (755)
[OK]   ✅ All permissions verified and fixed
[CREDS] Using AWS credentials from: /var/www/.aws/credentials
[CREDS] Credential management library loaded (region: me-central-1)
[INFO] Using web-provided environment: identity-universe-main-dev
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
═══════════════════════════════════════════════════════════════════════════════
OBSERVABILITY CELL PROVISIONING
═══════════════════════════════════════════════════════════════════════════════
[INFO] Application Cell: identity-universe-main-dev
[INFO] Observability Cell: obs-identity-universe-main-dev
[INFO] Service: identity | Zone: universe | Env: dev

[INFO] Step 1/10: Provisioning network infrastructure...
[INFO]   Using existing IP for obs: 10.100.1.180
[INFO]   Allocated new IP for metrics: 10.100.1.193
[2026-01-02 04:43:25 UTC] USER=www-data EUID=0 PID=1435723 ACTION=fsop ARGS=cp /tmp/tmp.YyotqSBol5 /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
[INFO]   Allocated new IP for dashboards: 10.100.1.194
[2026-01-02 04:43:25 UTC] USER=www-data EUID=0 PID=1435740 ACTION=fsop ARGS=cp /tmp/tmp.djtsLynn8X /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
[INFO]   Allocated new IP for logstore: 10.100.1.195
[2026-01-02 04:43:25 UTC] USER=www-data EUID=0 PID=1435757 ACTION=fsop ARGS=cp /tmp/tmp.SN4Qpi6aFq /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
[INFO]   Allocated new IP for traces: 10.100.1.196
[2026-01-02 04:43:25 UTC] USER=www-data EUID=0 PID=1435774 ACTION=fsop ARGS=cp /tmp/tmp.0VQNNK8eIw /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
[INFO]   Allocated new IP for alerts: 10.100.1.197
[2026-01-02 04:43:26 UTC] USER=www-data EUID=0 PID=1435791 ACTION=fsop ARGS=cp /tmp/tmp.Rt0JnOTM8i /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
[INFO]   Allocated new IP for telemetry: 10.100.1.198
[2026-01-02 04:43:26 UTC] USER=www-data EUID=0 PID=1435817 ACTION=fsop ARGS=cp /tmp/tmp.SIlzoUs7HW /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
[INFO]   Allocated observability IPs:
[INFO]     metrics: 10.100.1.193
[INFO]     alerts: 10.100.1.197
[INFO]     dashboards: 10.100.1.194
[INFO]     traces: 10.100.1.196
[INFO]     telemetry: 10.100.1.198
[INFO]     logstore: 10.100.1.195
[INFO]     proxy: 10.100.1.180
[INFO]     obs: 10.100.1.180
[ OK ] Network infrastructure allocated
[INFO] Cleaning up ports from previous environments...
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-identity-universe-main-dev
[INFO] IP Address: 10.100.1.180
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Found 7 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.180...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup completed successfully
[INFO] Configuring IP aliases on network interface...
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING NETWORK IP ALIASES
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Interface: lo
[INFO] IP Count: 8

[INFO] Configuring: metrics → 10.100.1.193
[INFO]   Configuring IP alias: 10.100.1.193/32 on lo
[OK]     ✅ IP 10.100.1.193 configured successfully on lo
[OK]     ✅ IP 10.100.1.193 verified on network interface
[INFO] Configuring: alerts → 10.100.1.197
[INFO]   Configuring IP alias: 10.100.1.197/32 on lo
[OK]     ✅ IP 10.100.1.197 configured successfully on lo
[OK]     ✅ IP 10.100.1.197 verified on network interface
[INFO] Configuring: dashboards → 10.100.1.194
[INFO]   Configuring IP alias: 10.100.1.194/32 on lo
[OK]     ✅ IP 10.100.1.194 configured successfully on lo
[OK]     ✅ IP 10.100.1.194 verified on network interface
[INFO] Configuring: traces → 10.100.1.196
[INFO]   Configuring IP alias: 10.100.1.196/32 on lo
[OK]     ✅ IP 10.100.1.196 configured successfully on lo
[OK]     ✅ IP 10.100.1.196 verified on network interface
[INFO] Configuring: telemetry → 10.100.1.198
[INFO]   Configuring IP alias: 10.100.1.198/32 on lo
[OK]     ✅ IP 10.100.1.198 configured successfully on lo
[OK]     ✅ IP 10.100.1.198 verified on network interface
[INFO] Configuring: logstore → 10.100.1.195
[INFO]   Configuring IP alias: 10.100.1.195/32 on lo
[OK]     ✅ IP 10.100.1.195 configured successfully on lo
[OK]     ✅ IP 10.100.1.195 verified on network interface
[INFO] Configuring: proxy → 10.100.1.180
[INFO]   Configuring IP alias: 10.100.1.180/32 on lo
[OK]     ✅ IP 10.100.1.180 configured successfully on lo
[OK]     ✅ IP 10.100.1.180 verified on network interface
[INFO] Configuring: obs → 10.100.1.180
[INFO]   IP 10.100.1.180 already configured on network interface

[OK]   ═══════════════════════════════════════════════════════════════
[OK]   ✅ All IP aliases configured successfully
[OK]   ═══════════════════════════════════════════════════════════════
[INFO] Current IP configuration on lo:
      inet 127.0.0.1/8 scope host lo
      inet 10.100.60.2/32 scope global lo
      inet 10.100.1.182/32 scope global lo
      inet 10.100.1.187/32 scope global lo
      inet 10.100.1.183/32 scope global lo
      inet 10.100.1.186/32 scope global lo
      inet 10.100.1.188/32 scope global lo
      inet 10.100.1.184/32 scope global lo
      inet 10.100.1.181/32 scope global lo
      inet 10.100.1.192/32 scope global lo:pgbouncer
      inet 10.100.1.193/32 scope global lo
      inet 10.100.1.197/32 scope global lo
      inet 10.100.1.194/32 scope global lo
      inet 10.100.1.196/32 scope global lo
      inet 10.100.1.198/32 scope global lo
      inet 10.100.1.195/32 scope global lo
      inet 10.100.1.180/32 scope global lo

[OK]   IP aliases configured on network interface
[INFO] Step 2/10: Creating DNS entries...
[INFO] Configuring DNS entries in /etc/hosts...
[INFO]   Added: metrics-identity-universe-main-dev-prometheus.fastorder.com → 10.100.1.193
[INFO]   Added: alerts-identity-universe-main-dev-alertmanager.fastorder.com → 10.100.1.197
[INFO]   Added: dashboards-identity-universe-main-dev-grafana.fastorder.com → 10.100.1.194
[INFO]   Added: traces-identity-universe-main-dev-tempo.fastorder.com → 10.100.1.196
[INFO]   Added: telemetry-identity-universe-main-dev-opentelemetry.fastorder.com → 10.100.1.198
[INFO]   Added: logstore-identity-universe-main-dev-clickhouse.fastorder.com → 10.100.1.195
[INFO]   Added: observe-identity-universe-main-dev.fastorder.com → 10.100.1.180
[INFO] Adding observability integration aliases...
[INFO]   Added alias: metrics-identity-universe-main-dev.fastorder.com → 10.100.1.193
[INFO]   Added alias: alerts-identity-universe-main-dev.fastorder.com → 10.100.1.197
[INFO]   Added alias: dashboards-identity-universe-main-dev.fastorder.com → 10.100.1.194
[INFO]   Added alias: traces-identity-universe-main-dev.fastorder.com → 10.100.1.196
[INFO]   Added alias: telemetry-identity-universe-main-dev.fastorder.com → 10.100.1.198
[INFO]   Added alias: logstore-identity-universe-main-dev.fastorder.com → 10.100.1.195
[2026-01-02 04:43:27 UTC] USER=www-data EUID=0 PID=1436198 ACTION=fsop ARGS=sed -i /observe-identity-universe-main-dev.fastorder.com/d /etc/hosts
[INFO]   Added alias: observe-identity-universe-main-dev.fastorder.com → 10.100.1.180
[OK]   DNS entries created
[INFO] Step 3/10: Creating AWS Secrets Manager structure...
[INFO] Creating AWS Secrets Manager structure
[INFO]   Base path: fastorder/observability/identity/universe/dev
[INFO]   Observability Cell: obs-identity-universe-main-dev
[INFO]   Application Cell: identity-universe-main-dev
[INFO]   Exists: fastorder/observability/identity/universe/dev/metrics
[INFO]   Exists: fastorder/observability/identity/universe/dev/dashboards
[INFO]   Exists: fastorder/observability/identity/universe/dev/logstore
[INFO]   Exists: fastorder/observability/identity/universe/dev/traces
[INFO]   Exists: fastorder/observability/identity/universe/dev/telemetry
[INFO]   Exists: fastorder/observability/identity/universe/dev/alerts
[INFO] Secrets structure created successfully
[OK]   Secrets structure created
[INFO] Step 4/10: Generating mTLS certificates...
[INFO] Generating mTLS certificates for observability cell
[INFO]   Observability Cell: obs-identity-universe-main-dev
[INFO]   Components: prometheus,grafana,loki,tempo,otlp_collector,clickhouse,alertmanager
[INFO]   Creating certificate directory: /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[2026-01-02 04:43:39 UTC] USER=www-data EUID=0 PID=1436257 ACTION=fsop ARGS=mkdir -p /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[2026-01-02 04:43:39 UTC] USER=www-data EUID=0 PID=1436266 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[INFO]   Generating CA certificate for obs-identity-universe-main-dev
[2026-01-02 04:43:39 UTC] USER=www-data EUID=0 PID=1436275 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-key.pem 4096
[2026-01-02 04:43:39 UTC] USER=www-data EUID=0 PID=1436380 ACTION=fsop ARGS=openssl req -new -x509 -days 3650 -key /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-key.pem -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=obs-identity-universe-main-dev-ca
[2026-01-02 04:43:40 UTC] USER=www-data EUID=0 PID=1436389 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-key.pem
[2026-01-02 04:43:40 UTC] USER=www-data EUID=0 PID=1436398 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem
[INFO]   CA certificate created: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem
[INFO]   Generating certificate for: prometheus
[2026-01-02 04:43:40 UTC] USER=www-data EUID=0 PID=1436407 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/prometheus-key.pem 2048
[2026-01-02 04:43:40 UTC] USER=www-data EUID=0 PID=1436416 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-universe-main-dev/prometheus-key.pem -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/prometheus-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=prometheus.obs-identity-universe-main-dev
[2026-01-02 04:43:40 UTC] USER=www-data EUID=0 PID=1436425 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-universe-main-dev/prometheus-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/prometheus-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = prometheus.obs-identity-universe-main-dev
[2026-01-02 04:43:40 UTC] USER=www-data EUID=0 PID=1436434 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/prometheus-key.pem
[2026-01-02 04:43:40 UTC] USER=www-data EUID=0 PID=1436443 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/prometheus-cert.pem
[2026-01-02 04:43:40 UTC] USER=www-data EUID=0 PID=1436452 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-universe-main-dev/prometheus-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/prometheus-cert.pem
[INFO]   Generating certificate for: grafana
[2026-01-02 04:43:40 UTC] USER=www-data EUID=0 PID=1436461 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/grafana-key.pem 2048
[2026-01-02 04:43:41 UTC] USER=www-data EUID=0 PID=1436470 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-universe-main-dev/grafana-key.pem -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/grafana-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=grafana.obs-identity-universe-main-dev
[2026-01-02 04:43:41 UTC] USER=www-data EUID=0 PID=1436479 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-universe-main-dev/grafana-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/grafana-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = grafana.obs-identity-universe-main-dev
[2026-01-02 04:43:41 UTC] USER=www-data EUID=0 PID=1436488 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/grafana-key.pem
[2026-01-02 04:43:41 UTC] USER=www-data EUID=0 PID=1436497 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/grafana-cert.pem
[2026-01-02 04:43:41 UTC] USER=www-data EUID=0 PID=1436506 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-universe-main-dev/grafana-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/grafana-cert.pem
[INFO]   Generating certificate for: loki
[2026-01-02 04:43:41 UTC] USER=www-data EUID=0 PID=1436515 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/loki-key.pem 2048
[2026-01-02 04:43:41 UTC] USER=www-data EUID=0 PID=1436524 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-universe-main-dev/loki-key.pem -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/loki-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=loki.obs-identity-universe-main-dev
[2026-01-02 04:43:42 UTC] USER=www-data EUID=0 PID=1436533 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-universe-main-dev/loki-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/loki-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = loki.obs-identity-universe-main-dev
[2026-01-02 04:43:42 UTC] USER=www-data EUID=0 PID=1436542 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/loki-key.pem
[2026-01-02 04:43:42 UTC] USER=www-data EUID=0 PID=1436551 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/loki-cert.pem
[2026-01-02 04:43:42 UTC] USER=www-data EUID=0 PID=1436560 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-universe-main-dev/loki-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/loki-cert.pem
[INFO]   Generating certificate for: tempo
[2026-01-02 04:43:42 UTC] USER=www-data EUID=0 PID=1436569 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/tempo-key.pem 2048
[2026-01-02 04:43:42 UTC] USER=www-data EUID=0 PID=1436578 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-universe-main-dev/tempo-key.pem -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/tempo-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=tempo.obs-identity-universe-main-dev
[2026-01-02 04:43:42 UTC] USER=www-data EUID=0 PID=1436587 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-universe-main-dev/tempo-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/tempo-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = tempo.obs-identity-universe-main-dev
[2026-01-02 04:43:42 UTC] USER=www-data EUID=0 PID=1436596 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/tempo-key.pem
[2026-01-02 04:43:42 UTC] USER=www-data EUID=0 PID=1436605 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/tempo-cert.pem
[2026-01-02 04:43:42 UTC] USER=www-data EUID=0 PID=1436614 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-universe-main-dev/tempo-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/tempo-cert.pem
[INFO]   Generating certificate for: otlp_collector
[2026-01-02 04:43:42 UTC] USER=www-data EUID=0 PID=1436623 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/otlp_collector-key.pem 2048
[2026-01-02 04:43:43 UTC] USER=www-data EUID=0 PID=1436633 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-universe-main-dev/otlp_collector-key.pem -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/otlp_collector-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=otlp_collector.obs-identity-universe-main-dev
[2026-01-02 04:43:43 UTC] USER=www-data EUID=0 PID=1436642 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-universe-main-dev/otlp_collector-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/otlp_collector-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = otlp_collector.obs-identity-universe-main-dev
[2026-01-02 04:43:43 UTC] USER=www-data EUID=0 PID=1436651 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/otlp_collector-key.pem
[2026-01-02 04:43:43 UTC] USER=www-data EUID=0 PID=1436660 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/otlp_collector-cert.pem
[2026-01-02 04:43:43 UTC] USER=www-data EUID=0 PID=1436669 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-universe-main-dev/otlp_collector-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/otlp_collector-cert.pem
[INFO]   Generating certificate for: clickhouse
[2026-01-02 04:43:43 UTC] USER=www-data EUID=0 PID=1436678 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/clickhouse-key.pem 2048
[2026-01-02 04:43:43 UTC] USER=www-data EUID=0 PID=1436689 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-universe-main-dev/clickhouse-key.pem -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/clickhouse-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=clickhouse.obs-identity-universe-main-dev
[2026-01-02 04:43:43 UTC] USER=www-data EUID=0 PID=1436698 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-universe-main-dev/clickhouse-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/clickhouse-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = clickhouse.obs-identity-universe-main-dev
[2026-01-02 04:43:43 UTC] USER=www-data EUID=0 PID=1436707 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/clickhouse-key.pem
[2026-01-02 04:43:43 UTC] USER=www-data EUID=0 PID=1436716 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/clickhouse-cert.pem
[2026-01-02 04:43:43 UTC] USER=www-data EUID=0 PID=1436725 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-universe-main-dev/clickhouse-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/clickhouse-cert.pem
[INFO]   Generating certificate for: alertmanager
[2026-01-02 04:43:43 UTC] USER=www-data EUID=0 PID=1436734 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/alertmanager-key.pem 2048
[2026-01-02 04:43:44 UTC] USER=www-data EUID=0 PID=1436743 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-universe-main-dev/alertmanager-key.pem -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/alertmanager-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=alertmanager.obs-identity-universe-main-dev
[2026-01-02 04:43:44 UTC] USER=www-data EUID=0 PID=1436752 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-universe-main-dev/alertmanager-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/alertmanager-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = alertmanager.obs-identity-universe-main-dev
[2026-01-02 04:43:44 UTC] USER=www-data EUID=0 PID=1436761 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/alertmanager-key.pem
[2026-01-02 04:43:44 UTC] USER=www-data EUID=0 PID=1436770 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/alertmanager-cert.pem
[2026-01-02 04:43:44 UTC] USER=www-data EUID=0 PID=1436779 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-universe-main-dev/alertmanager-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/alertmanager-cert.pem
[INFO]   Generating PHP client certificate for metrics service...
[2026-01-02 04:43:44 UTC] USER=www-data EUID=0 PID=1436788 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/php-client-key.pem 2048
[2026-01-02 04:43:44 UTC] USER=www-data EUID=0 PID=1436797 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-universe-main-dev/php-client-key.pem -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/php-client-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Dashboard/CN=php-metrics-client.obs-identity-universe-main-dev
[2026-01-02 04:43:44 UTC] USER=www-data EUID=0 PID=1436806 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-universe-main-dev/php-client-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/php-client-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Dashboard, CN = php-metrics-client.obs-identity-universe-main-dev
[2026-01-02 04:43:44 UTC] USER=www-data EUID=0 PID=1436815 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/php-client-key.pem
[2026-01-02 04:43:44 UTC] USER=www-data EUID=0 PID=1436824 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/php-client-cert.pem
[2026-01-02 04:43:44 UTC] USER=www-data EUID=0 PID=1436833 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-identity-universe-main-dev/php-client-key.pem
[2026-01-02 04:43:44 UTC] USER=www-data EUID=0 PID=1436842 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-identity-universe-main-dev/php-client-cert.pem
[2026-01-02 04:43:44 UTC] USER=www-data EUID=0 PID=1436851 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-universe-main-dev/php-client-csr.pem
[INFO]   PHP client certificate created: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/php-client-cert.pem
[INFO]   Generating Apache client certificate for mTLS reverse proxy...
[2026-01-02 04:43:44 UTC] USER=www-data EUID=0 PID=1436860 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/apache-client-key.pem 2048
[2026-01-02 04:43:45 UTC] USER=www-data EUID=0 PID=1436869 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-universe-main-dev/apache-client-key.pem -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/apache-client-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=ReverseProxy/CN=apache-proxy.obs-identity-universe-main-dev
[2026-01-02 04:43:45 UTC] USER=www-data EUID=0 PID=1436878 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-universe-main-dev/apache-client-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-universe-main-dev/apache-client-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = ReverseProxy, CN = apache-proxy.obs-identity-universe-main-dev
[2026-01-02 04:43:45 UTC] USER=www-data EUID=0 PID=1436905 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/apache-client-key.pem
[2026-01-02 04:43:45 UTC] USER=www-data EUID=0 PID=1436914 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/apache-client-combined.pem
[2026-01-02 04:43:45 UTC] USER=www-data EUID=0 PID=1436923 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/apache-client-cert.pem
[2026-01-02 04:43:45 UTC] USER=www-data EUID=0 PID=1436932 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-identity-universe-main-dev/apache-client-key.pem
[2026-01-02 04:43:45 UTC] USER=www-data EUID=0 PID=1436941 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-identity-universe-main-dev/apache-client-cert.pem
[2026-01-02 04:43:45 UTC] USER=www-data EUID=0 PID=1436950 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-identity-universe-main-dev/apache-client-combined.pem
[2026-01-02 04:43:45 UTC] USER=www-data EUID=0 PID=1436959 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-universe-main-dev/apache-client-csr.pem
[INFO]   Apache client certificate created: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/apache-client-cert.pem
[INFO]   Apache combined cert+key: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/apache-client-combined.pem
[INFO]   Storing mTLS certificates in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/identity/universe/main/dev/mtls/php-client-hAPErs",
    "Name": "fastorder/observability/identity/universe/main/dev/mtls/php-client",
    "VersionId": "01aac6b3-e05f-4f78-a5e1-ec603bfa8d15"
}
[INFO]   mTLS certificates stored in Secrets Manager: fastorder/observability/identity/universe/main/dev/mtls/php-client
[INFO] mTLS certificates generated successfully
[INFO]   Certificate directory: /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[INFO]   PHP client cert: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/php-client-cert.pem
[INFO]   PHP client key: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/php-client-key.pem
[INFO]   Apache client cert: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/apache-client-cert.pem
[INFO]   Apache combined (for SSLProxyMachineCertificateFile): /etc/fastorder/observability/certs/obs-identity-universe-main-dev/apache-client-combined.pem
[OK]   mTLS certificates generated
[INFO] Step 5/10: Deploying log storage backend...
[INFO]   Provider: clickhouse (selected)
[INFO]   Note: Deployed before telemetry (OtelCol depends on log storage)
[INFO]   FQDN: logstore-identity-universe-main-dev-clickhouse.fastorder.com
[INFO]   IP: 10.100.1.195
[INFO] Deploying log backend: clickhouse...
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] LOG STORAGE BACKEND DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: clickhouse
[INFO] Observability Cell: obs-identity-universe-main-dev
[INFO] FQDN: logstore-identity-universe-main-dev-clickhouse.fastorder.com
[INFO] IP: 10.100.1.195
[INFO] S3 Bucket: fastorder-logs-universe-dev
[INFO] Retention: 90 days
[INFO] ═══════════════════════════════════════════════════════════════

[2026-01-02 04:43:49 UTC] USER=unknown EUID=33 PID=1437000 ACTION=fsop ARGS=chmod +x /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh
/bin/chmod: changing permissions of '/opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh': Operation not permitted
[INFO] Using provider: clickhouse
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=identity, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-identity-universe-main-dev...
[2026-01-02 04:43:49 UTC] USER=www-data EUID=0 PID=1437017 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-02 04:43:49 UTC] USER=www-data EUID=0 PID=1437026 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-02 04:43:49 UTC] USER=www-data EUID=0 PID=1437035 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-02 04:43:49 UTC] USER=www-data EUID=0 PID=1437044 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-identity-universe-main-dev
[INFO] IP Address: 10.100.1.195
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Found 7 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.195...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[INFO] Binding ClickHouse to allocated IP: 10.100.1.195
[INFO] Deploying ClickHouse for obs-identity-universe-main-dev
[INFO]   FQDN: logstore-identity-universe-main-dev-clickhouse.fastorder.com
[INFO]   Allocated IP: 10.100.1.195
[INFO]   VM IP: 10.100.1.195
[INFO]   Ports: HTTP=8123 TCP=9000 Interserver=9009
[INFO]   S3 Bucket: fastorder-logs-universe-dev (region=me-central-1)
[INFO]   Retention: 90 days
[INFO] Checking if ClickHouse is installed...
[OK]   ClickHouse already installed
[2026-01-02 04:43:49 UTC] USER=www-data EUID=0 PID=1437175 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-server-obs-identity-universe-main-dev/config.d
[2026-01-02 04:43:50 UTC] USER=www-data EUID=0 PID=1437184 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-server-obs-identity-universe-main-dev/users.d
[2026-01-02 04:43:50 UTC] USER=www-data EUID=0 PID=1437193 ACTION=fsop ARGS=mkdir -p /var/lib/clickhouse-obs-identity-universe-main-dev
[2026-01-02 04:43:50 UTC] USER=www-data EUID=0 PID=1437202 ACTION=fsop ARGS=mkdir -p /var/log/clickhouse-server-obs-identity-universe-main-dev
[2026-01-02 04:43:50 UTC] USER=www-data EUID=0 PID=1437211 ACTION=passthru ARGS=chmod 755 /etc/clickhouse-server-obs-identity-universe-main-dev
[2026-01-02 04:43:50 UTC] USER=www-data EUID=0 PID=1437220 ACTION=passthru ARGS=chmod 700 /var/lib/clickhouse-obs-identity-universe-main-dev
[2026-01-02 04:43:50 UTC] USER=www-data EUID=0 PID=1437229 ACTION=passthru ARGS=chmod 750 /var/log/clickhouse-server-obs-identity-universe-main-dev
[INFO] Found existing logs_writer credentials in Secrets Manager - reusing to maintain sync
[INFO] Found existing metrics_reader credentials in Secrets Manager - reusing to maintain sync
[INFO] TLS configuration exported for clickhouse
[INFO]   Cert: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/clickhouse-cert.pem
[INFO]   Key:  /etc/fastorder/observability/certs/obs-identity-universe-main-dev/clickhouse-key.pem
[INFO]   CA:   /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem
[INFO] Configuring certificate permissions for clickhouse (user: clickhouse)
[INFO] Initializing certificate directory for obs-identity-universe-main-dev...
[2026-01-02 04:43:54 UTC] USER=www-data EUID=0 PID=1437270 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-02 04:43:54 UTC] USER=www-data EUID=0 PID=1437279 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-02 04:43:54 UTC] USER=www-data EUID=0 PID=1437288 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-02 04:43:54 UTC] USER=www-data EUID=0 PID=1437297 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[INFO]   Setting file permissions...
[2026-01-02 04:43:54 UTC] USER=www-data EUID=0 PID=1437307 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/clickhouse-cert.pem
[2026-01-02 04:43:54 UTC] USER=www-data EUID=0 PID=1437316 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem
[2026-01-02 04:43:54 UTC] USER=www-data EUID=0 PID=1437325 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/clickhouse-key.pem
[INFO]   Setting file ownership...
[2026-01-02 04:43:54 UTC] USER=www-data EUID=0 PID=1437334 ACTION=passthru ARGS=chown root:clickhouse /etc/fastorder/observability/certs/obs-identity-universe-main-dev/clickhouse-key.pem
[2026-01-02 04:43:54 UTC] USER=www-data EUID=0 PID=1437343 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-identity-universe-main-dev/clickhouse-cert.pem /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for clickhouse
[INFO] Creating ClickHouse configuration...
[2026-01-02 04:43:54 UTC] USER=www-data EUID=0 PID=1437389 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /etc/clickhouse-server-obs-identity-universe-main-dev
[2026-01-02 04:43:54 UTC] USER=www-data EUID=0 PID=1437398 ACTION=passthru ARGS=bash -c chmod 640 /etc/clickhouse-server-obs-identity-universe-main-dev/*.xml
[OK]   ClickHouse configuration created
[INFO] Creating logs table schema...
[2026-01-02 04:43:54 UTC] USER=www-data EUID=0 PID=1437416 ACTION=passthru ARGS=sed -i s/__RETENTION_DAYS__/90/g /etc/clickhouse-server-obs-identity-universe-main-dev/logs_schema.sql
[2026-01-02 04:43:54 UTC] USER=www-data EUID=0 PID=1437425 ACTION=passthru ARGS=chmod 644 /etc/clickhouse-server-obs-identity-universe-main-dev/logs_schema.sql
[OK]   Logs schema created
[INFO] Creating systemd service...
[2026-01-02 04:43:54 UTC] USER=www-data EUID=0 PID=1437443 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /var/lib/clickhouse-obs-identity-universe-main-dev
[2026-01-02 04:43:54 UTC] USER=www-data EUID=0 PID=1437452 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /var/log/clickhouse-server-obs-identity-universe-main-dev
[2026-01-02 04:43:54 UTC] USER=www-data EUID=0 PID=1437461 ACTION=passthru ARGS=chmod 700 /var/lib/clickhouse-obs-identity-universe-main-dev
[OK]   Systemd service created
[INFO] Starting ClickHouse service...
[2026-01-02 04:43:54 UTC] USER=www-data EUID=0 PID=1437470 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:43:55 UTC] USER=www-data EUID=0 PID=1437515 ACTION=passthru ARGS=systemctl enable clickhouse-server-obs-identity-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/clickhouse-server-obs-identity-universe-main-dev.service -> /etc/systemd/system/clickhouse-server-obs-identity-universe-main-dev.service.
[2026-01-02 04:43:56 UTC] USER=www-data EUID=0 PID=1437560 ACTION=passthru ARGS=systemctl start clickhouse-server-obs-identity-universe-main-dev.service
[INFO] Waiting for ClickHouse to be ready...
[OK]   ClickHouse is ready
[INFO] Initializing database schema...
[OK]   Schema initialized
[INFO] Storing ClickHouse credentials in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/identity/universe/main/dev/clickhouse/server/logs_writer-1bySvx",
    "Name": "fastorder/observability/identity/universe/main/dev/clickhouse/server/logs_writer",
    "VersionId": "52215f9e-9652-4593-978e-d2462f20434d"
}
[OK]   logs_writer credentials stored and verified in Secrets Manager
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/identity/universe/main/dev/clickhouse/server/metrics_reader-FWUJ3A",
    "Name": "fastorder/observability/identity/universe/main/dev/clickhouse/server/metrics_reader",
    "VersionId": "253f2129-adf4-4418-baf3-d72fc3f685a9"
}
[OK]   metrics_reader credentials stored and verified in Secrets Manager
[INFO] Validating ClickHouse deployment...
[INFO] ClickHouse version: 25.10.1.3832
[INFO] Tables created: .inner_id.011e41e4-8dea-4166-b115-664bc47cf7f6
.inner_id.5536e2fc-d26f-4212-a547-565b2c4b92f1
application_logs
error_logs_mv
iam_audit_event
metrics_all
otel_logs
request_logs_mv
security_access
[INFO] Test log inserted. Total logs: 1
[OK]   ✅ ClickHouse deployment validated

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ ClickHouse Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   FQDN: logstore-identity-universe-main-dev-clickhouse.fastorder.com
[INFO]   IP: 10.100.1.195
[INFO]   HTTP Port: 8123
[INFO]   Native Port: 9000
[INFO]   Database: logs
[INFO]   Retention: 90 days
[INFO]   Storage: Tiered (Local → S3: fastorder-logs-universe-dev in me-central-1)
[INFO] 
[INFO] Credentials stored in AWS Secrets Manager:
[INFO]   Writers: fastorder/observability/identity/universe/main/dev/clickhouse/server/logs_writer
[INFO]   Readers: fastorder/observability/identity/universe/main/dev/clickhouse/server/metrics_reader (for PHP metrics service)
[INFO] 
[INFO] Example queries (using credentials from Secrets Manager):
[INFO]   # Write logs:
[INFO]   clickhouse-client --host logstore-identity-universe-main-dev-clickhouse.fastorder.com --port 9000 --user logs_writer --password '***' --query 'SELECT 1'
[INFO] 
[INFO]   # Read metrics (PHP metrics service):
[INFO]   clickhouse-client --host logstore-identity-universe-main-dev-clickhouse.fastorder.com --port 9000 --user metrics_reader --password '***' --query 'SELECT * FROM system.metrics'
[INFO] 
[INFO] HTTPS Setup (run on web-03/skeleton server):
[INFO]   # Set up HTTPS reverse proxy with Let's Encrypt:
[INFO]   OBS_CELL=obs-identity-universe-main-dev BACKEND_IP=10.100.1.195 sudo bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/../https/setup-clickhouse-https.sh
[INFO] 
[INFO]   # Or add --setup-https flag when running this script
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Log Storage Backend Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: clickhouse
[INFO] FQDN: logstore-identity-universe-main-dev-clickhouse.fastorder.com
[INFO] IP: 10.100.1.195
[INFO] Retention: 90 days
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering ClickHouse in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       ClickHouse
[INFO]   Identifier:        identity-universe-main-dev-clickhouse
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.195
[INFO]   Port:              8443
[INFO]   FQDN:              logstore-identity-universe-main-dev-clickhouse.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-universe-main-dev (service=identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ INVALID REQUEST
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] Response: {"success":false,"error":"Invalid JSON: Control character error, possibly incorrectly encoded"}
[ERROR] 
[ERROR] Request payload:
  {
    "env_id": "identity-universe-main-dev",
    "application": "ClickHouse",
    "identifier": "identity-universe-main-dev-clickhouse",
    "identifier_parent": "cluster",
    "ip": "10.100.1.195",
    "port": 8443,
    "fqdn": "logstore-identity-universe-main-dev-clickhouse.fastorder.com",
    "status": "running",
    "meta": {
      "role": "log_storage",
      "provider": "clickhouse",
      "version": "25.10
  1.3832",
      "http_port": 8123,
      "native_port": 9000,
      "https_port": 8443,
      "protocol": "https",
      "metrics_enabled": true,
      "metrics_port": 8123,
      "metrics_path": "/metrics",
      "health_endpoint": "https://logstore-identity-universe-main-dev-clickhouse.fastorder.com/ping",
      "retention_days": 90,
      "s3_bucket": "fastorder-logs-universe-dev"
  }
  }
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[WARN] ⚠️  Failed to register ClickHouse (service is running)
[OK]   clickhouse deployed successfully
[OK]   Log storage backend deployed
[INFO] Step 6/10: Deploying telemetry collector...
[INFO]   Provider: otlp (backend implementation - internal)
[INFO]   Endpoint: telemetry-identity-universe-main-dev-opentelemetry.fastorder.com (stable, exposed to clients)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] TELEMETRY COLLECTOR DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: otlp
[INFO] Observability Cell: obs-identity-universe-main-dev
[INFO] FQDN: telemetry-identity-universe-main-dev-opentelemetry.fastorder.com
[INFO] IP: 10.100.1.198
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: otlp
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Telemetry/provider/otlp.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=identity, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-identity-universe-main-dev...
[2026-01-02 04:44:11 UTC] USER=www-data EUID=0 PID=1438463 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-02 04:44:11 UTC] USER=www-data EUID=0 PID=1438472 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-02 04:44:11 UTC] USER=www-data EUID=0 PID=1438481 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-02 04:44:11 UTC] USER=www-data EUID=0 PID=1438490 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-identity-universe-main-dev
[INFO] IP Address: 10.100.1.198
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-identity-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Found 8 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.198...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[INFO] Binding to allocated IP: 10.100.1.198
[INFO] Deploying OpenTelemetry Collector for observability cell: obs-identity-universe-main-dev
[INFO] FQDN:         telemetry-identity-universe-main-dev-opentelemetry.fastorder.com
[INFO] Allocated IP: 10.100.1.198
[INFO] VM IP:        10.100.1.198
[INFO] Ports:        gRPC=4317 HTTP=4318 Metrics=8888 Prom=8889
[OK]   User 'otelcol' already exists
[INFO] Checking if OpenTelemetry Collector is installed...
[OK]   OpenTelemetry Collector already installed at /usr/local/bin/otelcol-contrib
[INFO] Creating configuration/data directories...
[2026-01-02 04:44:12 UTC] USER=www-data EUID=0 PID=1438627 ACTION=passthru ARGS=mkdir -p /etc/otelcol/obs-identity-universe-main-dev
[2026-01-02 04:44:12 UTC] USER=www-data EUID=0 PID=1438636 ACTION=passthru ARGS=mkdir -p /var/lib/otelcol/obs-identity-universe-main-dev
[2026-01-02 04:44:12 UTC] USER=www-data EUID=0 PID=1438646 ACTION=passthru ARGS=chown -R otelcol:otelcol /etc/otelcol/obs-identity-universe-main-dev /var/lib/otelcol/obs-identity-universe-main-dev
[2026-01-02 04:44:12 UTC] USER=www-data EUID=0 PID=1438655 ACTION=passthru ARGS=chmod 0750 /etc/otelcol/obs-identity-universe-main-dev
[2026-01-02 04:44:12 UTC] USER=www-data EUID=0 PID=1438664 ACTION=passthru ARGS=chmod 0750 /var/lib/otelcol/obs-identity-universe-main-dev
[INFO] Retrieving ClickHouse credentials from Secrets Manager...
[OK]   Retrieved ClickHouse credentials from Secrets Manager
[INFO] Creating OpenTelemetry Collector configuration...
[INFO] ClickHouse exporter enabled: tcp://logstore-identity-universe-main-dev-clickhouse.fastorder.com:9000
[2026-01-02 04:44:14 UTC] USER=www-data EUID=0 PID=1438706 ACTION=passthru ARGS=chown otelcol:otelcol /etc/otelcol/obs-identity-universe-main-dev/config.yaml
[2026-01-02 04:44:14 UTC] USER=www-data EUID=0 PID=1438715 ACTION=passthru ARGS=chmod 0640 /etc/otelcol/obs-identity-universe-main-dev/config.yaml
[OK]   Configuration created at /etc/otelcol/obs-identity-universe-main-dev/config.yaml
[INFO] Setting up TLS certificate permissions...
[INFO] Configuring certificate permissions for otlp_collector (user: otelcol)
[INFO] Initializing certificate directory for obs-identity-universe-main-dev...
[2026-01-02 04:44:14 UTC] USER=www-data EUID=0 PID=1438724 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-02 04:44:14 UTC] USER=www-data EUID=0 PID=1438733 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-02 04:44:14 UTC] USER=www-data EUID=0 PID=1438742 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-02 04:44:14 UTC] USER=www-data EUID=0 PID=1438751 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[INFO]   Setting file permissions...
[2026-01-02 04:44:14 UTC] USER=www-data EUID=0 PID=1438761 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/otlp_collector-cert.pem
[2026-01-02 04:44:14 UTC] USER=www-data EUID=0 PID=1438770 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem
[2026-01-02 04:44:14 UTC] USER=www-data EUID=0 PID=1438779 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/otlp_collector-key.pem
[INFO]   Setting file ownership...
[2026-01-02 04:44:14 UTC] USER=www-data EUID=0 PID=1438788 ACTION=passthru ARGS=chown root:otelcol /etc/fastorder/observability/certs/obs-identity-universe-main-dev/otlp_collector-key.pem
[2026-01-02 04:44:14 UTC] USER=www-data EUID=0 PID=1438797 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-identity-universe-main-dev/otlp_collector-cert.pem /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for otlp_collector
[OK]   Certificate permissions configured
[INFO] Creating systemd service: otelcol-obs-identity-universe-main-dev
[OK]   Systemd service created at /etc/systemd/system/otelcol-obs-identity-universe-main-dev.service
[INFO] Adding /etc/hosts entry for telemetry-identity-universe-main-dev-opentelemetry.fastorder.com -> 10.100.1.198
[2026-01-02 04:44:14 UTC] USER=www-data EUID=0 PID=1438817 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*telemetry-identity-universe-main-dev-opentelemetry.fastorder.com/10.100.1.198    telemetry-identity-universe-main-dev-opentelemetry.fastorder.com/ /etc/hosts
[OK]   Updated /etc/hosts entry to use VM_IP
[INFO] Storing OTLP configuration metadata in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/identity/universe/main/dev/otlp/collector-pqGgov",
    "Name": "fastorder/observability/identity/universe/main/dev/otlp/collector",
    "VersionId": "d52e75d7-03a3-401d-a82b-3b2412e8448e"
}
[OK]   Configuration metadata stored/updated in AWS Secrets Manager: fastorder/observability/identity/universe/main/dev/otlp/collector
[INFO] Enabling and starting OpenTelemetry Collector service...
[2026-01-02 04:44:16 UTC] USER=www-data EUID=0 PID=1438831 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:44:17 UTC] USER=www-data EUID=0 PID=1438876 ACTION=passthru ARGS=systemctl enable otelcol-obs-identity-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/otelcol-obs-identity-universe-main-dev.service -> /etc/systemd/system/otelcol-obs-identity-universe-main-dev.service.
[2026-01-02 04:44:17 UTC] USER=www-data EUID=0 PID=1438921 ACTION=passthru ARGS=systemctl restart otelcol-obs-identity-universe-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-01-02 04:44:20 UTC] USER=www-data EUID=0 PID=1438953 ACTION=passthru ARGS=systemctl is-active --quiet otelcol-obs-identity-universe-main-dev.service
[OK]   ✅ OpenTelemetry Collector is running
[OK]   ✅ gRPC endpoint listening on port 4317
[OK]   ✅ HTTP endpoint listening on port 4318
[OK]   ✅ Prometheus metrics endpoint listening on port 8889
[INFO] Service logs (last 10 lines):
[2026-01-02 04:44:20 UTC] USER=www-data EUID=0 PID=1438968 ACTION=passthru ARGS=journalctl -u otelcol-obs-identity-universe-main-dev.service -n 10 --no-pager
Jan 02 04:44:18 web-03 otelcol-obs-identity-universe-main-dev[1438928]: 2026-01-02T04:44:18.148Z        info        internal/resourcedetection.go:125        began detecting resource information        {"kind": "processor", "name": "resourcedetection", "pipeline": "traces"}
Jan 02 04:44:18 web-03 otelcol-obs-identity-universe-main-dev[1438928]: 2026-01-02T04:44:18.149Z        info        system/system.go:201        This attribute changed from int to string. Temporarily switch back to int using the feature gate.        {"kind": "processor", "name": "resourcedetection", "pipeline": "traces", "attribute": "host.cpu.family", "feature gate": "processor.resourcedetection.hostCPUModelAndFamilyAsString"}
Jan 02 04:44:18 web-03 otelcol-obs-identity-universe-main-dev[1438928]: 2026-01-02T04:44:18.149Z        info        system/system.go:220        This attribute changed from int to string. Temporarily switch back to int using the feature gate.        {"kind": "processor", "name": "resourcedetection", "pipeline": "traces", "attribute": "host.cpu.model.id", "feature gate": "processor.resourcedetection.hostCPUModelAndFamilyAsString"}
Jan 02 04:44:18 web-03 otelcol-obs-identity-universe-main-dev[1438928]: 2026-01-02T04:44:18.149Z        info        internal/resourcedetection.go:139        detected resource information        {"kind": "processor", "name": "resourcedetection", "pipeline": "traces", "resource": {"host.name":"web-03","os.type":"linux"}}
Jan 02 04:44:18 web-03 otelcol-obs-identity-universe-main-dev[1438928]: 2026-01-02T04:44:18.199Z        info        otlpreceiver@v0.91.0/otlp.go:83        Starting GRPC server        {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "10.100.1.198:4317"}
Jan 02 04:44:18 web-03 otelcol-obs-identity-universe-main-dev[1438928]: 2026-01-02T04:44:18.199Z        info        otlpreceiver@v0.91.0/otlp.go:101        Starting HTTP server        {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "10.100.1.198:4318"}
Jan 02 04:44:18 web-03 otelcol-obs-identity-universe-main-dev[1438928]: 2026-01-02T04:44:18.200Z        info        prometheusreceiver@v0.91.0/metrics_receiver.go:240        Starting discovery manager        {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
Jan 02 04:44:18 web-03 otelcol-obs-identity-universe-main-dev[1438928]: 2026-01-02T04:44:18.200Z        info        prometheusreceiver@v0.91.0/metrics_receiver.go:231        Scrape job added        {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "jobName": "otel-collector"}
Jan 02 04:44:18 web-03 otelcol-obs-identity-universe-main-dev[1438928]: 2026-01-02T04:44:18.201Z        info        service@v0.91.0/service.go:171        Everything is ready. Begin running and processing data.
Jan 02 04:44:18 web-03 otelcol-obs-identity-universe-main-dev[1438928]: 2026-01-02T04:44:18.201Z        info        prometheusreceiver@v0.91.0/metrics_receiver.go:282        Starting scrape manager        {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Telemetry Collector Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: otlp
[INFO] FQDN: telemetry-identity-universe-main-dev-opentelemetry.fastorder.com
[INFO] IP: 10.100.1.198
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering OpenTelemetry Collector in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       OpenTelemetry Collector
[INFO]   Identifier:        identity-universe-main-dev-opentelemetry
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.198
[INFO]   Port:              4317
[INFO]   FQDN:              telemetry-identity-universe-main-dev-opentelemetry.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-universe-main-dev (service=identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 9309b136-2d2b-4fb2-ab55-c49e398173d2
[SUCCESS] Environment UUID: b6092921-3a75-44ac-9080-96a7ca43bec0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✅ OpenTelemetry Collector registered in dashboard
[INFO] Setting up OpenTelemetry Collector metrics collection timer...
[2026-01-02 04:44:21 UTC] USER=www-data EUID=0 PID=1439026 ACTION=passthru ARGS=mv /tmp/otelcol-metrics-identity-universe-main-dev.service /etc/systemd/system/
[2026-01-02 04:44:21 UTC] USER=www-data EUID=0 PID=1439035 ACTION=passthru ARGS=mv /tmp/otelcol-metrics-identity-universe-main-dev.timer /etc/systemd/system/
[2026-01-02 04:44:21 UTC] USER=www-data EUID=0 PID=1439044 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:44:22 UTC] USER=www-data EUID=0 PID=1439089 ACTION=passthru ARGS=systemctl enable otelcol-metrics-identity-universe-main-dev.timer
Created symlink /etc/systemd/system/timers.target.wants/otelcol-metrics-identity-universe-main-dev.timer -> /etc/systemd/system/otelcol-metrics-identity-universe-main-dev.timer.
[2026-01-02 04:44:23 UTC] USER=www-data EUID=0 PID=1439134 ACTION=passthru ARGS=systemctl start otelcol-metrics-identity-universe-main-dev.timer
[OK]   ✅ Metrics collection timer installed and started
[OK]   Telemetry collector (otlp) deployed successfully

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Step 7/10: METRICS BACKEND DEPLOYMENT
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Provider: prometheus
[INFO]   OBS Cell: obs-identity-universe-main-dev
[INFO]   FQDN: metrics-identity-universe-main-dev-prometheus.fastorder.com
[INFO]   IP: 10.100.1.193
[INFO]   Script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/deploy-metrics.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📊 METRICS DEPLOYMENT WRAPPER STARTED
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Script: deploy-metrics.sh
[INFO] Timestamp: 2026-01-02 04:44:23 UTC
[INFO] Arguments: --provider prometheus --obs-cell obs-identity-universe-main-dev --fqdn metrics-identity-universe-main-dev-prometheus.fastorder.com --ip 10.100.1.193

[INFO] ═══════════════════════════════════════════════════════════════
[INFO] METRICS DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: prometheus
[INFO] Observability Cell: obs-identity-universe-main-dev
[INFO] FQDN: metrics-identity-universe-main-dev-prometheus.fastorder.com
[INFO] IP: 10.100.1.193
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: prometheus
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/provider/prometheus.sh

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/provider/prometheus.sh
[INFO]   OBS_CELL: obs-identity-universe-main-dev
[INFO]   FQDN: metrics-identity-universe-main-dev-prometheus.fastorder.com
[INFO]   IP: 10.100.1.193
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🚀 PROMETHEUS DEPLOYMENT STARTED
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Script: prometheus.sh
[INFO] Timestamp: 2026-01-02 04:44:23 UTC
[INFO] Arguments: --obs-cell obs-identity-universe-main-dev --fqdn metrics-identity-universe-main-dev-prometheus.fastorder.com --ip 10.100.1.193

[INFO] Parsed: SERVICE=identity, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Step 1/12: Sourcing centralized libraries...
[INFO]   Library directory: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/lib
[INFO]   Sourcing port_allocator.sh...
[OK]     ✓ port_allocator.sh loaded
[INFO]   Sourcing cert_permissions.sh...
[OK]     ✓ cert_permissions.sh loaded
[INFO]   Sourcing port_cleanup.sh...
[OK]     ✓ port_cleanup.sh loaded
[OK]   Step 1/12: Libraries sourced successfully

[INFO] Step 2/12: Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-identity-universe-main-dev...
[2026-01-02 04:44:23 UTC] USER=www-data EUID=0 PID=1439194 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-02 04:44:23 UTC] USER=www-data EUID=0 PID=1439226 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-02 04:44:23 UTC] USER=www-data EUID=0 PID=1439255 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-02 04:44:23 UTC] USER=www-data EUID=0 PID=1439264 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-identity-universe-main-dev
[INFO] IP Address: 10.100.1.193
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-identity-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-identity-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service prometheus-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Found 10 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.193...

[INFO] Scanning 15 ports...

[INFO]   ⚠️  NodeExporter: 10.100.1.193:9100 - OCCUPIED

[WARN] Found 1 occupied port(s) out of 15 total
[WARN] Will attempt to free occupied ports...

[OK]   Port 10.100.1.193:9100 occupied but service obs-identity-universe-main-dev is running (OK - idempotent)

[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Port Cleanup Summary for obs-identity-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Total ports checked:  15
[INFO] Already free:         14
[INFO] Occupied (cleaned):   1
[OK]   Successfully freed:   1

[OK]   ✅ All ports are now FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[OK]   Step 2/12: Port cleanup completed

[INFO] Step 3/12: Allocating ports...
[OK]   Step 3/12: Ports allocated

[INFO] Step 4/12: Setting up configuration...
[INFO]   Observability cell: obs-identity-universe-main-dev
[INFO]   FQDN: metrics-identity-universe-main-dev-prometheus.fastorder.com
[INFO]   IP: 10.100.1.193
[INFO]   Prometheus Port: 9090
[INFO] Step 5/12: Checking if Prometheus is installed...
[OK]   Prometheus already installed at /usr/local/bin/prometheus
[OK]   Step 5/12: Prometheus binary ready

[INFO] Step 5.1/12: Creating configuration directories early (required for Node Exporter config)...
[INFO]   Config: /etc/prometheus/obs-identity-universe-main-dev
[INFO]   Data: /var/lib/prometheus/obs-identity-universe-main-dev
[INFO]   Rules: /etc/prometheus/obs-identity-universe-main-dev/rules
[2026-01-02 04:44:24 UTC] USER=www-data EUID=0 PID=1439454 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-identity-universe-main-dev
[2026-01-02 04:44:24 UTC] USER=www-data EUID=0 PID=1439463 ACTION=passthru ARGS=mkdir -p /var/lib/prometheus/obs-identity-universe-main-dev
[2026-01-02 04:44:24 UTC] USER=www-data EUID=0 PID=1439472 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-identity-universe-main-dev/rules
[2026-01-02 04:44:24 UTC] USER=www-data EUID=0 PID=1439481 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-identity-universe-main-dev/targets
[OK]   Step 5.1/12: Directories created early

[INFO] Step 6/12: Setting up Node Exporter...
[INFO] Checking if Node Exporter is installed...
[OK]   Node Exporter already installed at /usr/local/bin/node_exporter
[INFO] Creating Node Exporter TLS web config...
[INFO] Creating Node Exporter systemd service with TLS...
[2026-01-02 04:44:24 UTC] USER=www-data EUID=0 PID=1439508 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:44:24 UTC] USER=www-data EUID=0 PID=1439553 ACTION=passthru ARGS=systemctl enable node_exporter-obs-identity-universe-main-dev.service
[2026-01-02 04:44:25 UTC] USER=www-data EUID=0 PID=1439598 ACTION=passthru ARGS=systemctl restart node_exporter-obs-identity-universe-main-dev.service
[OK]   Step 6/12: Node Exporter ready

[INFO] Step 7/12: Creating configuration directories...
[INFO]   Config: /etc/prometheus/obs-identity-universe-main-dev
[INFO]   Data: /var/lib/prometheus/obs-identity-universe-main-dev
[INFO]   Rules: /etc/prometheus/obs-identity-universe-main-dev/rules
[2026-01-02 04:44:25 UTC] USER=www-data EUID=0 PID=1439612 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-identity-universe-main-dev
[2026-01-02 04:44:25 UTC] USER=www-data EUID=0 PID=1439621 ACTION=passthru ARGS=mkdir -p /var/lib/prometheus/obs-identity-universe-main-dev
[2026-01-02 04:44:25 UTC] USER=www-data EUID=0 PID=1439630 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-identity-universe-main-dev/rules
[OK]   Step 7/12: Directories created

[INFO] Step 8/12: Creating Prometheus configuration...
[INFO] Generated FQDNs:
[INFO]   Prometheus:   metrics-identity-universe-main-dev-prometheus.fastorder.com
[INFO]   Alertmanager: alerts-identity-universe-main-dev-alertmanager.fastorder.com
[INFO]   Grafana:      dashboards-identity-universe-main-dev-grafana.fastorder.com
[INFO]   Otelcol:      telemetry-identity-universe-main-dev-opentelemetry.fastorder.com
[OK]   Step 8/12: Configuration created at /etc/prometheus/obs-identity-universe-main-dev/prometheus.yml

[INFO] Step 9/12: Creating TLS/HTTPS web config...
[OK]   Step 9/12: Web config created at /etc/prometheus/obs-identity-universe-main-dev/web-config.yml
[INFO]   TLS cert: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/prometheus-cert.pem
[INFO]   TLS key: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/prometheus-key.pem
[INFO]   CA cert: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem

[INFO] Creating basic alerting rules...
[OK]   Alerting rules created
[2026-01-02 04:44:25 UTC] USER=www-data EUID=0 PID=1439666 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-identity-universe-main-dev/targets
[2026-01-02 04:44:25 UTC] USER=www-data EUID=0 PID=1439675 ACTION=passthru ARGS=bash -c cat > '/etc/prometheus/obs-identity-universe-main-dev/targets/.placeholder.yml' << 'EOF'
# Placeholder file to prevent file_sd_configs warning
# Application targets will be added here automatically
[]
EOF
[INFO] Step 10/12: Creating systemd service...
[INFO]   Service: prometheus-obs-identity-universe-main-dev
[INFO] Binding to: 10.100.1.193:9090
[OK]   Step 10/12: Systemd service created at /etc/systemd/system/prometheus-obs-identity-universe-main-dev.service

[INFO] Step 11/12: Configuring certificate permissions...
[INFO]   Looking for certificates in: /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[OK]     ✓ All certificate files exist
[INFO] Configuring certificate permissions for prometheus (user: root)
[INFO] Initializing certificate directory for obs-identity-universe-main-dev...
[2026-01-02 04:44:25 UTC] USER=www-data EUID=0 PID=1439694 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-02 04:44:25 UTC] USER=www-data EUID=0 PID=1439703 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-02 04:44:26 UTC] USER=www-data EUID=0 PID=1439712 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-02 04:44:26 UTC] USER=www-data EUID=0 PID=1439721 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[INFO]   Setting file permissions...
[2026-01-02 04:44:26 UTC] USER=www-data EUID=0 PID=1439731 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/prometheus-cert.pem
[2026-01-02 04:44:26 UTC] USER=www-data EUID=0 PID=1439740 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem
[2026-01-02 04:44:26 UTC] USER=www-data EUID=0 PID=1439749 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/prometheus-key.pem
[INFO]   Setting file ownership...
[2026-01-02 04:44:26 UTC] USER=www-data EUID=0 PID=1439758 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-identity-universe-main-dev/prometheus-key.pem
[2026-01-02 04:44:26 UTC] USER=www-data EUID=0 PID=1439767 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-identity-universe-main-dev/prometheus-cert.pem /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for prometheus
[OK]   Step 11/12: Certificate permissions configured

[INFO] Adding /etc/hosts entry for metrics-identity-universe-main-dev-prometheus.fastorder.com -> 10.100.1.193
[2026-01-02 04:44:26 UTC] USER=www-data EUID=0 PID=1439778 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*metrics-identity-universe-main-dev-prometheus.fastorder.com/10.100.1.193    metrics-identity-universe-main-dev-prometheus.fastorder.com/ /etc/hosts
[OK]   Updated /etc/hosts entry to use VM_IP
[INFO] Validating Prometheus configuration...
Checking /etc/prometheus/obs-identity-universe-main-dev/prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: /etc/prometheus/obs-identity-universe-main-dev/prometheus.yml is valid prometheus config file syntax

Checking /etc/prometheus/obs-identity-universe-main-dev/rules/basic_alerts.yml
  SUCCESS: 4 rules found

[OK]   ✅ Configuration is valid
[INFO] Storing Prometheus configuration in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/identity/universe/main/dev/prometheus/server-iJyIMW",
    "Name": "fastorder/observability/identity/universe/main/dev/prometheus/server",
    "VersionId": "afe03cfa-a170-4adf-a5dd-440da3c95263"
}
[OK]   Configuration stored in AWS Secrets Manager
[INFO] Step 12/12: Starting Prometheus service...
[INFO]   Reloading systemd daemon...
[2026-01-02 04:44:28 UTC] USER=www-data EUID=0 PID=1439808 ACTION=passthru ARGS=systemctl daemon-reload
[OK]     ✓ Systemd daemon reloaded
[INFO]   Enabling service...
[2026-01-02 04:44:29 UTC] USER=www-data EUID=0 PID=1439855 ACTION=passthru ARGS=systemctl enable prometheus-obs-identity-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus-obs-identity-universe-main-dev.service -> /etc/systemd/system/prometheus-obs-identity-universe-main-dev.service.
[OK]     ✓ Service enabled
[INFO]   Starting service...
[2026-01-02 04:44:29 UTC] USER=www-data EUID=0 PID=1439900 ACTION=passthru ARGS=systemctl restart prometheus-obs-identity-universe-main-dev.service
[OK]     ✓ Service start command issued

[INFO] Validating Prometheus deployment...
[2026-01-02 04:44:32 UTC] USER=www-data EUID=0 PID=1439921 ACTION=passthru ARGS=systemctl is-active --quiet prometheus-obs-identity-universe-main-dev.service
[OK]   ✅ Prometheus is running
[OK]   ✅ Prometheus web interface listening on port 9090
[OK]   ✅ Prometheus health check passed (HTTPS)
[INFO] ═══════════════════════════════════════════════════════════════
[OK]   Prometheus Web UI: https://metrics-identity-universe-main-dev-prometheus.fastorder.com:9090
[OK]   Targets: https://metrics-identity-universe-main-dev-prometheus.fastorder.com:9090/targets
[OK]   Alerts: https://metrics-identity-universe-main-dev-prometheus.fastorder.com:9090/alerts
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service logs (last 10 lines):
[2026-01-02 04:44:34 UTC] USER=www-data EUID=0 PID=1439936 ACTION=passthru ARGS=journalctl -u prometheus-obs-identity-universe-main-dev.service -n 10 --no-pager
Jan 02 04:44:29 web-03 prometheus-obs-identity-universe-main-dev[1439907]: ts=2026-01-02T04:44:29.853Z caller=head.go:682 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=9.667µs
Jan 02 04:44:29 web-03 prometheus-obs-identity-universe-main-dev[1439907]: ts=2026-01-02T04:44:29.853Z caller=head.go:690 level=info component=tsdb msg="Replaying WAL, this may take a while"
Jan 02 04:44:29 web-03 prometheus-obs-identity-universe-main-dev[1439907]: ts=2026-01-02T04:44:29.854Z caller=head.go:761 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
Jan 02 04:44:29 web-03 prometheus-obs-identity-universe-main-dev[1439907]: ts=2026-01-02T04:44:29.854Z caller=head.go:798 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=103.802µs wal_replay_duration=743.588µs wbl_replay_duration=402ns total_replay_duration=1.39812ms
Jan 02 04:44:29 web-03 prometheus-obs-identity-universe-main-dev[1439907]: ts=2026-01-02T04:44:29.859Z caller=main.go:1045 level=info fs_type=EXT4_SUPER_MAGIC
Jan 02 04:44:29 web-03 prometheus-obs-identity-universe-main-dev[1439907]: ts=2026-01-02T04:44:29.859Z caller=main.go:1048 level=info msg="TSDB started"
Jan 02 04:44:29 web-03 prometheus-obs-identity-universe-main-dev[1439907]: ts=2026-01-02T04:44:29.859Z caller=main.go:1230 level=info msg="Loading configuration file" filename=/etc/prometheus/obs-identity-universe-main-dev/prometheus.yml
Jan 02 04:44:29 web-03 prometheus-obs-identity-universe-main-dev[1439907]: ts=2026-01-02T04:44:29.862Z caller=main.go:1267 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/obs-identity-universe-main-dev/prometheus.yml totalDuration=3.093938ms db_storage=2.242µs remote_storage=2.421µs web_handler=858ns query_engine=1.934µs scrape=483.006µs scrape_sd=105.662µs notify=57.597µs notify_sd=12.825µs rules=1.713329ms tracing=12.766µs
Jan 02 04:44:29 web-03 prometheus-obs-identity-universe-main-dev[1439907]: ts=2026-01-02T04:44:29.862Z caller=main.go:1009 level=info msg="Server is ready to receive web requests."
Jan 02 04:44:29 web-03 prometheus-obs-identity-universe-main-dev[1439907]: ts=2026-01-02T04:44:29.862Z caller=manager.go:1012 level=info component="rule manager" msg="Starting rule manager..."
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Provider script completed with exit code: 0
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Metrics Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: prometheus
[INFO] FQDN: metrics-identity-universe-main-dev-prometheus.fastorder.com
[INFO] IP: 10.100.1.193
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Step 7/10: METRICS DEPLOYMENT RESULT
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Exit code: 0
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ Metrics backend (prometheus) deployed successfully
[INFO] Step 8/10: Deploying traces backend...
[INFO]   Provider: tempo (selected)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] TRACES DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: tempo
[INFO] Observability Cell: obs-identity-universe-main-dev
[INFO] FQDN: traces-identity-universe-main-dev-tempo.fastorder.com
[INFO] IP: 10.100.1.196
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: tempo
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Traces/provider/tempo.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=identity, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-identity-universe-main-dev...
[2026-01-02 04:44:34 UTC] USER=www-data EUID=0 PID=1439956 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-02 04:44:34 UTC] USER=www-data EUID=0 PID=1439965 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-02 04:44:34 UTC] USER=www-data EUID=0 PID=1439974 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-02 04:44:35 UTC] USER=www-data EUID=0 PID=1439983 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-identity-universe-main-dev
[INFO] IP Address: 10.100.1.196
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service clickhouse-server-obs-identity-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-identity-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service prometheus-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-identity-universe-main-dev.service belongs to current cell (skipping)
[INFO] Service grafana-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Found 11 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.196...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[INFO] Binding Tempo to allocated IP: 10.100.1.196
[INFO] Deploying Grafana Tempo for observability cell: obs-identity-universe-main-dev
[INFO] FQDN: traces-identity-universe-main-dev-tempo.fastorder.com
[INFO] Allocated IP: 10.100.1.196
[INFO] VM IP: 10.100.1.196
[INFO] Ports: HTTP=3200 gRPC=9095, OTLP gRPC=4317, OTLP HTTP=4318
[INFO] Checking if Grafana Tempo is installed...
[OK]   Grafana Tempo already installed at /usr/local/bin/tempo
[INFO] Preparing configuration and data directories...
[2026-01-02 04:44:35 UTC] USER=www-data EUID=0 PID=1440135 ACTION=passthru ARGS=mkdir -p /etc/tempo/obs-identity-universe-main-dev
[2026-01-02 04:44:35 UTC] USER=www-data EUID=0 PID=1440144 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-identity-universe-main-dev
[2026-01-02 04:44:35 UTC] USER=www-data EUID=0 PID=1440153 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-identity-universe-main-dev/wal
[2026-01-02 04:44:35 UTC] USER=www-data EUID=0 PID=1440162 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-identity-universe-main-dev/blocks
[2026-01-02 04:44:35 UTC] USER=www-data EUID=0 PID=1440171 ACTION=passthru ARGS=chown -R tempo:tempo /etc/tempo/obs-identity-universe-main-dev /var/lib/tempo/obs-identity-universe-main-dev
[2026-01-02 04:44:35 UTC] USER=www-data EUID=0 PID=1440180 ACTION=passthru ARGS=chmod 750 /etc/tempo/obs-identity-universe-main-dev /var/lib/tempo/obs-identity-universe-main-dev
[INFO] Creating Grafana Tempo configuration...
[INFO] TLS configuration exported for tempo
[INFO]   Cert: /etc/fastorder/observability/certs/obs-identity-universe-main-dev/tempo-cert.pem
[INFO]   Key:  /etc/fastorder/observability/certs/obs-identity-universe-main-dev/tempo-key.pem
[INFO]   CA:   /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem
[INFO] Setting up certificate permissions for Tempo...
[INFO] Configuring certificate permissions for tempo (user: tempo)
[INFO] Initializing certificate directory for obs-identity-universe-main-dev...
[2026-01-02 04:44:35 UTC] USER=www-data EUID=0 PID=1440195 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-02 04:44:35 UTC] USER=www-data EUID=0 PID=1440204 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-02 04:44:35 UTC] USER=www-data EUID=0 PID=1440213 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-02 04:44:35 UTC] USER=www-data EUID=0 PID=1440222 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[INFO]   Setting file permissions...
[2026-01-02 04:44:35 UTC] USER=www-data EUID=0 PID=1440232 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/tempo-cert.pem
[2026-01-02 04:44:35 UTC] USER=www-data EUID=0 PID=1440241 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem
[2026-01-02 04:44:35 UTC] USER=www-data EUID=0 PID=1440250 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-identity-universe-main-dev/tempo-key.pem
[INFO]   Setting file ownership...
[2026-01-02 04:44:35 UTC] USER=www-data EUID=0 PID=1440259 ACTION=passthru ARGS=chown root:tempo /etc/fastorder/observability/certs/obs-identity-universe-main-dev/tempo-key.pem
[2026-01-02 04:44:35 UTC] USER=www-data EUID=0 PID=1440268 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-identity-universe-main-dev/tempo-cert.pem /etc/fastorder/observability/certs/obs-identity-universe-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for tempo
[2026-01-02 04:44:36 UTC] USER=www-data EUID=0 PID=1440287 ACTION=passthru ARGS=chown tempo:tempo /etc/tempo/obs-identity-universe-main-dev/config.yaml
[2026-01-02 04:44:36 UTC] USER=www-data EUID=0 PID=1440296 ACTION=passthru ARGS=chmod 640 /etc/tempo/obs-identity-universe-main-dev/config.yaml
[OK]   Configuration created at /etc/tempo/obs-identity-universe-main-dev/config.yaml
[INFO] Creating systemd service: tempo-obs-identity-universe-main-dev
[OK]   Systemd service created
[INFO] Adding /etc/hosts entry for traces-identity-universe-main-dev-tempo.fastorder.com -> 10.100.1.196
[2026-01-02 04:44:36 UTC] USER=www-data EUID=0 PID=1440315 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*traces-identity-universe-main-dev-tempo.fastorder.com/10.100.1.196    traces-identity-universe-main-dev-tempo.fastorder.com/ /etc/hosts
[OK]   Updated /etc/hosts entry to use VM_IP
[INFO] Storing Tempo configuration in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/identity/universe/main/dev/tempo/server-K33Bhl",
    "Name": "fastorder/observability/identity/universe/main/dev/tempo/server",
    "VersionId": "2e290008-2ff7-4e80-9066-69b474e7b361"
}
[OK]   Tempo configuration stored/updated in AWS Secrets Manager: fastorder/observability/identity/universe/main/dev/tempo/server
[WARN] Port cleanup library not found, skipping automatic cleanup
[INFO] Adding iptables redirect for Tempo internal communication (optional)...
[2026-01-02 04:44:38 UTC] USER=www-data EUID=0 PID=1440340 ACTION=passthru ARGS=iptables -t nat -A OUTPUT -p tcp -d 127.0.0.1 --dport 9095 -j DNAT --to-destination 10.100.1.196:9095
ERROR: passthru not allowed: iptables
[WARN] Could not add iptables redirect (iptables not allowed in wrapper)
[WARN] Tempo will still work - clients should connect to 10.100.1.196:9095 directly
[INFO] Enabling and starting Grafana Tempo service...
[2026-01-02 04:44:38 UTC] USER=www-data EUID=0 PID=1440348 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:44:38 UTC] USER=www-data EUID=0 PID=1440395 ACTION=passthru ARGS=systemctl enable tempo-obs-identity-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/tempo-obs-identity-universe-main-dev.service -> /etc/systemd/system/tempo-obs-identity-universe-main-dev.service.
[2026-01-02 04:44:39 UTC] USER=www-data EUID=0 PID=1440440 ACTION=passthru ARGS=systemctl restart tempo-obs-identity-universe-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-01-02 04:44:42 UTC] USER=www-data EUID=0 PID=1440556 ACTION=passthru ARGS=systemctl is-active --quiet tempo-obs-identity-universe-main-dev.service
[OK]   ✅ Grafana Tempo is running
[OK]   ✅ HTTP endpoint listening on port 3200
[OK]   ✅ OTLP gRPC endpoint listening on port 4317
[OK]   ✅ OTLP HTTP endpoint listening on port 4318
[INFO] Service logs (last 10 lines):
[2026-01-02 04:44:42 UTC] USER=www-data EUID=0 PID=1440571 ACTION=passthru ARGS=journalctl -u tempo-obs-identity-universe-main-dev.service -n 10 --no-pager
Jan 02 04:44:39 web-03 tempo-obs-identity-universe-main-dev[1440447]: level=info ts=2026-01-02T04:44:39.561015079Z caller=lifecycler.go:493 msg="auto-joining cluster after timeout" ring=ingester
Jan 02 04:44:39 web-03 tempo-obs-identity-universe-main-dev[1440447]: level=info ts=2026-01-02T04:44:39.561184412Z caller=ring.go:297 msg="ring doesn't exist in KV store yet"
Jan 02 04:44:39 web-03 tempo-obs-identity-universe-main-dev[1440447]: level=info ts=2026-01-02T04:44:39.560725863Z caller=module_service.go:82 msg=starting module=distributor
Jan 02 04:44:39 web-03 tempo-obs-identity-universe-main-dev[1440447]: level=info ts=2026-01-02T04:44:39.562032788Z caller=basic_lifecycler.go:297 msg="instance not found in the ring" instance=web-03 ring=compactor
Jan 02 04:44:39 web-03 tempo-obs-identity-universe-main-dev[1440447]: level=info ts=2026-01-02T04:44:39.562976494Z caller=worker.go:246 msg="total worker concurrency updated" totalConcurrency=20
Jan 02 04:44:39 web-03 tempo-obs-identity-universe-main-dev[1440447]: ts=2026-01-02T04:44:39Z level=info msg="Starting GRPC server" component=tempo endpoint=10.100.1.196:4317
Jan 02 04:44:39 web-03 tempo-obs-identity-universe-main-dev[1440447]: ts=2026-01-02T04:44:39Z level=info msg="Starting HTTP server" component=tempo endpoint=10.100.1.196:4318
Jan 02 04:44:39 web-03 tempo-obs-identity-universe-main-dev[1440447]: level=info ts=2026-01-02T04:44:39.566430181Z caller=compactor.go:127 msg="waiting until compactor is ACTIVE in the ring"
Jan 02 04:44:39 web-03 tempo-obs-identity-universe-main-dev[1440447]: level=info ts=2026-01-02T04:44:39.568185079Z caller=compactor.go:133 msg="compactor is ACTIVE in the ring"
Jan 02 04:44:39 web-03 tempo-obs-identity-universe-main-dev[1440447]: level=info ts=2026-01-02T04:44:39.568238742Z caller=compactor.go:142 msg="waiting until compactor ring topology is stable" min_waiting=1m0s max_waiting=5m0s

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Traces Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: tempo
[INFO] FQDN: traces-identity-universe-main-dev-tempo.fastorder.com
[INFO] IP: 10.100.1.196
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering Tempo in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Tempo
[INFO]   Identifier:        identity-universe-main-dev-tempo
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.196
[INFO]   Port:              3200
[INFO]   FQDN:              traces-identity-universe-main-dev-tempo.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-universe-main-dev (service=identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: f451b3bc-e53e-4fc2-8be7-5644df03d3aa
[SUCCESS] Environment UUID: b6092921-3a75-44ac-9080-96a7ca43bec0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✅ Tempo registered in dashboard
[OK]   Traces backend (tempo) deployed successfully
[INFO] Step 9/10: Deploying dashboards...
[INFO]   Provider: grafana (selected)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] DASHBOARDS DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: grafana
[INFO] Observability Cell: obs-identity-universe-main-dev
[INFO] FQDN: dashboards-identity-universe-main-dev-grafana.fastorder.com
[INFO] IP: 10.100.1.194
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: grafana
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Dashboards/provider/grafana.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=identity, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Binding to allocated IP: 10.100.1.194
[INFO] Deploying Grafana for observability cell: obs-identity-universe-main-dev
[INFO] FQDN: dashboards-identity-universe-main-dev-grafana.fastorder.com
[INFO] Allocated IP: 10.100.1.194
[INFO] VM IP: 10.100.1.194
[INFO] HTTP Port: 3000
[INFO] Checking if Grafana is installed...
[OK]   Grafana already installed
[INFO] Installing Grafana plugins...
[INFO] Installing ClickHouse datasource plugin...
[WARN] Failed to install ClickHouse plugin (may need internet access)
[INFO] Validating TLS certificate and key...
[INFO] Setting certificate permissions...
[OK]   TLS cert/key found and permissions set
[INFO] Creating configuration and data directories...
[2026-01-02 04:44:43 UTC] USER=www-data EUID=0 PID=1440641 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-identity-universe-main-dev
[2026-01-02 04:44:43 UTC] USER=www-data EUID=0 PID=1440650 ACTION=passthru ARGS=mkdir -p /var/lib/grafana/obs-identity-universe-main-dev
[2026-01-02 04:44:43 UTC] USER=www-data EUID=0 PID=1440659 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-identity-universe-main-dev/provisioning/datasources
[2026-01-02 04:44:43 UTC] USER=www-data EUID=0 PID=1440668 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-identity-universe-main-dev/provisioning/dashboards
[2026-01-02 04:44:43 UTC] USER=www-data EUID=0 PID=1440677 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-identity-universe-main-dev/provisioning/notifiers
[INFO] Creating Grafana configuration at /etc/grafana/obs-identity-universe-main-dev/grafana.ini...
[OK]   Configuration created
[INFO] Creating Prometheus datasource provisioning...
[OK]   Prometheus datasource provisioned
[INFO] Creating Tempo datasource provisioning...
[OK]   Tempo datasource provisioned
[INFO] Creating Loki datasource provisioning...
[OK]   Loki datasource provisioned
[INFO] Creating ClickHouse datasource provisioning...
[OK]   Retrieved ClickHouse credentials from Secrets Manager
[OK]   ClickHouse datasource provisioned
[INFO] Creating systemd service: grafana-obs-identity-universe-main-dev
[OK]   Systemd service created
[2026-01-02 04:44:45 UTC] USER=www-data EUID=0 PID=1440765 ACTION=passthru ARGS=chown -R grafana:grafana /etc/grafana/obs-identity-universe-main-dev
[2026-01-02 04:44:45 UTC] USER=www-data EUID=0 PID=1440774 ACTION=passthru ARGS=chown -R grafana:grafana /var/lib/grafana/obs-identity-universe-main-dev
[2026-01-02 04:44:45 UTC] USER=www-data EUID=0 PID=1440783 ACTION=passthru ARGS=chmod 750 /etc/grafana/obs-identity-universe-main-dev /var/lib/grafana/obs-identity-universe-main-dev
[INFO] Adding /etc/hosts entry for dashboards-identity-universe-main-dev-grafana.fastorder.com -> 10.100.1.194
[WARN] /etc/hosts entry already exists
[INFO] Storing Grafana credentials in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/identity/universe/main/dev/grafana/admin-lH9cm8",
    "Name": "fastorder/observability/identity/universe/main/dev/grafana/admin",
    "VersionId": "674733b5-7ca1-46b6-9dfe-cbfda1a412e3"
}
[OK]   Credentials stored in AWS Secrets Manager: fastorder/observability/identity/universe/main/dev/grafana/admin
[INFO] Enabling and starting Grafana service...
[2026-01-02 04:44:47 UTC] USER=www-data EUID=0 PID=1440798 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:44:48 UTC] USER=www-data EUID=0 PID=1440844 ACTION=passthru ARGS=systemctl enable grafana-obs-identity-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/grafana-obs-identity-universe-main-dev.service -> /etc/systemd/system/grafana-obs-identity-universe-main-dev.service.
[2026-01-02 04:44:48 UTC] USER=www-data EUID=0 PID=1440891 ACTION=passthru ARGS=systemctl restart grafana-obs-identity-universe-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-01-02 04:44:54 UTC] USER=www-data EUID=0 PID=1440951 ACTION=passthru ARGS=systemctl is-active --quiet grafana-obs-identity-universe-main-dev.service
[OK]   ✅ Grafana is running
[OK]   ✅ Grafana web interface listening on port 3000
[INFO] ═══════════════════════════════════════════════════════════════
[OK]   Grafana Dashboard URL: https://dashboards-identity-universe-main-dev-grafana.fastorder.com:3000
[OK]   Username: admin
[OK]   Password is stored in AWS Secrets Manager at: fastorder/observability/identity/universe/main/dev/grafana/admin
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service logs (last 10 lines):
[2026-01-02 04:44:54 UTC] USER=www-data EUID=0 PID=1440962 ACTION=passthru ARGS=journalctl -u grafana-obs-identity-universe-main-dev.service -n 10 --no-pager
Jan 02 04:44:54 web-03 grafana-obs-identity-universe-main-dev[1440900]: logger=migrator t=2026-01-02T04:44:54.016066141Z level=info msg="Migration successfully executed" id="create builtin role table" duration=960.181µs
Jan 02 04:44:54 web-03 grafana-obs-identity-universe-main-dev[1440900]: logger=migrator t=2026-01-02T04:44:54.0238718Z level=info msg="Executing migration" id="add index builtin_role.role_id"
Jan 02 04:44:54 web-03 grafana-obs-identity-universe-main-dev[1440900]: logger=migrator t=2026-01-02T04:44:54.025994569Z level=info msg="Migration successfully executed" id="add index builtin_role.role_id" duration=2.123922ms
Jan 02 04:44:54 web-03 grafana-obs-identity-universe-main-dev[1440900]: logger=migrator t=2026-01-02T04:44:54.032548997Z level=info msg="Executing migration" id="add index builtin_role.name"
Jan 02 04:44:54 web-03 grafana-obs-identity-universe-main-dev[1440900]: logger=migrator t=2026-01-02T04:44:54.034912377Z level=info msg="Migration successfully executed" id="add index builtin_role.name" duration=2.363596ms
Jan 02 04:44:54 web-03 grafana-obs-identity-universe-main-dev[1440900]: logger=migrator t=2026-01-02T04:44:54.043128471Z level=info msg="Executing migration" id="Add column org_id to builtin_role table"
Jan 02 04:44:54 web-03 grafana-obs-identity-universe-main-dev[1440900]: logger=migrator t=2026-01-02T04:44:54.052852002Z level=info msg="Migration successfully executed" id="Add column org_id to builtin_role table" duration=9.718269ms
Jan 02 04:44:54 web-03 grafana-obs-identity-universe-main-dev[1440900]: logger=migrator t=2026-01-02T04:44:54.060559261Z level=info msg="Executing migration" id="add index builtin_role.org_id"
Jan 02 04:44:54 web-03 grafana-obs-identity-universe-main-dev[1440900]: logger=migrator t=2026-01-02T04:44:54.062563704Z level=info msg="Migration successfully executed" id="add index builtin_role.org_id" duration=2.006517ms
Jan 02 04:44:54 web-03 grafana-obs-identity-universe-main-dev[1440900]: logger=migrator t=2026-01-02T04:44:54.06859384Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role"

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Dashboards Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: grafana
[INFO] FQDN: dashboards-identity-universe-main-dev-grafana.fastorder.com
[INFO] IP: 10.100.1.194
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering Grafana in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Grafana
[INFO]   Identifier:        identity-universe-main-dev-grafana
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.194
[INFO]   Port:              3000
[INFO]   FQDN:              dashboards-identity-universe-main-dev-grafana.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-universe-main-dev (service=identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: b1de8d21-7aef-4e9c-a3d9-bb0a41ac3f53
[SUCCESS] Environment UUID: b6092921-3a75-44ac-9080-96a7ca43bec0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✅ Grafana registered in dashboard
[OK]   Dashboards (grafana) deployed successfully
[INFO] Step 10/10: Deploying alerting...
[INFO]   Provider: alertmanager (selected)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] ALERTING DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: alertmanager
[INFO] Observability Cell: obs-identity-universe-main-dev
[INFO] FQDN: alerts-identity-universe-main-dev-alertmanager.fastorder.com
[INFO] IP: 10.100.1.197
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Ports: Web=9093 Cluster=9094 (bound to IP: 10.100.1.197)

[INFO] Using provider: alertmanager
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Alerting/provider/alertmanager.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=identity, ZONE=universe, BRANCH=main, ENV=dev
[INFO] Binding to allocated IP: 10.100.1.197
[INFO] Deploying Alertmanager for observability cell: obs-identity-universe-main-dev
[INFO] FQDN: alerts-identity-universe-main-dev-alertmanager.fastorder.com
[INFO] Allocated IP: 10.100.1.197
[INFO] VM IP: 10.100.1.197
[INFO] Ports: Web=9093 Cluster=9094
[INFO] Checking if Alertmanager is installed...
[OK]   Alertmanager already installed at /usr/local/bin/alertmanager
[INFO] Validating TLS certificate and key...
[OK]   TLS cert/key found in /etc/fastorder/observability/certs/obs-identity-universe-main-dev
[INFO] Creating configuration and data directories...
[2026-01-02 04:44:54 UTC] USER=www-data EUID=0 PID=1441022 ACTION=passthru ARGS=mkdir -p /etc/alertmanager/obs-identity-universe-main-dev
[2026-01-02 04:44:55 UTC] USER=www-data EUID=0 PID=1441031 ACTION=passthru ARGS=mkdir -p /var/lib/alertmanager/obs-identity-universe-main-dev
[2026-01-02 04:44:55 UTC] USER=www-data EUID=0 PID=1441040 ACTION=passthru ARGS=mkdir -p /etc/alertmanager/obs-identity-universe-main-dev/templates
[INFO] Creating Alertmanager configuration...
[OK]   Alertmanager configuration created at /etc/alertmanager/obs-identity-universe-main-dev/alertmanager.yml
[INFO] Creating notification templates...
[OK]   Notification templates created
[INFO] Creating Alertmanager web TLS configuration with mTLS...
[OK]   Web mTLS configuration created at /etc/alertmanager/obs-identity-universe-main-dev/web-config.yml
[INFO] Validating Alertmanager configuration...
[2026-01-02 04:44:55 UTC] USER=www-data EUID=0 PID=1441090 ACTION=passthru ARGS=chmod 644 /etc/alertmanager/obs-identity-universe-main-dev/alertmanager.yml
Checking '/etc/alertmanager/obs-identity-universe-main-dev/alertmanager.yml'  SUCCESS
Found:
 - global config
 - route
 - 6 inhibit rules
 - 5 receivers
 - 1 templates
  SUCCESS

[OK]   ✅ Configuration is valid
[INFO] Creating systemd service: alertmanager-obs-identity-universe-main-dev
[OK]   Systemd service created
[2026-01-02 04:44:55 UTC] USER=www-data EUID=0 PID=1441116 ACTION=passthru ARGS=chown alertmanager:alertmanager /etc/fastorder/observability/certs/obs-identity-universe-main-dev/alertmanager-key.pem
[2026-01-02 04:44:55 UTC] USER=www-data EUID=0 PID=1441125 ACTION=passthru ARGS=chown alertmanager:alertmanager /etc/fastorder/observability/certs/obs-identity-universe-main-dev/alertmanager-cert.pem
[2026-01-02 04:44:55 UTC] USER=www-data EUID=0 PID=1441147 ACTION=passthru ARGS=chown -R alertmanager:alertmanager /etc/alertmanager/obs-identity-universe-main-dev
[2026-01-02 04:44:55 UTC] USER=www-data EUID=0 PID=1441157 ACTION=passthru ARGS=chown -R alertmanager:alertmanager /var/lib/alertmanager/obs-identity-universe-main-dev
[2026-01-02 04:44:55 UTC] USER=www-data EUID=0 PID=1441166 ACTION=passthru ARGS=chmod 750 /etc/alertmanager/obs-identity-universe-main-dev /var/lib/alertmanager/obs-identity-universe-main-dev
[INFO] Adding /etc/hosts entry for alerts-identity-universe-main-dev-alertmanager.fastorder.com -> 10.100.1.197
[WARN] /etc/hosts entry already exists
[INFO] Storing Alertmanager configuration in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/identity/universe/main/dev/alertmanager/server-oPwmnx",
    "Name": "fastorder/observability/identity/universe/main/dev/alertmanager/server",
    "VersionId": "6a9107e9-e543-41e8-9e28-19c5df2a0953"
}
[OK]   Configuration stored in AWS Secrets Manager: fastorder/observability/identity/universe/main/dev/alertmanager/server
[INFO] Enabling and starting Alertmanager service...
[2026-01-02 04:44:58 UTC] USER=www-data EUID=0 PID=1441216 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:44:58 UTC] USER=www-data EUID=0 PID=1441282 ACTION=passthru ARGS=systemctl enable alertmanager-obs-identity-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/alertmanager-obs-identity-universe-main-dev.service -> /etc/systemd/system/alertmanager-obs-identity-universe-main-dev.service.
[2026-01-02 04:44:59 UTC] USER=www-data EUID=0 PID=1441330 ACTION=passthru ARGS=systemctl restart alertmanager-obs-identity-universe-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-01-02 04:45:02 UTC] USER=www-data EUID=0 PID=1441832 ACTION=passthru ARGS=systemctl is-active --quiet alertmanager-obs-identity-universe-main-dev.service
[OK]   ✅ Alertmanager is running
[OK]   ✅ Alertmanager HTTPS web interface listening on port 9093
[OK]   ✅ Alertmanager cluster port listening on port 9094
[WARN] ⚠️  Alertmanager health check not responding yet (HTTPS)
[INFO] ═══════════════════════════════════════════════════════════════
[OK]   Alertmanager Web UI: https://alerts-identity-universe-main-dev-alertmanager.fastorder.com:9093
[OK]   API Endpoint:        https://alerts-identity-universe-main-dev-alertmanager.fastorder.com:9093/api/v2
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service logs (last 10 lines):
[2026-01-02 04:45:02 UTC] USER=www-data EUID=0 PID=1442052 ACTION=passthru ARGS=journalctl -u alertmanager-obs-identity-universe-main-dev.service -n 10 --no-pager
Jan 02 04:44:59 web-03 systemd[1]: Started Alertmanager - obs-identity-universe-main-dev.
Jan 02 04:44:59 web-03 alertmanager-obs-identity-universe-main-dev[1441337]: ts=2026-01-02T04:44:59.607Z caller=main.go:245 level=info msg="Starting Alertmanager" version="(version=0.26.0, branch=HEAD, revision=d7b4f0c7322e7151d6e3b1e31cbc15361e295d8d)"
Jan 02 04:44:59 web-03 alertmanager-obs-identity-universe-main-dev[1441337]: ts=2026-01-02T04:44:59.608Z caller=main.go:246 level=info build_context="(go=go1.20.7, platform=linux/amd64, user=root@df8d7debeef4, date=20230824-11:11:58, tags=netgo)"
Jan 02 04:44:59 web-03 alertmanager-obs-identity-universe-main-dev[1441337]: ts=2026-01-02T04:44:59.615Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s
Jan 02 04:44:59 web-03 alertmanager-obs-identity-universe-main-dev[1441337]: ts=2026-01-02T04:44:59.691Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/obs-identity-universe-main-dev/alertmanager.yml
Jan 02 04:44:59 web-03 alertmanager-obs-identity-universe-main-dev[1441337]: ts=2026-01-02T04:44:59.694Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/obs-identity-universe-main-dev/alertmanager.yml
Jan 02 04:44:59 web-03 alertmanager-obs-identity-universe-main-dev[1441337]: ts=2026-01-02T04:44:59.701Z caller=tls_config.go:274 level=info msg="Listening on" address=10.100.1.197:9093
Jan 02 04:44:59 web-03 alertmanager-obs-identity-universe-main-dev[1441337]: ts=2026-01-02T04:44:59.702Z caller=tls_config.go:310 level=info msg="TLS is enabled." http2=true address=10.100.1.197:9093
Jan 02 04:45:01 web-03 alertmanager-obs-identity-universe-main-dev[1441337]: ts=2026-01-02T04:45:01.617Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001093732s
Jan 02 04:45:02 web-03 alertmanager-obs-identity-universe-main-dev[1441337]: 2026/01/02 04:45:02 http: TLS handshake error from 10.100.1.197:47316: tls: client didn't provide a certificate

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Alerting Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: alertmanager
[INFO] FQDN: alerts-identity-universe-main-dev-alertmanager.fastorder.com
[INFO] IP: 10.100.1.197
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Setting up HTTPS reverse proxy...
[INFO] Backend port: 9093
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Alertmanager HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-identity-universe-main-dev
  FQDN:         alerts-identity-universe-main-dev-alertmanager.fastorder.com
  Backend:      https://alerts-identity-universe-main-dev-alertmanager.fastorder.com:9093/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.197
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[ERROR] This script must be run as root or with sudo
[WARN] ⚠️  HTTPS setup failed (Alertmanager is still running on HTTP)
[INFO] Registering Alertmanager in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Alertmanager
[INFO]   Identifier:        identity-universe-main-dev-alertmanager
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.197
[INFO]   Port:              9093
[INFO]   FQDN:              alerts-identity-universe-main-dev-alertmanager.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-universe-main-dev (service=identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 4a568ddd-801f-4d5b-b11e-d5423eff0ecf
[SUCCESS] Environment UUID: b6092921-3a75-44ac-9080-96a7ca43bec0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✅ Alertmanager registered in dashboard
[OK]   Alerting (alertmanager) deployed successfully
[INFO] Step 10.5: Deploying Blackbox Exporter for synthetic monitoring...
[BLACKBOX] Starting Blackbox Exporter deployment for obs-identity-universe-main-dev
[BLACKBOX] VM IP: 10.100.1.193
[BLACKBOX] Version: 0.25.0
[BLACKBOX] Checking prerequisites...
[BLACKBOX] Creating directories...
[BLACKBOX] Downloading Blackbox Exporter v0.25.0...
Sorry, user www-data is not allowed to execute '/usr/bin/mv /tmp/tmp.woumQZCzGe/blackbox_exporter-0.25.0.linux-amd64/blackbox_exporter /usr/local/bin/' as root on web-03.
[WARN] Blackbox Exporter deployment failed (non-fatal, synthetic monitoring disabled)
[INFO] Step 11/13: Configuring HTTPS reverse proxies...
[INFO] Setting up Prometheus HTTPS proxy...
[2026-01-02 04:45:06 UTC] USER=www-data EUID=0 PID=1443203 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/https/setup-prometheus-https.sh --obs-cell obs-identity-universe-main-dev --backend-ip 10.100.1.193
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Prometheus HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-identity-universe-main-dev
  FQDN:         metrics-identity-universe-main-dev-prometheus.fastorder.com
  Backend:      https://metrics-identity-universe-main-dev-prometheus.fastorder.com:9090/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.193
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity (will retry up to 60s)...
[OK] Backend is accessible
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate exists and is valid for 89 more days
[INFO] Creating HTTPS VirtualHost with reverse proxy...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Setting up certificate auto-renewal...
[OK] Auto-renewal configured
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated
[INFO] Verifying HTTPS setup...
[OK] HTTPS endpoint is working

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Prometheus HTTPS Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HTTPS Endpoints:
  Health:   https://metrics-identity-universe-main-dev-prometheus.fastorder.com/-/healthy
  Ready:    https://metrics-identity-universe-main-dev-prometheus.fastorder.com/-/ready
  Graph:    https://metrics-identity-universe-main-dev-prometheus.fastorder.com/graph
  Targets:  https://metrics-identity-universe-main-dev-prometheus.fastorder.com/targets
  Alerts:   https://metrics-identity-universe-main-dev-prometheus.fastorder.com/alerts
  API:      https://metrics-identity-universe-main-dev-prometheus.fastorder.com/api/v1/...

Apache VirtualHosts:
  HTTP:  /etc/apache2/sites-available/metrics-identity-universe-main-dev-prometheus.fastorder.com.conf
  HTTPS: /etc/apache2/sites-available/metrics-identity-universe-main-dev-prometheus.fastorder.com-ssl.conf

Certificate:
  Path:     /etc/letsencrypt/live/metrics-identity-universe-main-dev-prometheus.fastorder.com/
  Renewal:  certbot renew --cert-name metrics-identity-universe-main-dev-prometheus.fastorder.com

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   Prometheus HTTPS proxy configured
[INFO] Setting up Grafana HTTPS proxy...
[2026-01-02 04:45:11 UTC] USER=www-data EUID=0 PID=1444832 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Dashboards/https/setup-grafana-https.sh --obs-cell obs-identity-universe-main-dev --backend-ip 10.100.1.194
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-identity-universe-main-dev
  FQDN:         dashboards-identity-universe-main-dev-grafana.fastorder.com
  Backend:      https://dashboards-identity-universe-main-dev-grafana.fastorder.com:3000/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.194
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate already exists
[INFO] Creating HTTPS VirtualHost...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana HTTPS Setup Complete!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  Grafana URL: https://dashboards-identity-universe-main-dev-grafana.fastorder.com/
  Metrics:     https://dashboards-identity-universe-main-dev-grafana.fastorder.com/metrics

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   Grafana HTTPS proxy configured
[INFO] Setting up OpenTelemetry Collector HTTPS proxy...
[2026-01-02 04:45:12 UTC] USER=www-data EUID=0 PID=1445363 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Telemetry/https/setup-otelcol-https.sh --obs-cell obs-identity-universe-main-dev --backend-ip 10.100.1.198
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OpenTelemetry Collector HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-identity-universe-main-dev
  FQDN:         telemetry-identity-universe-main-dev-opentelemetry.fastorder.com
  Backend:      http://telemetry-identity-universe-main-dev-opentelemetry.fastorder.com:8888/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.198
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[OK] Backend is accessible and returning metrics via HTTPS
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate exists and is valid for 89 more days
[INFO] Creating HTTPS VirtualHost with reverse proxy...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Setting up certificate auto-renewal...
[OK] Auto-renewal configured
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated
[INFO] Verifying HTTPS setup...
[OK] HTTPS endpoint is working and returning metrics

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OpenTelemetry Collector HTTPS Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HTTPS Endpoints:
  Metrics: https://telemetry-identity-universe-main-dev-opentelemetry.fastorder.com/metrics

Apache VirtualHosts:
  HTTP:  /etc/apache2/sites-available/telemetry-identity-universe-main-dev-opentelemetry.fastorder.com.conf
  HTTPS: /etc/apache2/sites-available/telemetry-identity-universe-main-dev-opentelemetry.fastorder.com-ssl.conf

Certificate:
  Path:     /etc/letsencrypt/live/telemetry-identity-universe-main-dev-opentelemetry.fastorder.com/
  Renewal:  certbot renew --cert-name telemetry-identity-universe-main-dev-opentelemetry.fastorder.com

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   OpenTelemetry Collector HTTPS proxy configured
[INFO] Setting up ClickHouse HTTPS proxy...
[2026-01-02 04:45:16 UTC] USER=www-data EUID=0 PID=1447358 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/https/setup-clickhouse-https.sh --obs-cell obs-identity-universe-main-dev --backend-ip 10.100.1.195
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ClickHouse HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-identity-universe-main-dev
  FQDN:         logstore-identity-universe-main-dev.fastorder.com
  Backend:      http://logstore-identity-universe-main-dev.fastorder.com:8123/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.195
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity (will retry up to 60s)...
[OK] Backend is accessible
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate exists and is valid for 89 more days
[INFO] Creating HTTPS VirtualHost with reverse proxy...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Setting up certificate auto-renewal...
[OK] Auto-renewal configured
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated
[INFO] Verifying HTTPS setup...
[OK] HTTPS endpoint is working

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ ClickHouse HTTPS Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HTTPS Endpoints:
  Health:     https://logstore-identity-universe-main-dev.fastorder.com/
  Dashboard:  https://logstore-identity-universe-main-dev.fastorder.com/dashboard
  Playground: https://logstore-identity-universe-main-dev.fastorder.com/play
  Metrics:    https://logstore-identity-universe-main-dev.fastorder.com/metrics

Login Instructions:
  1. Get credentials from skeleton: POST /api/monitoring/clickhouse/credentials
  2. Use auto-login URL: https://logstore-identity-universe-main-dev.fastorder.com/dashboard#user=<USER>&password=<PASS>
  3. Or use skeleton monitoring dashboard for one-click access

Apache VirtualHosts:
  HTTP:  /etc/apache2/sites-available/logstore-identity-universe-main-dev.fastorder.com.conf
  HTTPS: /etc/apache2/sites-available/logstore-identity-universe-main-dev.fastorder.com-ssl.conf

Certificate:
  Path: /etc/letsencrypt/live/logstore-identity-universe-main-dev.fastorder.com/
  Auto-renewal: Enabled via certbot.timer

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ClickHouse HTTPS proxy configured
[INFO] Setting up Tempo HTTPS proxy...
[2026-01-02 04:45:20 UTC] USER=www-data EUID=0 PID=1449605 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Traces/https/setup-tempo-https.sh --obs-cell obs-identity-universe-main-dev --backend-ip 10.100.1.196
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana Tempo HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-identity-universe-main-dev
  FQDN:         traces-identity-universe-main-dev-tempo.fastorder.com
  Backend:      https://traces-identity-universe-main-dev-tempo.fastorder.com:3200/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.196
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[WARN] Cannot verify Tempo health endpoint (it may not be running yet), continuing anyway...
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate already exists
[INFO] Generating Apache client certificate for mTLS backend connection...
[OK] Apache client certificate already exists
[INFO] Creating HTTPS VirtualHost with mTLS backend...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana Tempo HTTPS Setup Complete!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  Tempo URL:    https://traces-identity-universe-main-dev-tempo.fastorder.com/
  Ready:        https://traces-identity-universe-main-dev-tempo.fastorder.com/ready
  Metrics:      https://traces-identity-universe-main-dev-tempo.fastorder.com/metrics
  Build Info:   https://traces-identity-universe-main-dev-tempo.fastorder.com/api/status/buildinfo

  Note: Tempo backend must be running at traces-identity-universe-main-dev-tempo.fastorder.com:3200 (10.100.1.196)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   Tempo HTTPS proxy configured
[INFO] Setting up Alertmanager HTTPS proxy...
[2026-01-02 04:45:20 UTC] USER=www-data EUID=0 PID=1449684 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Alerting/https/setup-alertmanager-https.sh --obs-cell obs-identity-universe-main-dev --backend-ip 10.100.1.197
[INFO] Backend port: 9093
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Alertmanager HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-identity-universe-main-dev
  FQDN:         alerts-identity-universe-main-dev-alertmanager.fastorder.com
  Backend:      https://alerts-identity-universe-main-dev-alertmanager.fastorder.com:9093/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.197
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[WARN] Backend health check inconclusive - proceeding anyway
[INFO] Creating HTTP VirtualHost for ACME challenge...
[OK] HTTP VirtualHost created
[INFO] Obtaining Let's Encrypt certificate...
[OK] Certificate exists and is valid for 89 more days
[INFO] Creating HTTPS VirtualHost with reverse proxy...
[OK] HTTPS VirtualHost created and Apache reloaded
[INFO] Setting up certificate auto-renewal...
[OK] Auto-renewal configured
[INFO] Updating /etc/hosts...
[OK] /etc/hosts updated
[INFO] Verifying HTTPS setup...
OK[OK] HTTPS endpoint is working

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Alertmanager HTTPS Setup Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HTTPS Endpoints:
  Health:   https://alerts-identity-universe-main-dev-alertmanager.fastorder.com/-/healthy
  Ready:    https://alerts-identity-universe-main-dev-alertmanager.fastorder.com/-/ready
  Web UI:   https://alerts-identity-universe-main-dev-alertmanager.fastorder.com/
  API:      https://alerts-identity-universe-main-dev-alertmanager.fastorder.com/api/v2/...

Apache VirtualHosts:
  HTTP:  /etc/apache2/sites-available/alerts-identity-universe-main-dev-alertmanager.fastorder.com.conf
  HTTPS: /etc/apache2/sites-available/alerts-identity-universe-main-dev-alertmanager.fastorder.com-ssl.conf

Certificate:
  Path:     /etc/letsencrypt/live/alerts-identity-universe-main-dev-alertmanager.fastorder.com/
  Renewal:  certbot renew --cert-name alerts-identity-universe-main-dev-alertmanager.fastorder.com

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   Alertmanager HTTPS proxy configured
[OK]   HTTPS reverse proxies configured
[INFO] Step 12/13: Configuring firewall rules (network segmentation)...

[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING FIREWALL RULES FOR OBSERVABILITY CELL
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Cell ID: obs-identity-universe-main-dev
[INFO] Internal Network: 10.0.0.0/8

[INFO] Discovering dashboard/skeleton VM IPs...
[INFO]   Discovered skeleton IP: 142.93.238.16 (skeleton.fastorder.com)
[INFO] Authorized dashboard IPs:
[INFO]   - 10.100.60.2
[INFO]   - 142.93.238.16

[INFO] Configuring UFW firewall rules...
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1449937 ACTION=passthru ARGS=ufw --force enable
ERROR: passthru not allowed: ufw
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1449945 ACTION=passthru ARGS=ufw default deny incoming
ERROR: passthru not allowed: ufw
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1449953 ACTION=passthru ARGS=ufw default allow outgoing
ERROR: passthru not allowed: ufw
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1449961 ACTION=passthru ARGS=ufw allow 22/tcp comment SSH
ERROR: passthru not allowed: ufw
[INFO]   Allowing prometheus (port 9090) from internal network...
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1449970 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 9090 proto tcp comment Obs: prometheus from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing alertmanager (port 9093) from internal network...
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1449978 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 9093 proto tcp comment Obs: alertmanager from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing clickhouse (port 8123) from internal network...
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1449986 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 8123 proto tcp comment Obs: clickhouse from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing grafana (port 3000) from internal network...
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1449994 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 3000 proto tcp comment Obs: grafana from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing otelcol (port 4318) from internal network...
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1450004 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 4318 proto tcp comment Obs: otelcol from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing loki (port 3100) from internal network...
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1450012 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 3100 proto tcp comment Obs: loki from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing tempo (port 3200) from internal network...
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1450020 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 3200 proto tcp comment Obs: tempo from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing dashboard access from 10.100.60.2...
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1450028 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 9090 proto tcp comment Dashboard: prometheus
ERROR: passthru not allowed: ufw
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1450036 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 9093 proto tcp comment Dashboard: alertmanager
ERROR: passthru not allowed: ufw
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1450044 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 8123 proto tcp comment Dashboard: clickhouse
ERROR: passthru not allowed: ufw
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1450052 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3000 proto tcp comment Dashboard: grafana
ERROR: passthru not allowed: ufw
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1450060 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 4318 proto tcp comment Dashboard: otelcol
ERROR: passthru not allowed: ufw
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1450068 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3100 proto tcp comment Dashboard: loki
ERROR: passthru not allowed: ufw
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1450076 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3200 proto tcp comment Dashboard: tempo
ERROR: passthru not allowed: ufw
[INFO]   Allowing dashboard access from 142.93.238.16...
[2026-01-02 04:45:24 UTC] USER=www-data EUID=0 PID=1450084 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 9090 proto tcp comment Dashboard: prometheus
ERROR: passthru not allowed: ufw
[2026-01-02 04:45:25 UTC] USER=www-data EUID=0 PID=1450092 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 9093 proto tcp comment Dashboard: alertmanager
ERROR: passthru not allowed: ufw
[2026-01-02 04:45:25 UTC] USER=www-data EUID=0 PID=1450100 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 8123 proto tcp comment Dashboard: clickhouse
ERROR: passthru not allowed: ufw
[2026-01-02 04:45:25 UTC] USER=www-data EUID=0 PID=1450116 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 4318 proto tcp comment Dashboard: otelcol
ERROR: passthru not allowed: ufw
[2026-01-02 04:45:25 UTC] USER=www-data EUID=0 PID=1450124 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 3100 proto tcp comment Dashboard: loki
ERROR: passthru not allowed: ufw
[2026-01-02 04:45:25 UTC] USER=www-data EUID=0 PID=1450132 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 3200 proto tcp comment Dashboard: tempo
ERROR: passthru not allowed: ufw
[2026-01-02 04:45:25 UTC] USER=www-data EUID=0 PID=1450140 ACTION=passthru ARGS=ufw allow 443/tcp comment HTTPS obs-proxy
ERROR: passthru not allowed: ufw
[2026-01-02 04:45:25 UTC] USER=www-data EUID=0 PID=1450148 ACTION=passthru ARGS=ufw reload
ERROR: passthru not allowed: ufw
[OK]   UFW firewall rules configured

[OK]   ═══════════════════════════════════════════════════════════════
[OK]   ✅ Firewall configuration completed
[OK]   ═══════════════════════════════════════════════════════════════

[INFO] Current firewall status:
[2026-01-02 04:45:25 UTC] USER=www-data EUID=0 PID=1450158 ACTION=passthru ARGS=ufw status numbered
ERROR: passthru not allowed: ufw
[OK]   Firewall rules configured
[INFO] Step 13/13: Configuring OAuth/SSO...
[INFO] OAuth/SSO configuration script not found, skipping...

[INFO] Running validation checks...
[INFO] Validation script not found, skipping...

[INFO] Registering observability components to dashboard...
[INFO] Components to register: metrics alerts dashboards traces telemetry logstore proxy
[INFO]   Skipping metrics - registered by deploy script
[INFO]   Skipping alerts - registered by deploy script
[INFO]   Skipping dashboards - registered by deploy script
[INFO]   Skipping traces - registered by deploy script
[INFO]   Skipping telemetry - registered by deploy script
[INFO]   Skipping logstore - registered by deploy script
[INFO]   Processing component: proxy
[INFO] Registering: proxy (obs-identity-universe-main-dev-proxy)
[INFO] Detected observability component, parsing: identity-universe-main-dev-proxy
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Observability Proxy
[INFO]   Identifier:        obs-identity-universe-main-dev-proxy
[INFO]   Identifier Parent: observability-cell
[INFO]   IP:                10.100.1.180
[INFO]   Port:              443
[INFO]   FQDN:              observe-identity-universe-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-universe-main-dev (service=identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 8ba6f394-fe26-44ac-bdcc-e7c704474340
[SUCCESS] Environment UUID: b6092921-3a75-44ac-9080-96a7ca43bec0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✓ Registered: proxy
[INFO] Registering short DNS aliases...
[OK]   ✓ Observability components registration completed

[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] Verifying all observability services are running...
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[OK]     ✓ prometheus-obs-identity-universe-main-dev.service is running
[OK]     ✓ tempo-obs-identity-universe-main-dev.service is running
[OK]     ✓ clickhouse-server-obs-identity-universe-main-dev.service is running
[OK]   ✓ All observability services verified running


═══════════════════════════════════════════════════════════════════════════════
[OK]   ✅ OBSERVABILITY CELL PROVISIONED: obs-identity-universe-main-dev
═══════════════════════════════════════════════════════════════════════════════

[INFO] DNS Entries:
  metrics-identity-universe-main-dev-prometheus.fastorder.com (10.100.1.193)
  alerts-identity-universe-main-dev-alertmanager.fastorder.com (10.100.1.197)
  dashboards-identity-universe-main-dev-grafana.fastorder.com (10.100.1.194)
  traces-identity-universe-main-dev-tempo.fastorder.com (10.100.1.196)
  telemetry-identity-universe-main-dev-opentelemetry.fastorder.com (10.100.1.198)
  logstore-identity-universe-main-dev-clickhouse.fastorder.com (10.100.1.195)
  observe-identity-universe-main-dev.fastorder.com (10.100.1.180)

[INFO] Secrets Path: fastorder/observability/identity/universe/dev/*

[INFO] Access (Purpose-Oriented URLs):
  Dashboards: https://dashboards-identity-universe-main-dev-grafana.fastorder.com (SSO enabled)
  Metrics: https://metrics-identity-universe-main-dev-prometheus.fastorder.com (internal only)
  Alerts: https://alerts-identity-universe-main-dev-alertmanager.fastorder.com
  Log Storage: https://logstore-identity-universe-main-dev-clickhouse.fastorder.com

[INFO] Backend Implementation (Internal - Not Exposed to Clients):
  Telemetry: otlp
  Metrics: prometheus
  Traces: tempo
  Dashboards: grafana
  Alerting: alertmanager
  Log Storage: clickhouse

[INFO] For applications in identity-universe-main-dev:
  - Metrics: Push to telemetry-identity-universe-main-dev-opentelemetry.fastorder.com:4318 (OTLP/HTTP)
  - Logs: Push to telemetry-identity-universe-main-dev-opentelemetry.fastorder.com:4318 (OTLP/HTTP)
  - Traces: Push to telemetry-identity-universe-main-dev-opentelemetry.fastorder.com:4317 (OTLP/gRPC)
  - Query Metrics: https://metrics-identity-universe-main-dev-prometheus.fastorder.com
  - Query Logs: https://logstore-identity-universe-main-dev-clickhouse.fastorder.com
  - Query Traces: https://traces-identity-universe-main-dev-tempo.fastorder.com

[INFO] Runbook: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/RUNBOOK.md
═══════════════════════════════════════════════════════════════════════════════

[INFO] Using search engine from SEARCH_ENGINE environment variable: elasticsearch
[INFO] Cleaning up any existing locks...

Starting search engine: elasticsearch
═══════════════════════════════════════════════

════════════════════════════════════════════════════════════════
           Elasticsearch Deployment Runner                        
════════════════════════════════════════════════════════════════

[INFO] Cleaning up any existing locks (without triggering package configurations)...
[WARNING] Lock cleanup skipped (wrapper not available or insufficient permissions)

🚀 Auto mode enabled - running automatic installation


Starting Automatic Installation...
═══════════════════════════════════════════════
Will execute all deployment tasks in sequence:

  [1] Install Elasticsearch Http (01-install-elasticsearch-http)
  [2] Make Https (02-make-https)
  [3] Create Index Llm (03-create-index-llm)
  [4] Monitoring Setup (10-monitoring-setup)

═══════════════════════════════════════════════
🚀 Auto mode - proceeding automatically...

Running automatic installation...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 1: Executing Install Elasticsearch Http
Folder: 01-install-elasticsearch-http
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

=== Elasticsearch HTTP Setup ===
Install and configure Elasticsearch with HTTP access
Architecture: Per-node VM IPs with default port (9200)

[INFO] Using web-provided environment: identity-universe-main-dev
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
Environment: 
Nodes: 1
Port: 9200 (default Elasticsearch port)
Coordinator endpoint: http://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

Checking if Elasticsearch is already installed for environment: ...
Validating Elasticsearch installation...
./run.sh: line 132: /var/www/html/skeleton.dev.fastorder.com/fixing/scripts/lib/elasticsearch_validator.sh: No such file or directory
⚠️  Elasticsearch installation issues detected. Attempting automatic repair...
./run.sh: line 134: /var/www/html/skeleton.dev.fastorder.com/fixing/scripts/lib/elasticsearch_validator.sh: No such file or directory
Executing: steps/01-setup-directories.sh
+ 01-setup-directories.sh:4:main: echo '=== Step 1: Creating directory structure ==='
=== Step 1: Creating directory structure ===
+++ 01-setup-directories.sh:4:main: dirname steps/01-setup-directories.sh
++ 01-setup-directories.sh:4:main: cd steps
++ 01-setup-directories.sh:4:main: pwd
+ 01-setup-directories.sh:4:main: SCRIPT_DIR=/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps
+ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/lib/provisioning-init.sh
++ 01-setup-directories.sh:4:main: RED='\033[0;31m'
++ 01-setup-directories.sh:4:main: GREEN='\033[0;32m'
++ 01-setup-directories.sh:4:main: YELLOW='\033[1;33m'
++ 01-setup-directories.sh:4:main: BLUE='\033[0;34m'
++ 01-setup-directories.sh:4:main: NC='\033[0m'
++ 01-setup-directories.sh:4:main: export TERM=dumb
++ 01-setup-directories.sh:4:main: TERM=dumb
++ 01-setup-directories.sh:4:main: export DEBIAN_FRONTEND=noninteractive
++ 01-setup-directories.sh:4:main: DEBIAN_FRONTEND=noninteractive
++ 01-setup-directories.sh:4:main: export NEEDRESTART_MODE=a
++ 01-setup-directories.sh:4:main: NEEDRESTART_MODE=a
++ 01-setup-directories.sh:4:main: export NEEDRESTART_SUSPEND=1
++ 01-setup-directories.sh:4:main: NEEDRESTART_SUSPEND=1
++ 01-setup-directories.sh:4:main: export DEBIAN_PRIORITY=critical
++ 01-setup-directories.sh:4:main: DEBIAN_PRIORITY=critical
++ 01-setup-directories.sh:4:main: export UCF_FORCE_CONFFOLD=1
++ 01-setup-directories.sh:4:main: UCF_FORCE_CONFFOLD=1
++ 01-setup-directories.sh:4:main: export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
++ 01-setup-directories.sh:4:main: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
++ 01-setup-directories.sh:4:main: [[ -n '' ]]
++ 01-setup-directories.sh:4:main: [[ -n /opt/fastorder/bash/scripts/env_app_setup/state ]]
++ 01-setup-directories.sh:4:main: [[ -d /opt/fastorder/bash/scripts/env_app_setup/state ]]
++ 01-setup-directories.sh:4:main: STATE_DIR=/opt/fastorder/bash/scripts/env_app_setup/state
++ 01-setup-directories.sh:4:main: export STATE_DIR
++ 01-setup-directories.sh:4:main: [[ -f /opt/fastorder/bash/scripts/env_app_setup/setup/setup.json ]]
++ 01-setup-directories.sh:4:main: SETUP_JSON=/opt/fastorder/bash/scripts/env_app_setup/setup/setup.json
++ 01-setup-directories.sh:4:main: FO_WRAPPER=/usr/local/bin/fastorder-provisioning-wrapper.sh
++ 01-setup-directories.sh:4:main: HTTP_PORT_BASE=9200
++ 01-setup-directories.sh:4:main: TRANSPORT_PORT_BASE=9300
++ 01-setup-directories.sh:4:main: PG_PORT_BASE=5432
++ 01-setup-directories.sh:4:main: APP_IP_SUBNETS=(['observability']='10.100.5' ['obs']='10.100.5' ['prometheus']='10.100.5' ['grafana']='10.100.5' ['loki']='10.100.5' ['tempo']='10.100.5' ['postgresql']='10.100.10' ['postgres']='10.100.10' ['pg']='10.100.10' ['elasticsearch']='10.100.20' ['es']='10.100.20' ['kafka']='10.100.30' ['redis']='10.100.40' ['mongodb']='10.100.50' ['mongo']='10.100.50' ['iam']='10.100.60' ['keycloak']='10.100.60' ['general']='10.100.1')
++ 01-setup-directories.sh:4:main: declare -A APP_IP_SUBNETS
++ 01-setup-directories.sh:4:main: APP_IP_RESERVED_START=(['observability']='2' ['postgresql']='2' ['elasticsearch']='2' ['kafka']='2' ['redis']='2' ['mongodb']='2' ['iam']='2' ['general']='50')
++ 01-setup-directories.sh:4:main: declare -A APP_IP_RESERVED_START
++ 01-setup-directories.sh:4:main: APP_IP_RESERVED_END=(['observability']='49' ['postgresql']='254' ['elasticsearch']='254' ['kafka']='254' ['redis']='254' ['mongodb']='254' ['iam']='254' ['general']='250')
++ 01-setup-directories.sh:4:main: declare -A APP_IP_RESERVED_END
+++ 01-setup-directories.sh:4:main: dirname /opt/fastorder/bash/scripts/env_app_setup/lib/provisioning-init.sh
++ 01-setup-directories.sh:4:main: _CONFIG_MGMT_LIB=/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
++ 01-setup-directories.sh:4:main: [[ -f /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh ]]
++ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
+++ 01-setup-directories.sh:4:main: set -Eeuo pipefail
+++ 01-setup-directories.sh:4:main: : /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
+++ 01-setup-directories.sh:4:main: STATE_DIR=/opt/fastorder/bash/scripts/env_app_setup/state
++ 01-setup-directories.sh:4:main: [[ /opt/fastorder/bash/scripts/env_app_setup/lib/provisioning-init.sh == \s\t\e\p\s\/\0\1\-\s\e\t\u\p\-\d\i\r\e\c\t\o\r\i\e\s\.\s\h ]]
++ 01-setup-directories.sh:4:main: set +e
++ 01-setup-directories.sh:4:main: set +u
++ 01-setup-directories.sh:4:main: set +o pipefail
++ 01-setup-directories.sh:4:main: set +E
+ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh
++ 01-setup-directories.sh:4:main: [[ /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh == \s\t\e\p\s\/\0\1\-\s\e\t\u\p\-\d\i\r\e\c\t\o\r\i\e\s\.\s\h ]]
+ 01-setup-directories.sh:4:main: init_environment
+ 01-setup-directories.sh:4:main: require_bin jq
+ 01-setup-directories.sh:4:main: for b in "$@"
+ 01-setup-directories.sh:4:main: command -v jq
+ 01-setup-directories.sh:4:main: local app_type=general
+ 01-setup-directories.sh:4:main: ENV_ID=identity-universe-main-dev
+ 01-setup-directories.sh:4:main: [[ -z identity-universe-main-dev ]]
+ 01-setup-directories.sh:4:main: [[ -z identity-universe-main-dev ]]
+ 01-setup-directories.sh:4:main: ENV_ID=identity-universe-main-dev
+ 01-setup-directories.sh:4:main: [[ -z identity-universe-main-dev ]]
+ 01-setup-directories.sh:4:main: [[ -z identity-universe-main-dev ]]
+ 01-setup-directories.sh:4:main: [[ -z identity-universe-main-dev ]]
++ 01-setup-directories.sh:4:main: env_dir_for identity-universe-main-dev
++ 01-setup-directories.sh:4:main: echo /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev
+ 01-setup-directories.sh:4:main: ENV_DIR=/opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev
++ 01-setup-directories.sh:4:main: topo_path_for identity-universe-main-dev
++ 01-setup-directories.sh:4:main: echo /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: TOPOLOGY_JSON=/opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: [[ ! -f /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json ]]
+ 01-setup-directories.sh:4:main: validate_topology_json /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: local topo=/opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: [[ -r /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json ]]
+ 01-setup-directories.sh:4:main: jq -e '
    .schema_version == 1
    and (.general.id        | type=="string")
    and (.general.shared_ip | type=="string")
    and (.general.service   | type=="string")
    and (.general.zone    | type=="string")
    and (.general.env       | type=="string")
  ' /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
++ 01-setup-directories.sh:4:main: jq -r .general.service /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: SERVICE=identity
++ 01-setup-directories.sh:4:main: jq -r .general.zone /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: zone=universe
++ 01-setup-directories.sh:4:main: jq -r .general.branch /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: BRANCH=main
++ 01-setup-directories.sh:4:main: jq -r .general.env /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: ENV=dev
++ 01-setup-directories.sh:4:main: jq -r '.general.es_nodes_num // 3' /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: ES_NODES_NUM=1
++ 01-setup-directories.sh:4:main: jq -r '.general.pg_workers_num // 3' /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: PG_WORKERS_NUM=1
++ 01-setup-directories.sh:4:main: jq -r '.general.pg_WORKERS_STANDBY_NUM // 3' /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: PG_WORKERS_STANDBY_NUM=3
++ 01-setup-directories.sh:4:main: jq -r '.general.pg_citus_enabled // "yes"' /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: PG_CITUS_ENABLED=yes
+ 01-setup-directories.sh:4:main: [[ general != \g\e\n\e\r\a\l ]]
++ 01-setup-directories.sh:4:main: jq -r .general.shared_ip /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: VM_IP=142.93.238.16
+ 01-setup-directories.sh:4:main: [[ general != \g\e\n\e\r\a\l ]]
++ 01-setup-directories.sh:4:main: jq -r '.general.shared_iface // empty' /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: IFACE=eth0:16
+ 01-setup-directories.sh:4:main: local FINAL_VM_IP=142.93.238.16
+ 01-setup-directories.sh:4:main: set -a
+ 01-setup-directories.sh:4:main: [[ -r /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/generated/general.env ]]
+ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/generated/general.env
++ 01-setup-directories.sh:4:main: ENV_ID=identity-universe-main-dev
++ 01-setup-directories.sh:4:main: SERVICE=identity
++ 01-setup-directories.sh:4:main: zone=universe
++ 01-setup-directories.sh:4:main: BRANCH=main
++ 01-setup-directories.sh:4:main: ENV=dev
++ 01-setup-directories.sh:4:main: VM_IP=142.93.238.16
++ 01-setup-directories.sh:4:main: IFACE=eth0:16
++ 01-setup-directories.sh:4:main: ROOT_DIR=/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
++ 01-setup-directories.sh:4:main: ENV_DIR=/opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev
++ 01-setup-directories.sh:4:main: TOPOLOGY_JSON=/opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
++ 01-setup-directories.sh:4:main: LOG_LEVEL=info
++ 01-setup-directories.sh:4:main: DEBUG_MODE=false
+ 01-setup-directories.sh:4:main: set +a
+ 01-setup-directories.sh:4:main: VM_IP=142.93.238.16
+ 01-setup-directories.sh:4:main: export ENV_ID SERVICE zone BRANCH ENV VM_IP IFACE ENV_DIR TOPOLOGY_JSON
+ 01-setup-directories.sh:4:main: export ES_NODES_NUM PG_WORKERS_NUM PG_WORKERS_STANDBY_NUM PG_CITUS_ENABLED
+ 01-setup-directories.sh:4:main: [[ general != \g\e\n\e\r\a\l ]]
+ 01-setup-directories.sh:4:main: info 'Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)'
+ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)'
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
+ 01-setup-directories.sh:4:main: return 0
+ 01-setup-directories.sh:4:main: SERVICE=identity
+ 01-setup-directories.sh:4:main: ZONE=universe
+ 01-setup-directories.sh:4:main: BRANCH=main
+ 01-setup-directories.sh:4:main: ENV=dev
++ 01-setup-directories.sh:4:main: env_id
++ 01-setup-directories.sh:4:main: '[' identity = auth ']'
++ 01-setup-directories.sh:4:main: '[' identity = item ']'
++ 01-setup-directories.sh:4:main: echo identity-universe-main-dev
+ 01-setup-directories.sh:4:main: ENV_ID=identity-universe-main-dev
+ 01-setup-directories.sh:4:main: env=identity-universe-main-dev
+ 01-setup-directories.sh:4:main: nodes=1
+ 01-setup-directories.sh:4:main: [[ 1 =~ ^[1-9][0-9]*$ ]]
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /etc/elasticsearch
[2026-01-02 04:45:44 UTC] USER=www-data EUID=0 PID=1450520 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /var/lib/elasticsearch
[2026-01-02 04:45:44 UTC] USER=www-data EUID=0 PID=1450529 ACTION=fsop ARGS=mkdir -p /var/lib/elasticsearch
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /var/log/elasticsearch
[2026-01-02 04:45:44 UTC] USER=www-data EUID=0 PID=1450538 ACTION=fsop ARGS=mkdir -p /var/log/elasticsearch
+ 01-setup-directories.sh:4:main: APP_NAME=search
+ 01-setup-directories.sh:4:main: TOPOLOGY_FILE=/opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: command -v jq
+ 01-setup-directories.sh:4:main: [[ -f /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json ]]
++ 01-setup-directories.sh:4:main: jq -r --arg app search '.applications[$app].vm_ip // empty' /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: COORD_IP=10.100.1.178
+ 01-setup-directories.sh:4:main: [[ -z 10.100.1.178 ]]
+ 01-setup-directories.sh:4:main: [[ 10.100.1.178 == \n\u\l\l ]]
++ 01-setup-directories.sh:4:main: get_application_domain search
++ 01-setup-directories.sh:4:main: local app_type=search
++ 01-setup-directories.sh:4:main: [[ search == \g\e\n\e\r\a\l ]]
++ 01-setup-directories.sh:4:main: jq -r --arg app search '.applications[$app].domain // empty' /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
+ 01-setup-directories.sh:4:main: COORD_DOMAIN=search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com
+ 01-setup-directories.sh:4:main: info 'Coordinator exists: search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.178)'
+ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Coordinator exists: search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.178)'
[INFO] Coordinator exists: search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.178)
+ 01-setup-directories.sh:4:main: (( i=1 ))
+ 01-setup-directories.sh:4:main: (( i<=nodes ))
++ 01-setup-directories.sh:4:main: printf %02d 1
+ 01-setup-directories.sh:4:main: node_num=01
+ 01-setup-directories.sh:4:main: IDENTIFIER=node-01
+ 01-setup-directories.sh:4:main: APP_NAME=search-node-01
+ 01-setup-directories.sh:4:main: read -r NODE_IP NODE_DOMAIN
++ 01-setup-directories.sh:4:main: setup_directories_per_node node-01 search-node-01
++ 01-setup-directories.sh:4:main: local IDENTIFIER=node-01
++ 01-setup-directories.sh:4:main: local APP_NAME=search-node-01
++ 01-setup-directories.sh:4:main: local env
+++ 01-setup-directories.sh:4:main: env_id
+++ 01-setup-directories.sh:4:main: '[' identity = auth ']'
+++ 01-setup-directories.sh:4:main: '[' identity = item ']'
+++ 01-setup-directories.sh:4:main: echo identity-universe-main-dev
++ 01-setup-directories.sh:4:main: env=identity-universe-main-dev
++ 01-setup-directories.sh:4:main: local TOPOLOGY_FILE=/opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
++ 01-setup-directories.sh:4:main: info 'Setting up Elasticsearch node: node-01'
++ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Setting up Elasticsearch node: node-01'
++ 01-setup-directories.sh:4:main: local NODE_IP NODE_DOMAIN
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /etc/elasticsearch/identity-universe-main-dev/node-01 /etc/elasticsearch/identity-universe-main-dev-node-01
+++ 01-setup-directories.sh:4:main: jq -r --arg app search-node-01 '.applications[$app].vm_ip // empty' /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
[2026-01-02 04:45:44 UTC] USER=www-data EUID=0 PID=1450556 ACTION=fsop ARGS=ln -sfn /etc/elasticsearch/identity-universe-main-dev/node-01 /etc/elasticsearch/identity-universe-main-dev-node-01
+ 01-setup-directories.sh:4:main: [[ 1 -eq 1 ]]
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /etc/elasticsearch/identity-universe-main-dev/node-01 /etc/elasticsearch/current
++ 01-setup-directories.sh:4:main: NODE_IP=10.100.1.179
++ 01-setup-directories.sh:4:main: [[ -z 10.100.1.179 ]]
++ 01-setup-directories.sh:4:main: [[ 10.100.1.179 == \n\u\l\l ]]
+++ 01-setup-directories.sh:4:main: get_application_domain search-node-01
+++ 01-setup-directories.sh:4:main: local app_type=search-node-01
+++ 01-setup-directories.sh:4:main: [[ search-node-01 == \g\e\n\e\r\a\l ]]
+++ 01-setup-directories.sh:4:main: jq -r --arg app search-node-01 '.applications[$app].domain // empty' /opt/fastorder/bash/scripts/env_app_setup/state/identity-universe-main-dev/topology.json
[2026-01-02 04:45:44 UTC] USER=www-data EUID=0 PID=1450567 ACTION=fsop ARGS=ln -sfn /etc/elasticsearch/identity-universe-main-dev/node-01 /etc/elasticsearch/current
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /var/lib/elasticsearch/identity-universe-main-dev/node-01 /var/lib/elasticsearch/current
++ 01-setup-directories.sh:4:main: NODE_DOMAIN=search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com
++ 01-setup-directories.sh:4:main: info 'Using existing node-01: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.179)'
++ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Using existing node-01: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.179)'
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh: line 13: printf: write error: Broken pipe
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /etc/elasticsearch/identity-universe-main-dev/node-01/certs
[2026-01-02 04:45:44 UTC] USER=www-data EUID=0 PID=1450577 ACTION=fsop ARGS=ln -sfn /var/lib/elasticsearch/identity-universe-main-dev/node-01 /var/lib/elasticsearch/current
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /var/log/elasticsearch/identity-universe-main-dev/node-01 /var/log/elasticsearch/current
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /var/lib/elasticsearch/identity-universe-main-dev/node-01/tmp
[2026-01-02 04:45:44 UTC] USER=www-data EUID=0 PID=1450595 ACTION=fsop ARGS=ln -sfn /var/log/elasticsearch/identity-universe-main-dev/node-01 /var/log/elasticsearch/current
+ 01-setup-directories.sh:4:main: (( i++ ))
+ 01-setup-directories.sh:4:main: (( i<=nodes ))
+ 01-setup-directories.sh:4:main: success 'Directory structure created for '\''identity-universe-main-dev'\'' with 1 node(s).'
+ 01-setup-directories.sh:4:main: printf '[ OK ] %s\n' 'Directory structure created for '\''identity-universe-main-dev'\'' with 1 node(s).'
[ OK ] Directory structure created for 'identity-universe-main-dev' with 1 node(s).
Executing: steps/02-install-dependencies.sh
=== Step 2: Installing/Validating Elasticsearch (latest) ===
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /var/log/elasticsearch/identity-universe-main-dev/node-01
++ 01-setup-directories.sh:4:main: id -u elasticsearch
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-universe-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/identity-universe-main-dev/node-01
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
[INFO] Cleaning dpkg/apt locks...
[2026-01-02 04:45:44 UTC] USER=www-data EUID=0 PID=1450657 ACTION=cleanup-dpkg-locks ARGS=
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chown -R elasticsearch:elasticsearch /var/log/elasticsearch/identity-universe-main-dev/node-01
steps/02-install-dependencies.sh: line 16: 1450655 Killed                  command sudo -n "$WRAP" cleanup-dpkg-locks
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chmod 0750 /etc/elasticsearch/identity-universe-main-dev/node-01
[2026-01-02 04:45:44 UTC] USER=www-data EUID=0 PID=1450675 ACTION=fsop ARGS=mkdir -p /etc/apt/keyrings
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chmod 0750 /var/lib/elasticsearch/identity-universe-main-dev/node-01
[2026-01-02 04:45:44 UTC] USER=www-data EUID=0 PID=1450694 ACTION=fsop ARGS=chmod 0755 /etc/apt/keyrings
[INFO] apt-get update…
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chmod 0750 /var/log/elasticsearch/identity-universe-main-dev/node-01
[2026-01-02 04:45:44 UTC] USER=www-data EUID=0 PID=1450713 ACTION=pkg ARGS=update
++ 01-setup-directories.sh:4:main: info 'Created dirs for identity-universe-main-dev/node-01 @ 10.100.1.179'
++ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Created dirs for identity-universe-main-dev/node-01 @ 10.100.1.179'
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh: line 13: printf: write error: Broken pipe
++ 01-setup-directories.sh:4:main: printf '%s\n' 10.100.1.179
/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh: line 58: printf: write error: Broken pipe
++ 01-setup-directories.sh:4:main: printf '%s\n' search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com
/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh: line 59: printf: write error: Broken pipe
Hit:1 https://artifacts.elastic.co/packages/8.x/apt stable InRelease
Hit:2 http://apt.postgresql.org/pub/repos/apt jammy-pgdg InRelease
Hit:3 https://packages.confluent.io/deb/7.6 stable InRelease
Hit:4 https://apt.grafana.com stable InRelease
Hit:5 https://packages.microsoft.com/repos/azure-cli jammy InRelease
Hit:6 https://packages.clickhouse.com/deb stable InRelease
Hit:7 https://deb.nodesource.com/node_22.x nodistro InRelease
Hit:8 https://mirrors.edge.kernel.org/ubuntu jammy InRelease
Hit:9 https://mirrors.edge.kernel.org/ubuntu jammy-backports InRelease
Hit:10 https://mirrors.edge.kernel.org/ubuntu jammy-security InRelease
Hit:11 https://mirrors.edge.kernel.org/ubuntu jammy-updates InRelease
Hit:12 https://ppa.launchpadcontent.net/ondrej/php/ubuntu jammy InRelease
Hit:13 https://repos.citusdata.com/community/ubuntu jammy InRelease
Reading package lists...
[INFO] Installed version : 8.19.9
[INFO] Candidate version : 8.19.9
✅ Elasticsearch already at latest (or only) available version.
✅ Elasticsearch installation validated.
🎉 Dependencies installed and up-to-date.
Executing: steps/03-create-env-configs.sh
=== Step 3: Creating environment configurations (master + nodes, TLS, units) ===
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
Configuring env: identity-universe-main-dev (nodes: 1, http: 9200, transport: 9300)
Using heap size: 1024m per node
[2026-01-02 04:46:05 UTC] USER=www-data EUID=0 PID=1451808 ACTION=fsop ARGS=chown root:root /etc/default/elasticsearch
[2026-01-02 04:46:05 UTC] USER=www-data EUID=0 PID=1451817 ACTION=fsop ARGS=chmod 0644 /etc/default/elasticsearch
[2026-01-02 04:46:05 UTC] USER=www-data EUID=0 PID=1451836 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/jvm.options
[2026-01-02 04:46:05 UTC] USER=www-data EUID=0 PID=1451845 ACTION=fsop ARGS=chmod 0644 /etc/elasticsearch/jvm.options
[2026-01-02 04:46:05 UTC] USER=www-data EUID=0 PID=1451863 ACTION=fsop ARGS=mkdir -p /etc/systemd/system.conf.d /etc/systemd/user.conf.d
[2026-01-02 04:46:05 UTC] USER=www-data EUID=0 PID=1451890 ACTION=passthru ARGS=systemctl daemon-reload
Current max_map_count: 262144
Current swappiness:   1
[2026-01-02 04:46:06 UTC] USER=www-data EUID=0 PID=1451968 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/log4j2.properties
[2026-01-02 04:46:06 UTC] USER=www-data EUID=0 PID=1451977 ACTION=fsop ARGS=chmod 0644 /etc/elasticsearch/log4j2.properties
[2026-01-02 04:46:06 UTC] USER=www-data EUID=0 PID=1451986 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-universe-main-dev/template
[2026-01-02 04:46:06 UTC] USER=www-data EUID=0 PID=1451995 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/identity-universe-main-dev /etc/elasticsearch/identity-universe-main-dev/template
[2026-01-02 04:46:06 UTC] USER=www-data EUID=0 PID=1452004 ACTION=fsop ARGS=chmod 0755 /etc/elasticsearch/identity-universe-main-dev
[2026-01-02 04:46:06 UTC] USER=www-data EUID=0 PID=1452013 ACTION=fsop ARGS=cp /etc/elasticsearch/jvm.options /etc/elasticsearch/identity-universe-main-dev/template/jvm.options
[INFO] 🌐 Registering general environment domain: identity-universe-main-dev.fastorder.com
[INFO]   Allocated VM IP: 10.100.1.50 for general environment
[INFO]   Configuring VM IP 10.100.1.50 on network interface...
[WARNING]   VM IP may already be configured or need manual setup
[WARNING]   Warning: VM IP 10.100.1.50 not found on network interfaces
[ OK ] ✅ Registered general domain identity-universe-main-dev.fastorder.com -> 10.100.1.50
[ OK ] ✅ DNS resolution verified for identity-universe-main-dev.fastorder.com
[INFO] → Configuring identity-universe-main-dev-node-01 (10.100.1.179) roles=[ master, data, data_hot, data_content, ingest ]
[2026-01-02 04:46:06 UTC] USER=www-data EUID=0 PID=1452099 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-universe-main-dev/node-01/certs /var/lib/elasticsearch/identity-universe-main-dev/node-01/tmp /var/log/elasticsearch/identity-universe-main-dev/node-01
[2026-01-02 04:46:07 UTC] USER=www-data EUID=0 PID=1452108 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-universe-main-dev/node-01
[2026-01-02 04:46:07 UTC] USER=www-data EUID=0 PID=1452117 ACTION=fsop ARGS=chmod 0750 /etc/elasticsearch/identity-universe-main-dev/node-01 /var/lib/elasticsearch/identity-universe-main-dev/node-01 /var/log/elasticsearch/identity-universe-main-dev/node-01
[2026-01-02 04:46:07 UTC] USER=www-data EUID=0 PID=1452126 ACTION=fsop ARGS=cp /etc/elasticsearch/identity-universe-main-dev/template/jvm.options /etc/elasticsearch/identity-universe-main-dev/node-01/jvm.options
[2026-01-02 04:46:07 UTC] USER=www-data EUID=0 PID=1452135 ACTION=fsop ARGS=sed -i s/^-Xms.*/-Xms1024m/ /etc/elasticsearch/identity-universe-main-dev/node-01/jvm.options
[2026-01-02 04:46:07 UTC] USER=www-data EUID=0 PID=1452144 ACTION=fsop ARGS=sed -i s/^-Xmx.*/-Xmx1024m/ /etc/elasticsearch/identity-universe-main-dev/node-01/jvm.options
[2026-01-02 04:46:07 UTC] USER=www-data EUID=0 PID=1452163 ACTION=fsop ARGS=cp /etc/elasticsearch/log4j2.properties /etc/elasticsearch/identity-universe-main-dev/node-01/log4j2.properties
[2026-01-02 04:46:07 UTC] USER=www-data EUID=0 PID=1452200 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-universe-main-dev/node-01
[2026-01-02 04:46:07 UTC] USER=www-data EUID=0 PID=1452209 ACTION=fsop ARGS=chmod 0644 /etc/elasticsearch/identity-universe-main-dev/node-01/elasticsearch.yml
[2026-01-02 04:46:07 UTC] USER=www-data EUID=0 PID=1452227 ACTION=fsop ARGS=chmod 0644 /etc/default/elasticsearch-identity-universe-main-dev-node-01
[2026-01-02 04:46:07 UTC] USER=www-data EUID=0 PID=1452250 ACTION=passthru ARGS=ip addr add 10.100.1.179/32 dev eth0 label eth0:179
[2026-01-02 04:46:07 UTC] USER=www-data EUID=0 PID=1452259 ACTION=fsop ARGS=sed -i /[[:space:]]search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com\([[:space:]]\|$\)/d /etc/hosts
[INFO]   → Also added short domain: search-identity-universe-main-dev.fastorder.com
[INFO] ✔ Created configuration for identity-universe-main-dev/node-01 (roles=single-node)
[2026-01-02 04:46:07 UTC] USER=www-data EUID=0 PID=1452289 ACTION=fsop ARGS=sed -i /[[:space:]]search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com\([[:space:]]\|$\)/d /etc/hosts
[INFO] ✔ Registered master domain search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com -> 10.100.1.179 (points to node-01)
[INFO] Cleaning up legacy non-templated elasticsearch-*.service units (if any)...
[INFO] No legacy units found.
[INFO] Creating base template: elasticsearch@.service
[2026-01-02 04:46:07 UTC] USER=www-data EUID=0 PID=1452335 ACTION=passthru ARGS=systemctl daemon-reload
[ OK ] Base template created & daemon reloaded
[ OK ] Created unit: elasticsearch@identity-universe-main-dev-node-01.service
[2026-01-02 04:46:08 UTC] USER=www-data EUID=0 PID=1452408 ACTION=passthru ARGS=systemctl daemon-reload

[ OK ] Environment configurations (master + nodes with TLS) created successfully!
[INFO] Environment: identity-universe-main-dev
[INFO] Nodes: 1
[INFO] HTTP Port: 9200
[INFO] Transport Port: 9300
[INFO] Heap Size: 1024m per node
[INFO] Master: search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.178)
[INFO]   node-01: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.179)
[INFO] Systemd units prepared (not started). Start sequence runs in Step 7.
Executing: steps/04-start-clusters.sh
=== Step 7: Starting Elasticsearch clusters (with waits) ===
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
Starting Elasticsearch cluster for environment: identity-universe-main-dev (1 nodes)
[INFO] === Ensuring VM IP services are started ===
[WARNING] VM IP service vm-ip-10-100-1-179.service not found - IP might not persist
[INFO] Manually configuring IP: 10.100.1.179
[2026-01-02 04:46:12 UTC] USER=www-data EUID=0 PID=1452486 ACTION=configure-network-interface ARGS=lo:search01 10.100.1.179
✓ lo:search01 <- 10.100.1.179
[INFO] Cleaning up any existing Elasticsearch processes and lock files...
[2026-01-02 04:46:12 UTC] USER=www-data EUID=0 PID=1452496 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@identity-universe-main-dev-node-01.service
[INFO] Stopping Elasticsearch services for environment: identity-universe-main-dev ...
[INFO] No active Elasticsearch services found for environment: identity-universe-main-dev
[INFO] Removing lock files from: /var/lib/elasticsearch/identity-universe-main-dev/node-01
[2026-01-02 04:46:12 UTC] USER=www-data EUID=0 PID=1452508 ACTION=fsop ARGS=find /var/lib/elasticsearch/identity-universe-main-dev/node-01 -name *.lock -delete
[2026-01-02 04:46:12 UTC] USER=www-data EUID=0 PID=1452517 ACTION=fsop ARGS=find /var/lib/elasticsearch/identity-universe-main-dev/node-01 -name node.lock -delete
[2026-01-02 04:46:12 UTC] USER=www-data EUID=0 PID=1452526 ACTION=fsop ARGS=find /var/lib/elasticsearch/identity-universe-main-dev/node-01 -name _state -type d -exec rm -rf {} +
[2026-01-02 04:46:12 UTC] USER=www-data EUID=0 PID=1452536 ACTION=fsop ARGS=find /tmp -name *elasticsearch*identity-universe-main-dev-node-01* -delete
[ OK ] Cleanup completed for environment: identity-universe-main-dev
[INFO] Checking for port conflicts before starting Elasticsearch...
[INFO] Checking for port conflicts on 10.100.1.178:9200 and 10.100.1.178:9300...
[ OK ] ✓ Ports 9200 and 9300 are available on 10.100.1.178
[INFO] Ensuring correct ownership of Elasticsearch directories...
[2026-01-02 04:46:14 UTC] USER=www-data EUID=0 PID=1452580 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch
[2026-01-02 04:46:15 UTC] USER=www-data EUID=0 PID=1452589 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /var/lib/elasticsearch
[2026-01-02 04:46:16 UTC] USER=www-data EUID=0 PID=1452598 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /var/log/elasticsearch
[ OK ] Directory ownership fixed
[INFO] === Starting Elasticsearch Nodes ===
[INFO] Starting 1 node(s) for cluster
▶ Starting elasticsearch@identity-universe-main-dev-node-01.service (search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200)
[2026-01-02 04:46:16 UTC] USER=www-data EUID=0 PID=1452611 ACTION=passthru ARGS=systemctl is-enabled --quiet elasticsearch@identity-universe-main-dev-node-01.service
[2026-01-02 04:46:17 UTC] USER=www-data EUID=0 PID=1452665 ACTION=passthru ARGS=systemctl start elasticsearch@identity-universe-main-dev-node-01.service
⏳ Waiting for TCP search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200 to be accessible (timeout 360s)...
✅ Port 9200 is accessible on search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com.
⏳ Waiting for ES HTTP readiness on http://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200 (timeout 300s)...
[ OK ] ES HTTP ready on search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
[ OK ] elasticsearch@identity-universe-main-dev-node-01.service is up and answering HTTP on search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Node 1 started successfully
[INFO] Cluster with 1 node(s) started successfully
⏳ Waiting for the cluster to elect master and settle...
⏳ Waiting for cluster health=green via search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200 (timeout 300s)...
[ OK ] Cluster is GREEN (nodes="number_of_nodes") on search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
[ OK ] Cluster identity-universe-main-dev is healthy and green!

[INFO] === Final Status Check ===
[2026-01-02 04:46:57 UTC] USER=www-data EUID=0 PID=1453272 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@identity-universe-main-dev-node-01.service
[ OK ] elasticsearch@identity-universe-main-dev-node-01.service is ACTIVE (search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200)
  └── HTTP responding on search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200 ✓
[ OK ] All 1 node(s) in environment 'identity-universe-main-dev' are running successfully!
[INFO] Node endpoints:
  - http://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200


[ OK ] Elasticsearch cluster started successfully!
[INFO] Environment: identity-universe-main-dev
[INFO] Nodes: 1
[INFO] Cluster endpoints:
  - http://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === Quick Cluster Information ===
Cluster Name: fastorder-identity-universe-main-dev
Node Name: identity-universe-main-dev-node-01
Version: 8.19.9
Architecture: 1 node(s), each on default port 9200

Cluster with 1 node(s) started successfully (each on port 9200)
Executing: steps/05-verify-setup.sh
=== Step 8: Verifying setup (with retries) ===
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
Verifying environment: identity-universe-main-dev (1 nodes, Single-node)
Main HTTP endpoint: http://search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
Testing network connectivity to search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200...
✓ Domain connection available
Testing HTTP response...
[ OK ] ✓ identity-universe-main-dev is responding on search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200

[INFO] === Cluster Health ===
{
  "cluster_name" : "fastorder-identity-universe-main-dev",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
[ OK ] Cluster status: GREEN ("number_of_nodes" nodes)

[INFO] === Cluster Nodes ===
ip           heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.100.1.179           53          98  45    6.74    5.77     3.31 dhims     *      identity-universe-main-dev-node-01

[INFO] === Single-Node Service Verification ===
Testing coordinator service (search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200)...
  ✓ Coordinator HTTP responding on search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
    Name: identity-universe-main-dev-node-01, Version: 8.19.9

[INFO] === Cluster State Summary ===
Using jq for formatted output:
jq parsing failed

[ OK ] === Verification Summary ===
[INFO] Environment: identity-universe-main-dev
[INFO] Nodes configured: 1
[INFO] Main endpoint: http://search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
[INFO] Service endpoint: http://search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200

[INFO] === Final Connectivity Test ===
  ✓ Coordinator: search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200

[ OK ] Single-node cluster is responding successfully!
[ OK ] Elasticsearch cluster 'identity-universe-main-dev' verification completed successfully!
Executing: steps/06-confirm-working.sh
=== Step 9: Comprehensive Cluster Verification (gated) ===
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
========================================
🔍 Verifying Environment: identity-universe-main-dev (1 nodes)
========================================
Domain: search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com
Environment: identity-universe-main-dev
Nodes: 1

[INFO] Testing network connectivity...
Setup type: Single-node
Testing endpoint: search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
[ OK ] ✓ Using domain: search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com


📡 Coordinator Service (elasticsearch@identity-universe-main-dev-node-01.service)
Endpoint: search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
--------------------------------
[2026-01-02 04:46:59 UTC] USER=www-data EUID=0 PID=1453412 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@identity-universe-main-dev-node-01.service
✅ Service: ACTIVE
⏳ Waiting for TCP search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200 to be accessible (timeout 5s)...
✅ Port 9200 is accessible on search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com.
✅ Port: LISTENING on search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
✅ HTTP: RESPONDING on search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
   Node name: identity-universe-main-dev-node-01

========================================
🏥 Cluster Health Check
========================================
Cluster Name: fastorder-identity-universe-main-dev
Nodes Count: "number_of_nodes"
Status: green
[ OK ] ✅ Cluster status: GREEN (healthy)

Full cluster health:
{
  "cluster_name" : "fastorder-identity-universe-main-dev",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

========================================
📊 Final Verification Results
========================================
[ OK ] ✅ Comprehensive verification PASSED!
[ OK ] Environment 'identity-universe-main-dev' with 1 nodes is fully operational

📋 QUICK DIAGNOSTIC COMMANDS:
----------------------------------------
# Test cluster endpoints:
curl http://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

# Check cluster health:
curl http://search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200/_cluster/health?pretty

# Check nodes info:
curl http://search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200/_cat/nodes?v

# Check all Elasticsearch ports:
sudo ss -tlnp | grep java

# Check systemd service status:
sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status elasticsearch@identity-universe-main-dev-node-01.service

# View recent logs:
sudo journalctl -u elasticsearch@identity-universe-main-dev-node-01.service -f

[INFO] Environment: identity-universe-main-dev
[INFO] Nodes: 1
[INFO] Port: 9200 (default Elasticsearch port)
[INFO] Coordinator endpoint: http://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

=== Elasticsearch HTTP Setup completed successfully! ===
Environment:  (1 nodes)
Port: 9200 (default Elasticsearch port)

✅ Coordinator endpoint: http://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

Quick test commands:
  curl http://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
  curl http://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty


✓ Step 1 completed successfully!

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 2: Executing Make Https
Folder: 02-make-https
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

=== Elasticsearch HTTPS Setup ===
Configure HTTPS/SSL for Elasticsearch cluster
[INFO] Using web-provided environment: identity-universe-main-dev
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
Environment: 
Nodes: 1
Node: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.179)
Port: 9200 (default port)

Executing: steps/01-generate-ssl-certificates.sh
==================================================================
STEP 1: Generate SSL certificates for Elasticsearch transport
==================================================================
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
Domain: identity-universe-main-dev.fastorder.com
Environment: identity-universe-main-dev
Nodes: 1
Per-node VM IPs and domains:
  Node 1: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.179)
Port: 9200 (default port for all nodes)

=== Generating SSL certificates for ES transport ===
[INFO] Generating certificates for environment: identity-universe-main-dev (1 nodes)
[INFO] Configuring certificates for 1 node(s)
[INFO] Certificate storage: /etc/fastorder/elasticsearch/certs/identity-universe-main-dev
[2026-01-02 04:47:01 UTC] USER=www-data EUID=0 PID=1453505 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/temp-1453482
[2026-01-02 04:47:01 UTC] USER=www-data EUID=0 PID=1453514 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/temp-1453482
[2026-01-02 04:47:01 UTC] USER=www-data EUID=0 PID=1453534 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/temp-1453482
[2026-01-02 04:47:01 UTC] USER=www-data EUID=0 PID=1453554 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/temp-1453482/instances.yml
[INFO] Creating certificate instances configuration...
  Adding node: identity-universe-main-dev-node-01 (search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com, 10.100.1.179)

[INFO] Certificate instances configuration:
instances:
  - name: identity-universe-main-dev-node-01
    dns: [ "identity-universe-main-dev-node-01", "localhost", "search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com" ]
    ip:  [ "10.100.1.179", "127.0.0.1" ]

[INFO] Creating Certificate Authority for identity-universe-main-dev...
[2026-01-02 04:47:01 UTC] USER=www-data EUID=0 PID=1453579 ACTION=fsop ARGS=mkdir -p /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01/certs
[2026-01-02 04:47:01 UTC] USER=www-data EUID=0 PID=1453588 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/fastorder/elasticsearch/certs/identity-universe-main-dev
[2026-01-02 04:47:01 UTC] USER=www-data EUID=0 PID=1453597 ACTION=fsop ARGS=chmod -R 755 /etc/fastorder/elasticsearch/certs/identity-universe-main-dev
[2026-01-02 04:47:02 UTC] USER=www-data EUID=0 PID=1453606 ACTION=fsop ARGS=rm -f /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01/certs/identity-universe-main-dev-ca.zip
yes: standard output: Broken pipe
[ OK ] ✓ CA certificate created

[INFO] Creating node certificates for identity-universe-main-dev...
yes: standard output: Broken pipe
[ OK ] ✓ Node certificates created

[INFO] Distributing certificates...
  Configuring certificates for node 1 (identity-universe-main-dev-node-01)...
[2026-01-02 04:47:09 UTC] USER=www-data EUID=0 PID=1453742 ACTION=fsop ARGS=cp /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01/certs/ca/ca.crt /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01/ca.crt
[2026-01-02 04:47:09 UTC] USER=www-data EUID=0 PID=1453757 ACTION=fsop ARGS=cp /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01/certs/nodes/identity-universe-main-dev-node-01.crt /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01/
[2026-01-02 04:47:09 UTC] USER=www-data EUID=0 PID=1453766 ACTION=fsop ARGS=cp /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01/certs/nodes/identity-universe-main-dev-node-01.key /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01/
[2026-01-02 04:47:09 UTC] USER=www-data EUID=0 PID=1453775 ACTION=fsop ARGS=chmod 644 /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01/identity-universe-main-dev-node-01.crt
[2026-01-02 04:47:09 UTC] USER=www-data EUID=0 PID=1453784 ACTION=fsop ARGS=chmod 600 /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01/identity-universe-main-dev-node-01.key
[ OK ]   ✓ Certificates copied for identity-universe-main-dev-node-01
[2026-01-02 04:47:09 UTC] USER=www-data EUID=0 PID=1453793 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01
[2026-01-02 04:47:09 UTC] USER=www-data EUID=0 PID=1453802 ACTION=fsop ARGS=find /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01/certs -type f -name *.key -exec chmod 600 {} ;
[2026-01-02 04:47:09 UTC] USER=www-data EUID=0 PID=1453813 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01/certs
[2026-01-02 04:47:09 UTC] USER=www-data EUID=0 PID=1453822 ACTION=fsop ARGS=rm -rf /etc/elasticsearch/temp-1453482
[ OK ] ✓ Certificates ready for environment: identity-universe-main-dev

[ OK ] ✓ SSL certificate generation completed successfully!
[INFO] Environment: identity-universe-main-dev
[INFO] Nodes configured: 1
[INFO] Per-node VM IPs and domains (each with default port 9200):
  Node 1: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.179)
[INFO] Certificate directory: /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01/certs

[INFO] === Certificate Summary ===
CA Certificate: /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01/certs/ca/ca.crt
Node Certificates:
  - identity-universe-main-dev-node-01: /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01/

[INFO] === Verification Commands ===
# Verify CA certificate:
openssl x509 -in /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01/certs/ca/ca.crt -text -noout

# Verify node certificates:
openssl x509 -in /etc/fastorder/elasticsearch/certs/identity-universe-main-dev/node-01/identity-universe-main-dev-node-01.crt -text -noout

[INFO] Next: Configure transport SSL in Elasticsearch configuration files
Executing: steps/02-enable-security-transport.sh
==================================================================
STEP 2: Enable security with transport SSL
==================================================================
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
Environment: identity-universe-main-dev
Nodes: 1
Node: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.179)

[INFO] === Single-Node Setup ===
[INFO] Enabling security (xpack.security.enabled: true)
[2026-01-02 04:47:10 UTC] USER=www-data EUID=0 PID=1453861 ACTION=fsop ARGS=sed -i /^xpack.security.enabled:/d /etc/elasticsearch/identity-universe-main-dev/node-01/elasticsearch.yml
[INFO] Disabling transport SSL (not needed for single-node)
[2026-01-02 04:47:10 UTC] USER=www-data EUID=0 PID=1453880 ACTION=fsop ARGS=sed -i /^xpack.security.transport.ssl.enabled:/d /etc/elasticsearch/identity-universe-main-dev/node-01/elasticsearch.yml

[ OK ] ==================================================================
[ OK ] Security and Transport SSL Configuration Complete
[ OK ] ==================================================================
[INFO] Environment: identity-universe-main-dev
[INFO] Nodes: 1
[INFO] Security enabled: true
[INFO] Transport SSL enabled: false (not required for single-node)

[INFO] === Next Step ===
Restart services to apply security configuration (step 04)
Executing: steps/03-http-ssl.sh
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
==================================================================
Direct HTTPS Configuration (native TLS, PEM)
==================================================================
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
Environment: identity-universe-main-dev
Nodes: 1
Node: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.179)
Port: 9200 (default port)
Domain: identity-universe-main-dev.fastorder.com

[INFO] === Single-Node Direct HTTPS Setup ===
🔐 Generated PKCS12 password: tipy1mFg... (32 chars)
[INFO] Using first node configuration: /etc/elasticsearch/identity-universe-main-dev/node-01
[2026-01-02 04:47:11 UTC] USER=www-data EUID=0 PID=1453943 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-universe-main-dev/node-01
[2026-01-02 04:47:11 UTC] USER=www-data EUID=0 PID=1453952 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-universe-main-dev/node-01
[INFO] Checking prerequisites...
[ OK ] ✓ Prerequisites verified
[INFO] Setting up temporary directories...
[2026-01-02 04:47:11 UTC] USER=www-data EUID=0 PID=1453963 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs
[2026-01-02 04:47:11 UTC] USER=www-data EUID=0 PID=1453972 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs
[2026-01-02 04:47:11 UTC] USER=www-data EUID=0 PID=1453981 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-universe-main-dev
[2026-01-02 04:47:11 UTC] USER=www-data EUID=0 PID=1453990 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-universe-main-dev/node-01
[2026-01-02 04:47:11 UTC] USER=www-data EUID=0 PID=1453999 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs
[2026-01-02 04:47:11 UTC] USER=www-data EUID=0 PID=1454008 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out
[ OK ] ✓ Directories created

[INFO] Building certificate instances configuration...
[INFO] Certificate instances configuration:
instances:
  - name: "identity-universe-main-dev-http"
    dns:  [ "localhost", "web-03", "search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com", "identity-universe-main-dev-node-01.fastorder.com", "search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com", "search-identity-universe-main-dev.fastorder.com", "identity-universe-main-dev-node-01.local" ]
    ip:   [ "10.100.1.179", "127.0.0.1", "::1" ]
[ OK ] ✓ Instances configuration created

[INFO] Generating HTTP Certificate Authority...
[2026-01-02 04:47:11 UTC] USER=www-data EUID=0 PID=1454026 ACTION=fsop ARGS=rm -f /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/http-ca.zip /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out/http-certs.zip
[2026-01-02 04:47:11 UTC] USER=www-data EUID=0 PID=1454035 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs
[2026-01-02 04:47:12 UTC] USER=www-data EUID=0 PID=1454044 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs
[2026-01-02 04:47:14 UTC] USER=www-data EUID=0 PID=1454087 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/http-ca.zip
Archive:  /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/http-ca.zip
  inflating: /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/ca/ca.crt  
  inflating: /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/ca/ca.key  
[ OK ] ✓ HTTP CA generated successfully

[INFO] Generating per-node HTTP certificates...
[2026-01-02 04:47:14 UTC] USER=www-data EUID=0 PID=1454099 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out
[2026-01-02 04:47:14 UTC] USER=www-data EUID=0 PID=1454108 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out
[2026-01-02 04:47:17 UTC] USER=www-data EUID=0 PID=1454150 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out/http-certs.zip
[2026-01-02 04:47:17 UTC] USER=www-data EUID=0 PID=1454159 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out/http
Archive:  /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out/http-certs.zip
  inflating: /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out/http/identity-universe-main-dev-http/identity-universe-main-dev-http.crt  
  inflating: /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out/http/identity-universe-main-dev-http/identity-universe-main-dev-http.key  
[ OK ] ✓ HTTP certificates generated successfully

[INFO] Installing certificates and configuring services...
[INFO] Configuring main service for single-node HTTPS...
[2026-01-02 04:47:17 UTC] USER=www-data EUID=0 PID=1454172 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-universe-main-dev/node-01/certs
[2026-01-02 04:47:17 UTC] USER=www-data EUID=0 PID=1454181 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-universe-main-dev/node-01/certs
[2026-01-02 04:47:17 UTC] USER=www-data EUID=0 PID=1454190 ACTION=fsop ARGS=cp /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out/http/identity-universe-main-dev-http/identity-universe-main-dev-http.crt /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http.crt
[2026-01-02 04:47:17 UTC] USER=www-data EUID=0 PID=1454199 ACTION=fsop ARGS=cp /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out/http/identity-universe-main-dev-http/identity-universe-main-dev-http.key /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http.key
[2026-01-02 04:47:17 UTC] USER=www-data EUID=0 PID=1454208 ACTION=fsop ARGS=cp /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/ca/ca.crt /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt
[2026-01-02 04:47:17 UTC] USER=www-data EUID=0 PID=1454217 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/identity-universe-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-01-02 04:47:17 UTC] USER=www-data EUID=0 PID=1454227 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-universe-main-dev/node-01/certs
[2026-01-02 04:47:17 UTC] USER=www-data EUID=0 PID=1454237 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http.key
[2026-01-02 04:47:17 UTC] USER=www-data EUID=0 PID=1454246 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http.crt /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt /etc/elasticsearch/identity-universe-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-01-02 04:47:17 UTC] USER=www-data EUID=0 PID=1454255 ACTION=fsop ARGS=sed -i -E -e /^\s*xpack\.security\.http\.ssl(\..*)?\s*:/d /etc/elasticsearch/identity-universe-main-dev/node-01/elasticsearch.yml
[2026-01-02 04:47:17 UTC] USER=www-data EUID=0 PID=1454264 ACTION=fsop ARGS=sed -i /^# --- BEGIN direct HTTPS (managed, PEM) ---$/,/^# --- END direct HTTPS (managed, PEM) ---$/d /etc/elasticsearch/identity-universe-main-dev/node-01/elasticsearch.yml
[2026-01-02 04:47:17 UTC] USER=www-data EUID=0 PID=1454273 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/identity-universe-main-dev/node-01/certs/fastorder_ra_root.crt
YAML: /etc/elasticsearch/identity-universe-main-dev/node-01/elasticsearch.yml
[ OK ]   ✓ Main service configured with HTTPS
[2026-01-02 04:47:17 UTC] USER=www-data EUID=0 PID=1454294 ACTION=fsop ARGS=rm -rf /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients
[2026-01-02 04:47:18 UTC] USER=www-data EUID=0 PID=1454303 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients
[2026-01-02 04:47:18 UTC] USER=www-data EUID=0 PID=1454312 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients
Archive:  /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client.zip
   creating: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/
  inflating: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.crt  
  inflating: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.key  
[INFO] Creating P12 keystore for es-client...
[2026-01-02 04:47:21 UTC] USER=www-data EUID=0 PID=1454370 ACTION=fsop ARGS=mv /tmp/es-client-1453914.p12 /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[ OK ] ✓ Created P12 keystore: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-01-02 04:47:21 UTC] USER=www-data EUID=0 PID=1454379 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients
[2026-01-02 04:47:21 UTC] USER=www-data EUID=0 PID=1454388 ACTION=fsop ARGS=chmod 640 /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.key
[2026-01-02 04:47:21 UTC] USER=www-data EUID=0 PID=1454397 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-01-02 04:47:21 UTC] USER=www-data EUID=0 PID=1454407 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.crt /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt
[INFO] Saving keystore passwords to secrets vault...
🔑 Configuring AWS credentials for secrets vault...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[INFO] 🔐 Vaulting search passwords to remote backend...
✅ Passwords vaulted to remote backend
✓ Keystore passwords saved to secrets vault: search/identity-universe-main-dev/keystore-passwords

[INFO] === Installing CA Certificate for Users ===
[INFO] HOME not set, skipping user CA installation

✓ Direct HTTPS configuration completed for environment: identity-universe-main-dev
[INFO] All services now serve HTTPS using PEM certificates
[INFO] Network binding: 10.100.1.179
[INFO] HTTPS endpoint: https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === Certificate Summary ===
CA Certificate: /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/ca/ca.crt
Certificate Directory: /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs
Main service certificates: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/

[INFO] === Next Steps ===
1. Restart Elasticsearch services to apply HTTPS configuration
2. Test HTTPS connectivity: curl https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
3. Verify certificates: openssl s_client -connect search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === SSL Certificate Information ===
• CA certificate installed for user - curl will work without -k flag
• Each certificate includes node-specific domain, VM IP, and localhost in Subject Alternative Names
• Certificates are valid for 3 years from generation date

[WARNING] Important: You'll need to restart Elasticsearch services for HTTPS to take effect
Executing: steps/04-restart-systemd-services.sh
==================================================================
STEP 4 (STRICT): Restart systemd services and verify secure health
==================================================================
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: identity-universe-main-dev
Nodes: 1
Per-node endpoints (all use default port 9200):
  Node 1: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.179)

[INFO] Building service list for environment: identity-universe-main-dev (1 nodes)
  - elasticsearch@identity-universe-main-dev-node-01.service (port 9200)

[INFO] Will restart 1 service(s) for environment: identity-universe-main-dev
[2026-01-02 04:47:26 UTC] USER=www-data EUID=0 PID=1454610 ACTION=passthru ARGS=systemctl daemon-reload

[INFO] === Ensuring VM IPs are configured correctly ===
[INFO] ✓ 10.100.1.179 already configured on eth0 for node-01

[INFO] === Ensuring transport SSL certificates for all nodes ===
[INFO] ✓ Transport certificates already exist for node-01

[INFO] === Restarting Services ===
↻ Restarting elasticsearch@identity-universe-main-dev-node-01.service ...
[2026-01-02 04:47:27 UTC] USER=www-data EUID=0 PID=1454663 ACTION=passthru ARGS=systemctl restart elasticsearch@identity-universe-main-dev-node-01.service
[2026-01-02 04:47:32 UTC] USER=www-data EUID=0 PID=1454769 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@identity-universe-main-dev-node-01.service
[ OK ] elasticsearch@identity-universe-main-dev-node-01.service is active
[INFO] Waiting 10s for Elasticsearch to start listening on ports...

[INFO] === Waiting for STRICT Secure Cluster Health ===
[INFO] Waiting for port 9200 on 10.100.1.179 (timeout 120s)...
[INFO] Waiting for cluster to form and be ready for write operations...
✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
[INFO] Cluster stable and ready for operations (20 consecutive healthy responses over 40s)

[INFO] Performing final cluster health check before password setup...
[INFO] Elastic password not found, running password setup...
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
╔════════════════════════════════════════════════════════════╗
║   Elasticsearch Password Management via AWS Secrets MGR   ║
╚════════════════════════════════════════════════════════════╝

Environment: identity-universe-main-dev
User:        elastic
Identifier:  node-01
AWS Secret:  fastorder/search/identity/universe/main/dev/elasticsearch/node-01

Using configuration path: /etc/elasticsearch/identity-universe-main-dev/node-01 (IDENTIFIER: node-01)
Node domain: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com
HTTP port: 9200 (default Elasticsearch port)
[INFO] xpack.security.enabled already true → no restart.
[INFO] No restart needed.
[2026-01-02 04:49:11 UTC] USER=www-data EUID=0 PID=1455807 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-universe-main-dev/node-01
[2026-01-02 04:49:11 UTC] USER=www-data EUID=0 PID=1455834 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/identity-universe-main-dev/node-01/users /etc/elasticsearch/identity-universe-main-dev/node-01/users_roles
[2026-01-02 04:49:11 UTC] USER=www-data EUID=0 PID=1455843 ACTION=fsop ARGS=chmod 660 /etc/elasticsearch/identity-universe-main-dev/node-01/users /etc/elasticsearch/identity-universe-main-dev/node-01/users_roles
✓ users/users_roles present and writable
[2026-01-02 04:49:11 UTC] USER=www-data EUID=0 PID=1455852 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/identity-universe-main-dev/node-01/elasticsearch.keystore
[2026-01-02 04:49:11 UTC] USER=www-data EUID=0 PID=1455861 ACTION=fsop ARGS=chmod 660 /etc/elasticsearch/identity-universe-main-dev/node-01/elasticsearch.keystore
✓ Keystore exists: /etc/elasticsearch/identity-universe-main-dev/node-01/elasticsearch.keystore
HTTPS is enabled in configuration
✓ Found HTTP CA certificate: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt
✓ Using client certificates for mTLS
Waiting for Elasticsearch to be reachable at https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200...
✓ Elasticsearch is reachable (HTTP 401)

ES_PATH_CONF: /etc/elasticsearch/identity-universe-main-dev/node-01
HTTP URL:    https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

Running HTTP reset (online, --batch)…
Note: Using HTTPS - tools will read SSL config from elasticsearch.yml
Command output:
Password for the [elastic] user successfully reset.
New value: h+yhmN0YCeA_Vt7epnWZ
Exit status: 0
✓ HTTP reset succeeded for elastic
Storing credentials in AWS Secrets Manager: fastorder/search/identity/universe/main/dev/elasticsearch/node-01
ℹ️  Setting Elasticsearch credentials in vault: fastorder/search/identity/universe/main/dev/elasticsearch/node-01
ℹ️  Setting secret in AWS Secrets Manager: fastorder/search/identity/universe/main/dev/elasticsearch/node-01
✅ Secret created: fastorder/search/identity/universe/main/dev/elasticsearch/node-01
✅ Elasticsearch credentials set in vault: fastorder/search/identity/universe/main/dev/elasticsearch/node-01
✓ Password stored in AWS Secrets Manager: fastorder/search/identity/universe/main/dev/elasticsearch/node-01
✓ Cache cleared for: fastorder/search/identity/universe/main/dev/elasticsearch/node-01

✓ Done. Password stored in AWS Secrets Manager: fastorder/search/identity/universe/main/dev/elasticsearch/node-01

Usage Examples:
  # Retrieve password using AWS CLI
  aws secretsmanager get-secret-value --secret-id fastorder/search/identity/universe/main/dev/elasticsearch/node-01 --region ${AWS_REGION:-me-central-1}

  # Using fastctl
  fastctl secrets get fastorder/search/identity/universe/main/dev/elasticsearch/node-01

  # Test connection
  curl -u elastic:$(fastctl secrets get fastorder/search/identity/universe/main/dev/elasticsearch/node-01 --field password) https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health
✓ Retrieved password from AWS Secrets Manager
[INFO] Testing cluster at: https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Using SSL CA certificate: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt
[INFO] Using client cert/key for mTLS
[INFO] Using client cert/key for mTLS: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.key
[INFO]   ⏳ waiting for secure cluster health (require 200) at https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200 (timeout 30s)...
[ OK ] Cluster health OK: green
[ OK ] Authentication successful with existing password

==================================================================
[ OK ] All services restarted successfully!
[ OK ] Cluster is healthy, HTTPS-secure, and responding with 200
[INFO] Environment: identity-universe-main-dev
[INFO] Services: 1
[INFO] Endpoint: https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === Manual verification (copy/paste) ===
curl -u 'elastic:h+yhmN0YCeA_Vt7epnWZ' \
  --cacert '/etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt' \
  --cert   '/etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.crt' \
  --key    '/etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.key' \
  'https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty'

[INFO] === Quick checks ===
curl -u 'elastic:h+yhmN0YCeA_Vt7epnWZ' --cacert '/etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt' --cert '/etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.crt' --key '/etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.key' https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cat/nodes?v
curl -u 'elastic:h+yhmN0YCeA_Vt7epnWZ' --cacert '/etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt' --cert '/etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.crt' --key '/etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.key' https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/
Executing: steps/05-test-elastic.sh
==================================================================
STEP 5: Test Elasticsearch Cluster
==================================================================
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: identity-universe-main-dev
Nodes: 1
Node: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.179)
Port: 9200 (default port)

[INFO] Using centralized test suite: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/02-make-https/lib/elasticsearch-test-suite.sh
[INFO] Executing centralized test suite with args: -v -t all --env identity-universe-main-dev -u elastic
[INFO] Using CURRENT_ENV_ID from environment: identity-universe-main-dev
[INFO] Loaded from topology.json: identity-universe-main-dev
[2026-01-02 04:49:27] Loaded environment: identity-universe-main-dev
[2026-01-02 04:49:27] Service: identity, Zone: universe, Branch: main, Env: dev
[2026-01-02 04:49:27] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-02 04:49:27] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-02 04:49:27] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
ℹ Using CURRENT_ENV_ID from environment: identity-universe-main-dev
ℹ Loaded from topology.json: identity-universe-main-dev
[2026-01-02 04:49:27] Loaded environment: identity-universe-main-dev
[2026-01-02 04:49:27] Service: identity, Zone: universe, Branch: main, Env: dev
[2026-01-02 04:49:27] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-02 04:49:27] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-02 04:49:27] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
ℹ Project root: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch
ℹ Environment:  identity-universe-main-dev
ℹ Nodes count:  1
ℹ Endpoint:      https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
ℹ Using CA:       /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt
ℹ Using mTLS:     /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.key
╔════════════════════════════════════════════╗
║    Elasticsearch Centralized Test Suite    ║
╚════════════════════════════════════════════╝

=== Authentication Test ===
✓ Loaded credentials for user elastic from AWS Secrets Manager
Curl (local): curl --cacert /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt --cert /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.crt --key /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.key  -u 'elastic:********' 'https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty'
✓ Local authentication successful (HTTP 200).
{
  "cluster_name" : "fastorder-identity-universe-main-dev",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 3,
  "active_shards" : 3,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Executing: steps/06-final-testing.sh
==================================================================
STEP 6: Final Testing and Verification
==================================================================
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: identity-universe-main-dev
Nodes: 1
Node: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.179)
Port: 9200 (default port)

[INFO] Using centralized test suite: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/02-make-https/lib/elasticsearch-test-suite.sh
[INFO] Using CURRENT_ENV_ID from environment: identity-universe-main-dev
[INFO] Loaded from topology.json: identity-universe-main-dev
[2026-01-02 04:49:30] Loaded environment: identity-universe-main-dev
[2026-01-02 04:49:30] Service: identity, Zone: universe, Branch: main, Env: dev
[2026-01-02 04:49:30] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-02 04:49:30] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-02 04:49:30] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
ℹ Using CURRENT_ENV_ID from environment: identity-universe-main-dev
ℹ Loaded from topology.json: identity-universe-main-dev
[2026-01-02 04:49:31] Loaded environment: identity-universe-main-dev
[2026-01-02 04:49:31] Service: identity, Zone: universe, Branch: main, Env: dev
[2026-01-02 04:49:31] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-02 04:49:31] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-02 04:49:31] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
ℹ Project root: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch
ℹ Environment:  identity-universe-main-dev
ℹ Nodes count:  1
ℹ Endpoint:      https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
ℹ Using CA:       /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt
ℹ Using mTLS:     /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.key
╔════════════════════════════════════════════╗
║    Elasticsearch Centralized Test Suite    ║
╚════════════════════════════════════════════╝

=== Authentication Test ===
✓ Loaded credentials for user elastic from AWS Secrets Manager
Curl (local): curl --cacert /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt --cert /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.crt --key /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.key  -u 'elastic:********' 'https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty'
✓ Local authentication successful (HTTP 200).
{
  "cluster_name" : "fastorder-identity-universe-main-dev",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 3,
  "active_shards" : 3,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Executing: steps/07-set-passwords.sh
==================================================================
STEP 7: Setting cluster passwords (bootstrap via alias)
==================================================================
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Using HTTPS with CA: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt (host: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com)
[INFO] Using centralized password setter: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/02-make-https/steps/../lib/elasticsearch-set-password.sh
[ OK ] Elastic password already valid (HTTP 200) via search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com; nothing to do.
Executing: steps/08-create-app-user.sh
==================================================================
STEP 8: Create Application User and Roles (cluster-scoped)
==================================================================
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: identity-universe-main-dev
Nodes: 1

[INFO] Using HTTPS with CA: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt (host: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com)
[ OK ] Retrieved elastic password from Vault (cluster scope).
[INFO] Configuration:
[INFO]   App User         : app_user
[INFO]   Read-only Role   : app_ro
[INFO]   Read-write Role  : app_rw
[INFO]   Index Patterns   : app-*,cdc-*,identity_universe_*,*_account_router
[INFO]   Endpoint         : https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] Creating read-only role: app_ro
[ OK ] ✓ Role app_ro ensured
[INFO] Creating read-write role: app_rw
[ OK ] ✓ Role app_rw ensured
[INFO] Creating/Updating application user: app_user
[ OK ] ✓ User app_user ensured
ℹ️  Setting Elasticsearch credentials in vault: fastorder/search/identity/universe/main/dev/elasticsearch/node-01/app_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/search/identity/universe/main/dev/elasticsearch/node-01/app_user
✅ Secret created: fastorder/search/identity/universe/main/dev/elasticsearch/node-01/app_user
✅ Elasticsearch credentials set in vault: fastorder/search/identity/universe/main/dev/elasticsearch/node-01/app_user
[ OK ] ✓ Stored app_user password under 'node-01/app_user'
ℹ️  Setting Elasticsearch credentials in vault: fastorder/search/identity/universe/main/dev/elasticsearch/cluster/app_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/search/identity/universe/main/dev/elasticsearch/cluster/app_user
✅ Secret created: fastorder/search/identity/universe/main/dev/elasticsearch/cluster/app_user
✅ Elasticsearch credentials set in vault: fastorder/search/identity/universe/main/dev/elasticsearch/cluster/app_user
[ OK ] ✓ Stored app_user password under 'cluster/app_user'
[INFO] Testing authentication for app_user...
[ OK ] ✓ Authentication test passed for app_user

[ OK ] ✓ Application user and roles created successfully!
[INFO] User    : app_user
[INFO] Roles   : app_ro, app_rw
[INFO] Patterns: app-*,cdc-*,identity_universe_*,*_account_router
[INFO] Endpoint: https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
Executing: steps/09-config.sh
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
Environment: identity-universe-main-dev
Nodes: 1
Node: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.179)
Port: 9200

✓ Auto mode: Cloud IMDS detected → MODE=role
[INFO] Mode: role

[INFO] AWS Region: me-central-1
[INFO] MODE=role → will purge any static S3 keys from each node keystore

[2026-01-02 04:51:00 UTC] USER=www-data EUID=0 PID=1457427 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-universe-main-dev/node-01
[INFO] • node-01 keystore cleared (role-based auth)
[2026-01-02 04:51:08 UTC] USER=www-data EUID=0 PID=1457577 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:51:08 UTC] USER=www-data EUID=0 PID=1457624 ACTION=passthru ARGS=systemctl restart elasticsearch@identity-universe-main-dev-node-01.service
✓ ✓ restarted elasticsearch@identity-universe-main-dev-node-01.service

⏳ Waiting for HTTPS readiness on https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Waiting HTTP readiness at https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/ (200/401/302)…
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
[OK] Ready: 401
⏳ Waiting for cluster health (green|yellow)
[INFO] Waiting health (green|yellow) at https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health…
[OK] 401 pre-auth received; security enabled.
✓ ✓ identity-universe-main-dev is responding via search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com

✓ ✓ AWS S3 configuration completed for environment: identity-universe-main-dev (1 nodes)
[INFO] Mode: role
[INFO] Region: me-central-1
Executing: steps/0ld-03-http-ssl.sh
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
==================================================================
STEP 5: HTTP SSL Configuration (Optional)
==================================================================
Environment: identity-universe-main-dev
Nodes: 1
Node: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.179)
Port: 9200 (default port)

[ OK ] 🚀 Auto mode/Default installation: Selecting Direct HTTPS configuration (option 1)

[ OK ] Configuring Direct HTTPS (Elasticsearch native SSL)...
──────────────────────────────────────────────────────────
[INFO] Environment: identity-universe-main-dev (1 nodes)
[INFO] Node: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.179)
[INFO] Port: 9200 (default port)
==================================================================
Direct HTTPS Configuration (native TLS, PEM)
==================================================================
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
Environment: identity-universe-main-dev
Nodes: 1
Node: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.179)
Port: 9200 (default port)
Domain: identity-universe-main-dev.fastorder.com

[INFO] === Single-Node Direct HTTPS Setup ===
🔐 Generated PKCS12 password: BwUPioL5... (32 chars)
[INFO] Using first node configuration: /etc/elasticsearch/identity-universe-main-dev/node-01
[2026-01-02 04:51:41 UTC] USER=www-data EUID=0 PID=1458098 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-universe-main-dev/node-01
[2026-01-02 04:51:41 UTC] USER=www-data EUID=0 PID=1458107 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-universe-main-dev/node-01
[INFO] Checking prerequisites...
[ OK ] ✓ Prerequisites verified
[INFO] Setting up temporary directories...
[2026-01-02 04:51:41 UTC] USER=www-data EUID=0 PID=1458118 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs
[2026-01-02 04:51:41 UTC] USER=www-data EUID=0 PID=1458127 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs
[2026-01-02 04:51:41 UTC] USER=www-data EUID=0 PID=1458136 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-universe-main-dev
[2026-01-02 04:51:41 UTC] USER=www-data EUID=0 PID=1458145 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-universe-main-dev/node-01
[2026-01-02 04:51:41 UTC] USER=www-data EUID=0 PID=1458154 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs
[2026-01-02 04:51:41 UTC] USER=www-data EUID=0 PID=1458163 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out
[ OK ] ✓ Directories created

[INFO] Building certificate instances configuration...
[INFO] Certificate instances configuration:
instances:
  - name: "identity-universe-main-dev-http"
    dns:  [ "localhost", "web-03", "search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com", "identity-universe-main-dev-node-01.fastorder.com", "search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com", "search-identity-universe-main-dev.fastorder.com", "identity-universe-main-dev-node-01.local" ]
    ip:   [ "10.100.1.179", "127.0.0.1", "::1" ]
[ OK ] ✓ Instances configuration created

[INFO] Generating HTTP Certificate Authority...
[2026-01-02 04:51:42 UTC] USER=www-data EUID=0 PID=1458181 ACTION=fsop ARGS=rm -f /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/http-ca.zip /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out/http-certs.zip
[2026-01-02 04:51:42 UTC] USER=www-data EUID=0 PID=1458190 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs
[2026-01-02 04:51:42 UTC] USER=www-data EUID=0 PID=1458199 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs
[2026-01-02 04:51:45 UTC] USER=www-data EUID=0 PID=1458341 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/http-ca.zip
Archive:  /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/http-ca.zip
  inflating: /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/ca/ca.crt  
  inflating: /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/ca/ca.key  
[ OK ] ✓ HTTP CA generated successfully

[INFO] Generating per-node HTTP certificates...
[2026-01-02 04:51:45 UTC] USER=www-data EUID=0 PID=1458353 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out
[2026-01-02 04:51:45 UTC] USER=www-data EUID=0 PID=1458362 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out
[2026-01-02 04:51:47 UTC] USER=www-data EUID=0 PID=1458400 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out/http-certs.zip
[2026-01-02 04:51:47 UTC] USER=www-data EUID=0 PID=1458409 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out/http
Archive:  /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out/http-certs.zip
  inflating: /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out/http/identity-universe-main-dev-http/identity-universe-main-dev-http.crt  
  inflating: /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out/http/identity-universe-main-dev-http/identity-universe-main-dev-http.key  
[ OK ] ✓ HTTP certificates generated successfully

[INFO] Installing certificates and configuring services...
[INFO] Configuring main service for single-node HTTPS...
[2026-01-02 04:51:47 UTC] USER=www-data EUID=0 PID=1458421 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-universe-main-dev/node-01/certs
[2026-01-02 04:51:47 UTC] USER=www-data EUID=0 PID=1458430 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-universe-main-dev/node-01/certs
[2026-01-02 04:51:47 UTC] USER=www-data EUID=0 PID=1458439 ACTION=fsop ARGS=cp /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out/http/identity-universe-main-dev-http/identity-universe-main-dev-http.crt /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http.crt
[2026-01-02 04:51:47 UTC] USER=www-data EUID=0 PID=1458448 ACTION=fsop ARGS=cp /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/out/http/identity-universe-main-dev-http/identity-universe-main-dev-http.key /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http.key
[2026-01-02 04:51:47 UTC] USER=www-data EUID=0 PID=1458457 ACTION=fsop ARGS=cp /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/ca/ca.crt /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt
[2026-01-02 04:51:48 UTC] USER=www-data EUID=0 PID=1458466 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/identity-universe-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-01-02 04:51:48 UTC] USER=www-data EUID=0 PID=1458475 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-universe-main-dev/node-01/certs
[2026-01-02 04:51:48 UTC] USER=www-data EUID=0 PID=1458484 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http.key
[2026-01-02 04:51:48 UTC] USER=www-data EUID=0 PID=1458493 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http.crt /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt /etc/elasticsearch/identity-universe-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-01-02 04:51:48 UTC] USER=www-data EUID=0 PID=1458502 ACTION=fsop ARGS=sed -i -E -e /^\s*xpack\.security\.http\.ssl(\..*)?\s*:/d /etc/elasticsearch/identity-universe-main-dev/node-01/elasticsearch.yml
[2026-01-02 04:51:48 UTC] USER=www-data EUID=0 PID=1458511 ACTION=fsop ARGS=sed -i /^# --- BEGIN direct HTTPS (managed, PEM) ---$/,/^# --- END direct HTTPS (managed, PEM) ---$/d /etc/elasticsearch/identity-universe-main-dev/node-01/elasticsearch.yml
[2026-01-02 04:51:48 UTC] USER=www-data EUID=0 PID=1458520 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/identity-universe-main-dev/node-01/certs/fastorder_ra_root.crt
YAML: /etc/elasticsearch/identity-universe-main-dev/node-01/elasticsearch.yml
[ OK ]   ✓ Main service configured with HTTPS
[2026-01-02 04:51:48 UTC] USER=www-data EUID=0 PID=1458539 ACTION=fsop ARGS=rm -rf /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients
[2026-01-02 04:51:48 UTC] USER=www-data EUID=0 PID=1458548 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients
[2026-01-02 04:51:48 UTC] USER=www-data EUID=0 PID=1458558 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients
Archive:  /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client.zip
   creating: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/
  inflating: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.crt  
  inflating: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.key  
[INFO] Creating P12 keystore for es-client...
[2026-01-02 04:51:50 UTC] USER=www-data EUID=0 PID=1458603 ACTION=fsop ARGS=mv /tmp/es-client-1458069.p12 /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[ OK ] ✓ Created P12 keystore: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-01-02 04:51:50 UTC] USER=www-data EUID=0 PID=1458612 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients
[2026-01-02 04:51:50 UTC] USER=www-data EUID=0 PID=1458621 ACTION=fsop ARGS=chmod 640 /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.key
[2026-01-02 04:51:50 UTC] USER=www-data EUID=0 PID=1458630 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-01-02 04:51:50 UTC] USER=www-data EUID=0 PID=1458639 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.crt /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt
[INFO] Saving keystore passwords to secrets vault...
🔑 Configuring AWS credentials for secrets vault...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[INFO] 🔐 Vaulting search passwords to remote backend...
✅ Passwords vaulted to remote backend
✓ Keystore passwords saved to secrets vault: search/identity-universe-main-dev/keystore-passwords

[INFO] === Installing CA Certificate for Users ===
[INFO] HOME not set, skipping user CA installation

✓ Direct HTTPS configuration completed for environment: identity-universe-main-dev
[INFO] All services now serve HTTPS using PEM certificates
[INFO] Network binding: 10.100.1.179
[INFO] HTTPS endpoint: https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === Certificate Summary ===
CA Certificate: /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs/ca/ca.crt
Certificate Directory: /etc/elasticsearch/identity-universe-main-dev/node-01/http-certs
Main service certificates: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/

[INFO] === Next Steps ===
1. Restart Elasticsearch services to apply HTTPS configuration
2. Test HTTPS connectivity: curl https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
3. Verify certificates: openssl s_client -connect search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === SSL Certificate Information ===
• CA certificate installed for user - curl will work without -k flag
• Each certificate includes node-specific domain, VM IP, and localhost in Subject Alternative Names
• Certificates are valid for 3 years from generation date

[WARNING] Important: You'll need to restart Elasticsearch services for HTTPS to take effect
[ OK ] ✓ Direct HTTPS configuration completed successfully

[ OK ] ==================================================================
[ OK ] HTTP SSL Configuration Complete
[ OK ] ==================================================================
[INFO] Environment: identity-universe-main-dev
[INFO] Nodes: 1
[INFO] Configuration applied to port: 9200 (default port for all nodes)

[INFO] === Next Steps ===
1. Verify Elasticsearch is running: systemctl status elasticsearch@identity-universe-main-dev-node-01.service
2. Test cluster health: curl https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health
3. Check SSL certificate: openssl s_client -connect search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
=== HTTPS Setup completed successfully! ===
Environment:  (1 nodes)
Domain: .fastorder.com
HTTPS endpoint: https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
Node IP: 10.100.1.179

✓ Step 2 completed successfully!

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 3: Executing Create Index Llm
Folder: 03-create-index-llm
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

==================================================================
Elasticsearch LLM/Semantic Search Setup
==================================================================
[INFO] Using web-provided environment: identity-universe-main-dev
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: identity-universe-main-dev
Service    : identity
🔍 Checking Elasticsearch availability…
✅ Elasticsearch is accessible at https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200

=== Phase 1: Common steps under /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps ===
   (no numbered steps in: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps)
=== Phase 2: Service-scoped steps for 'identity' under /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/identity ===
📚 Detected features: login

── Feature: login
▶️  Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/identity/login/01-create-model-and-pipeline.sh
==================================================================
STEP 1: Create Model and Ingest Pipeline
==================================================================
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] ES URL: https://search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Endpoint ID (ES): identity-text-embedding-001
[INFO] Provider model: text-embedding-3-large
[INFO] Pipeline ID: identity-embed-pipeline-001
[INFO] Checking authentication identity…
{
"username":"elastic","roles":["superuser"],"full_name":null,"email":null,"metadata":{"_reserved":true},"enabled":true,"authentication_realm":{"name":"reserved","type":"reserved"},"lookup_realm":{"name":"reserved","type":"reserved"},"authentication_type":"realm"
}
[INFO] Checking Elasticsearch license…
[INFO] License type: unknown
[WARN] Inference API requires Enterprise/Platinum license (found: unknown)
[WARN] Skipping inference endpoint and pipeline creation
[OK]   Setup completed (inference features skipped due to license)
✅ 01-create-model-and-pipeline.sh completed

▶️  Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/identity/login/02-create-index.sh
==================================================================
STEP 2: Create Semantic Search Index (initial bootstrap)
==================================================================
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] [create] Ensuring clean slate for: identity_universe_main_dev_account_router-000001
[INFO] Index identity_universe_main_dev_account_router-000001 does not exist (status 404), proceeding.
[INFO] [create] Creating index identity_universe_main_dev_account_router-000001 with write alias identity_universe_main_dev_account_router
[OK]   Index + alias ready.
   Index (concrete): identity_universe_main_dev_account_router-000001
   Alias (stable)  : identity_universe_main_dev_account_router  (is_write_index=true)
   Default pipeline: identity-embed-pipeline-001
   Vector dims     : 3072 (KNN cosine)
✅ 02-create-index.sh completed

▶️  Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/identity/login/03-llm.sh
==================================================================
STEP 2: Create Semantic Search Index (ILM bootstrap)
==================================================================
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] [cluster] Ensure disk watermarks permit allocation
[OK]   Cluster watermarks set/confirmed.
[INFO] [create] Create/Update ILM policy: identity-account-router-ilm
[OK]   ILM policy ready.
[INFO] [create] Create/Update index template: identity_universe_main_dev_account_router_template
[OK]   Index template ready.
[INFO] [check] Concrete index: identity_universe_main_dev_account_router-000001
[OK]   Concrete index identity_universe_main_dev_account_router-000001 already exists (skip create).
[INFO] [verify] Wait for index to be at least YELLOW
[OK]   Cluster health OK for identity_universe_main_dev_account_router-000001.
[INFO] [verify] Alias points to a concrete write index
[OK]   Alias verification passed.
[INFO] [explain] ILM status
{
  "indices" : {
    "identity_universe_main_dev_account_router-000001" : {
      "index" : "identity_universe_main_dev_account_router-000001",
      "managed" : false
    }
  }
}

[OK]   ILM/alias bootstrap complete.
   Index (concrete): identity_universe_main_dev_account_router-000001
   Alias (stable)  : identity_universe_main_dev_account_router  (is_write_index=true)
   ILM policy      : identity-account-router-ilm
   Default pipeline: identity-embed-pipeline-001
✅ 03-llm.sh completed

▶️  Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/identity/login/04-index-sample-data.sh
==================================================================
STEP 3: Index Sample Data
==================================================================
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[WARN] Pipeline 'identity-embed-pipeline-001' not found (HTTP 404); proceeding without it.
[INFO] [bulk] Index seed documents → identity_universe_main_dev_account_router
[WARN] Bulk completed with item-level errors. Showing first 50 lines:
{"errors":true,"took":0,"ingest_took":200,"items":[{"index":{"_index":"identity_universe_main_dev_account_router","_id":"auto-generated","status":400,"error":{"type":"illegal_argument_exception","reason":"pipeline with id [identity-embed-pipeline-001] does not exist"}}},{"index":{"_index":"identity_universe_main_dev_account_router","_id":"auto-generated","status":400,"error":{"type":"illegal_argument_exception","reason":"pipeline with id [identity-embed-pipeline-001] does not exist"}}},{"index":{"_index":"identity_universe_main_dev_account_router","_id":"auto-generated","status":400,"error":{"type":"illegal_argument_exception","reason":"pipeline with id [identity-embed-pipeline-001] does not exist"}}}]}[summary] items=3 errors=3
[INFO] [verify] Search a sample term: 'password'
  {
    "took" : 91,
    "timed_out" : false,
    "_shards" : {
      "total" : 1,
      "successful" : 1,
      "skipped" : 0,
      "failed" : 0
    },
    "hits" : {
      "total" : {
        "value" : 0,
        "relation" : "eq"
      },
      "max_score" : null,
      "hits" : [ ]
    }
  }
[OK]   Sample data indexing step completed.
✅ 04-index-sample-data.sh completed

▶️  Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/identity/login/05-create-cdc-index.sh
==================================================================
STEP 5: Create CDC Account Router Index (for dashboard visibility)
==================================================================
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Creating CDC index: identity_universe_main_dev_account_router
[OK]   Index identity_universe_main_dev_account_router already exists
✅ 05-create-cdc-index.sh completed

=== Phase 3: Optional search smoke tests ===
   (semantic search test script not found: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/search-semantic.sh)
   (hybrid search test script not found: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/hybrid-search.sh)
==================================================================
🎉 LLM/Semantic Search setup completed successfully!
==================================================================

Available commands:
  • Test semantic search:
    bash steps/search-semantic.sh en "password policy"
    bash steps/search-semantic.sh ar "كلمة المرور"

  • Test hybrid search:
    bash steps/hybrid-search.sh en "user authentication"
    bash steps/hybrid-search.sh ar "مصادقة المستخدم"

Alias   : identity_universe_main_dev_account_router
Index   : identity_universe_main_dev_account_router-000001
ILM     : identity-account-router-ilm
Model   : identity-text-embedding-001
Pipeline: identity-embed-pipeline-001
==================================================================

✓ Step 3 completed successfully!

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 4: Executing Monitoring Setup
Folder: 10-monitoring-setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using web-provided environment: identity-universe-main-dev
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Elasticsearch Monitoring Integration for identity-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-identity-universe-main-dev
[OK]   Observability cell endpoints registered for identity-universe-main-dev
[OK]   ✓ Observability cell is ready

[INFO] 2️⃣ Discovering Elasticsearch configuration...
[OK]   ✓ Found Elasticsearch at 10.100.1.179:9200

[INFO] 3️⃣ Setting up elasticsearch_exporter integration...
[INFO] Using elasticsearch_exporter port: 9114
[INFO] SSL certificates configured for elasticsearch_exporter:
[INFO]   CA cert: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt
[INFO]   Client cert: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.crt
[INFO]   Client key: /etc/elasticsearch/identity-universe-main-dev/node-01/certs/clients/es-client/es-client.key
[INFO] Checking observability cell readiness: obs-identity-universe-main-dev
[OK]   Observability cell endpoints registered for identity-universe-main-dev
[INFO] Setting up elasticsearch_exporter for identity-universe-main-dev
[INFO] Elasticsearch exporter will bind to: 10.100.1.179:9114
[2026-01-02 04:52:14 UTC] USER=www-data EUID=0 PID=1459455 ACTION=passthru ARGS=mv /tmp/elasticsearch_exporter-identity-universe-main-dev.service /etc/systemd/system/elasticsearch_exporter-identity-universe-main-dev.service
[2026-01-02 04:52:14 UTC] USER=www-data EUID=0 PID=1459464 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-02 04:52:15 UTC] USER=www-data EUID=0 PID=1459509 ACTION=passthru ARGS=systemctl enable elasticsearch_exporter-identity-universe-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/elasticsearch_exporter-identity-universe-main-dev.service -> /etc/systemd/system/elasticsearch_exporter-identity-universe-main-dev.service.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  IP Conflict Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: identity-universe-main-dev
IP Address:  10.100.1.179
Port:        9114
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔍 Checking IP conflict for identity-universe-main-dev on 10.100.1.179:9114...
✅ IP 10.100.1.179:9114 is available - no conflicts detected

🔍 Checking for orphaned processes that might conflict...
✅ No orphaned processes detected

✅ All checks passed - safe to proceed with identity-universe-main-dev setup
[2026-01-02 04:52:15 UTC] USER=www-data EUID=0 PID=1459583 ACTION=passthru ARGS=systemctl restart elasticsearch_exporter-identity-universe-main-dev.service
[OK]   elasticsearch_exporter configured on 10.100.1.179:9114
[INFO] Register this endpoint in metrics-identity-universe-main-dev.fastorder.com scrape config
[OK]   ✓ elasticsearch_exporter integration complete

[INFO] 3.5️⃣ Configuring Prometheus to scrape Elasticsearch metrics...
[2026-01-02 04:52:18 UTC] USER=www-data EUID=0 PID=1459642 ACTION=passthru ARGS=grep -q job_name: 'elasticsearch' /etc/prometheus/obs-identity-universe-main-dev/prometheus.yml
[INFO] Adding Elasticsearch scrape target to Prometheus configuration...
[2026-01-02 04:52:18 UTC] USER=www-data EUID=0 PID=1459665 ACTION=fsop ARGS=cp /etc/prometheus/obs-identity-universe-main-dev/prometheus.yml /etc/prometheus/obs-identity-universe-main-dev/prometheus.yml.backup-1767329538
[INFO] Created backup: /etc/prometheus/obs-identity-universe-main-dev/prometheus.yml.backup-1767329538
[2026-01-02 04:52:18 UTC] USER=www-data EUID=0 PID=1459686 ACTION=passthru ARGS=sed -i /# Prometheus self-monitoring/r /tmp/prometheus_es_add.yml /etc/prometheus/obs-identity-universe-main-dev/prometheus.yml
[2026-01-02 04:52:18 UTC] USER=www-data EUID=0 PID=1459708 ACTION=passthru ARGS=grep -q job_name: 'elasticsearch' /etc/prometheus/obs-identity-universe-main-dev/prometheus.yml
[INFO] ✓ Elasticsearch job successfully inserted into config
[INFO] Validating Prometheus configuration with promtool...
Checking /etc/prometheus/obs-identity-universe-main-dev/prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: /etc/prometheus/obs-identity-universe-main-dev/prometheus.yml is valid prometheus config file syntax

Checking /etc/prometheus/obs-identity-universe-main-dev/rules/basic_alerts.yml
  SUCCESS: 4 rules found

[OK]   ✓ Prometheus configuration validation PASSED
[OK]   ✓ Prometheus configuration updated successfully
[2026-01-02 04:52:18 UTC] USER=www-data EUID=0 PID=1459739 ACTION=passthru ARGS=systemctl is-active --quiet prometheus-obs-identity-universe-main-dev.service
[INFO] Reloading Prometheus configuration...
[2026-01-02 04:52:18 UTC] USER=www-data EUID=0 PID=1459760 ACTION=passthru ARGS=systemctl restart prometheus-obs-identity-universe-main-dev.service
[2026-01-02 04:52:22 UTC] USER=www-data EUID=0 PID=1459792 ACTION=passthru ARGS=systemctl is-active --quiet prometheus-obs-identity-universe-main-dev.service
[OK]   ✓ Prometheus reloaded successfully
[2026-01-02 04:52:22 UTC] USER=www-data EUID=0 PID=1459813 ACTION=fsop ARGS=rm -f /tmp/prometheus_es_add.yml

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Elasticsearch Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Service: elasticsearch_exporter-identity-universe-main-dev.service
[INFO] Metrics: http://localhost:9114/metrics
[INFO] Prometheus: https://metrics-identity-universe-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-identity-universe-main-dev.fastorder.com
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 4️⃣ Registering Elasticsearch nodes to monitoring database...
[INFO]    Constructed FQDN: search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com
[INFO] Registering: identity-universe-main-dev-node-01
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Elasticsearch
[INFO]   Identifier:        identity-universe-main-dev-node-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.179
[INFO]   Port:              9200
[INFO]   FQDN:              search-identity-universe-main-dev-elasticsearch-node-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-universe-main-dev (service=identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 6598639a-8a7c-4a4d-b013-d7cf0a9228d3
[SUCCESS] Environment UUID: b6092921-3a75-44ac-9080-96a7ca43bec0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✓ Registered: identity-universe-main-dev-node-01
[OK]   ✓ Elasticsearch node registration completed successfully

[INFO] 5️⃣ Verifying monitoring integration...

[INFO] Checking elasticsearch_exporter service...
[OK]   ✓ elasticsearch_exporter-identity-universe-main-dev.service is ACTIVE
[INFO] Checking Prometheus service...
[OK]   ✓ prometheus-obs-identity-universe-main-dev.service is ACTIVE
[INFO] Validating Prometheus configuration...
[OK]   ✓ Prometheus configuration is VALID
[INFO] Checking Prometheus targets (waiting 35s for first scrape cycle)...
[2026-01-02 04:52:58 UTC] USER=www-data EUID=0 PID=1460152 ACTION=passthru ARGS=grep -q tls_server_config /etc/prometheus/obs-identity-universe-main-dev/web-config.yml
[OK]   ✓ Prometheus has Elasticsearch target configured
[OK]   ✓ Elasticsearch target is UP and being scraped

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ All monitoring integration steps completed
[INFO] ✅ All verifications PASSED
[INFO] ✅ Elasticsearch registered to dashboard database
[INFO] ✅ Prometheus scraping Elasticsearch metrics
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


✓ Step 4 completed successfully!

════════════════════════════════════════════════════════════════
🎉 All deployment tasks completed successfully!

✓ ✅ Search infrastructure (elasticsearch) setup completed successfully

[INFO] Loaded from topology.json: identity-universe-main-dev
[2026-01-02 04:52:58] Loaded environment: identity-universe-main-dev
[2026-01-02 04:52:58] Service: identity, Zone: universe, Branch: main, Env: dev
[2026-01-02 04:52:58] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-02 04:52:58] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-02 04:52:58] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
[INFO] Starting finalizing setup process...
[INFO] Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps
[INFO] Environment: identity-universe-main-dev

[INFO] Found 3 step(s) to execute

[INFO] 📦 Step 1/3: enable_disable_all_applications...
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
════════════════════════════════════════════════════════════════════════════════
  Environment Services Management
════════════════════════════════════════════════════════════════════════════════
  Environment:  identity-universe-main-dev
  Action:       enable
  Triggered by: false
════════════════════════════════════════════════════════════════════════════════

🔍 Scanning for environment-specific services...
✅ Found 1 services for environment: identity-universe-main-dev

📋 Services to enable:
────────────────────────────────────────────────────────────────────────────────
  • elasticsearch@identity-universe-main-dev-node-01.service     [active/unmasked/enabled]
────────────────────────────────────────────────────────────────────────────────


❌ Cancelled by user
[OK] ✅ Step 1 completed: 01-enable_disable_all_applications.sh

[INFO] 📦 Step 2/3: verify monitoring...
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Monitoring Verification for identity-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Detecting installed services...
Failed to print table: Broken pipe
[OK] ✓ Elasticsearch detected
Failed to print table: Broken pipe
[OK] ✓ Kafka detected
Failed to print table: Broken pipe
[OK] ✓ PgBouncer detected

[INFO] Services to verify: elasticsearch kafka pgbouncer

[INFO] 2️⃣ Verifying exporters are running...
[OK] ✓ Elasticsearch exporter is running
[ERROR] ✗ Kafka JMX exporter not enabled in systemd service
[WARN] ⚠️  PgBouncer exporter is not running (may not be configured)

[INFO] 3️⃣ Verifying Prometheus configuration...
[2026-01-02 04:53:00 UTC] USER=www-data EUID=0 PID=1460367 ACTION=passthru ARGS=grep -q job_name: 'elasticsearch' /etc/prometheus/obs-identity-universe-main-dev/prometheus.yml
[OK] ✓ elasticsearch is configured in Prometheus
[2026-01-02 04:53:00 UTC] USER=www-data EUID=0 PID=1460388 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-identity-universe-main-dev/prometheus.yml
[WARN] ⚠️  kafka is not configured in Prometheus scrape targets
[2026-01-02 04:53:00 UTC] USER=www-data EUID=0 PID=1460409 ACTION=passthru ARGS=grep -q job_name: 'pgbouncer' /etc/prometheus/obs-identity-universe-main-dev/prometheus.yml
[WARN] ⚠️  pgbouncer is not configured in Prometheus scrape targets

[INFO] 4️⃣ Verifying Prometheus is actively scraping...
[OK] ✓ Prometheus is running
[OK] ✓ elasticsearch target is UP in Prometheus
[WARN] ⚠️  kafka target is not UP in Prometheus (may still be initializing)
[WARN] ⚠️  pgbouncer target is not UP in Prometheus (may still be initializing)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Monitoring Verification Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[WARN] Some monitoring issues were detected:

[WARN] Exporter Issues:
  - Kafka JMX exporter not enabled in systemd

[WARN] Prometheus Configuration Issues:
  - kafka not configured in Prometheus
  - pgbouncer not configured in Prometheus

[WARN] Automatically running monitoring setup scripts to fix issues...

[INFO] Running Kafka monitoring setup...
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Kafka Monitoring Integration for identity-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-identity-universe-main-dev
[OK]   Observability cell endpoints registered for identity-universe-main-dev
[OK]   ✓ Observability cell is ready

[INFO] 2️⃣ Setting up Kafka JMX exporter integration...
[INFO] Checking observability cell readiness: obs-identity-universe-main-dev
[OK]   Observability cell endpoints registered for identity-universe-main-dev
[INFO] Setting up Kafka JMX exporter for identity-universe-main-dev
[INFO] JMX Prometheus Java Agent already exists at /opt/kafka/libs/jmx_prometheus_javaagent.jar
[2026-01-02 04:53:06 UTC] USER=www-data EUID=0 PID=1460523 ACTION=passthru ARGS=mv /tmp/jmx_exporter.yml /opt/kafka/config/jmx_exporter.yml
[2026-01-02 04:53:06 UTC] USER=www-data EUID=0 PID=1460533 ACTION=passthru ARGS=chmod 644 /opt/kafka/config/jmx_exporter.yml
[OK]   JMX exporter configuration created at /opt/kafka/config/jmx_exporter.yml
[OK]   JMX exporter configuration created
[INFO] Configuring Kafka systemd services to use JMX exporter...
[2026-01-02 04:53:06 UTC] USER=www-data EUID=0 PID=1460563 ACTION=fsop ARGS=test -f /etc/systemd/system/[2026-01-02
[INFO] All Kafka services already configured with JMX exporter
[OK]   Kafka JMX exporter integration complete
[INFO] Metrics endpoint: http://142.93.238.16:9308/metrics
[INFO] Prometheus will automatically scrape: https://metrics-identity-universe-main-dev.fastorder.com:9090
[INFO] View dashboards at: https://dashboards-identity-universe-main-dev.fastorder.com
[OK]   ✓ Kafka JMX exporter integration complete
[INFO] Configuring KAFKA_OPTS environment variable for kafka user...
[2026-01-02 04:53:06 UTC] USER=www-data EUID=0 PID=1460584 ACTION=passthru ARGS=grep -q KAFKA_OPTS.*javaagent.*jmx_prometheus_javaagent /home/kafka/.bashrc
[OK]   ✓ KAFKA_OPTS already configured
[INFO] 2.5️⃣ Enabling JMX exporter in Kafka systemd service...
[OK]   ✓ JMX exporter already enabled in Kafka systemd services
[INFO] 2.6️⃣ Configuring Prometheus to scrape Kafka metrics...
[2026-01-02 04:53:06 UTC] USER=www-data EUID=0 PID=1460605 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-identity-universe-main-dev/prometheus.yml
[INFO] Adding Kafka scrape target to Prometheus configuration...
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[2026-01-02 04:53:06 UTC] USER=www-data EUID=0 PID=1460638 ACTION=passthru ARGS=sed -i /# Prometheus self-monitoring/r /tmp/prometheus_kafka_add.yml /etc/prometheus/obs-identity-universe-main-dev/prometheus.yml
[ERROR] Invalid Prometheus configuration - rolling back
[2026-01-02 04:53:07 UTC] USER=www-data EUID=0 PID=1460678 ACTION=passthru ARGS=sed -i /job_name: 'kafka'/,+6d /etc/prometheus/obs-identity-universe-main-dev/prometheus.yml
[2026-01-02 04:53:07 UTC] USER=www-data EUID=0 PID=1460699 ACTION=fsop ARGS=rm -f /tmp/prometheus_kafka_add.yml

[INFO] 3️⃣ Registering Kafka nodes to monitoring database...
[INFO] Registering Kafka Broker to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Kafka Broker
[INFO]   Identifier:        identity-universe-main-dev-broker-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                142.93.238.16
[INFO]   Port:              9092
[INFO]   FQDN:              eventbus-identity-universe-main-dev-kafka-broker-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-universe-main-dev (service=identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 4e57c665-e0f2-449b-ba3b-846938e81540
[SUCCESS] Environment UUID: b6092921-3a75-44ac-9080-96a7ca43bec0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✓ Kafka broker registered
[INFO] Registering Kafka Connect to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Kafka Connect
[INFO]   Identifier:        identity-universe-main-dev-connect-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8083
[INFO]   FQDN:              eventbus-identity-universe-main-dev-kafka-connect.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-universe-main-dev (service=identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 44479f46-1b71-4b9a-8458-a18075d37df6
[SUCCESS] Environment UUID: b6092921-3a75-44ac-9080-96a7ca43bec0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
[OK]   ✓ Kafka Connect registered
[INFO] Schema Registry not running, skipping registration

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Kafka Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Metrics: http://localhost:9308/metrics
[INFO] Prometheus: https://metrics-identity-universe-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-identity-universe-main-dev.fastorder.com
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✓ Kafka monitoring setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Step 2 completed: 02-verify-monitoring.sh

[INFO] 📦 Step 3/3: register backup infrastructure...
[INFO] Loaded environment: identity-universe-main-dev (svc=identity zone=universe env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔧 Registering Core Services & Backup Infrastructure for identity-universe-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Registering Main App...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Main App
[INFO]   Identifier:        identity-universe-main-dev-main-app
[INFO]   Identifier Parent: application
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8080
[INFO]   FQDN:              app-identity-universe-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-universe-main-dev (service=identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: dd57532c-ac81-4288-997b-7069186d983c
[SUCCESS] Environment UUID: b6092921-3a75-44ac-9080-96a7ca43bec0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
/opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps/03-register-backup-infrastructure.sh: line 70: ok: command not found

[INFO] 2️⃣ Registering Audit Service...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Audit Service
[INFO]   Identifier:        identity-universe-main-dev-audit
[INFO]   Identifier Parent: application
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8081
[INFO]   FQDN:              audit-identity-universe-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-universe-main-dev (service=identity, zone=universe, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: dbb7322b-bfb5-4bed-b936-21eb0341ba39
[SUCCESS] Environment UUID: b6092921-3a75-44ac-9080-96a7ca43bec0
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/b6092921-3a75-44ac-9080-96a7ca43bec0
/opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps/03-register-backup-infrastructure.sh: line 85: ok: command not found

[INFO] 3️⃣ Registering PostgreSQL Backup Node...
[ERROR] Invalid identifier format: backup-db
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register PostgreSQL backup node (non-blocking)

[INFO] 4️⃣ Registering Elasticsearch Backup Node...
[ERROR] Invalid identifier format: backup-search
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Elasticsearch backup node (non-blocking)

[INFO] 5️⃣ Registering Kafka Backup Node...
[ERROR] Invalid identifier format: backup-eventbus
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Kafka backup node (non-blocking)

[INFO] 6️⃣ Registering Backup Orchestrator...
[ERROR] Invalid identifier format: backup-orchestrator
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Backup orchestrator (non-blocking)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Core Services & Backup Infrastructure Registration Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Registered core services:
[INFO]   🚀 main-app            → Core application service
[INFO]   📋 audit               → Centralized audit logging (WORM)

[INFO] Registered backup nodes:
[INFO]   📦 backup-db           → PostgreSQL backup (pgBackRest, PITR)
[INFO]   📦 backup-search       → Elasticsearch snapshots (ILM, S3)
[INFO]   📦 backup-eventbus     → Kafka log segments (replication)
[INFO]   📦 backup-orchestrator → Central backup coordination

[INFO] Dashboard: https://skeleton.dev.fastorder.com/dashboard/monitoring
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Step 3 completed: 03-register-backup-infrastructure.sh


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ finalizing setup completed successfully!
[OK] Executed all 3 steps
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Environment: identity-universe-main-dev
[INFO] Service: identity
[INFO] Zone: universe
[INFO] Branch: main
[INFO] Env: dev