📊 Provisioning Job Status

════════════════════════════════════════════════════════════════
  FastOrder Pre-Flight Validation Checks
════════════════════════════════════════════════════════════════

[INFO] Checking SSH connectivity to target host...
[✓] Target is localhost, skipping SSH check

[INFO] Checking available disk space...
[⚠] Disk space limited: 20GB available (recommended: 50GB)
  → PostgreSQL + Elasticsearch may experience space pressure

[INFO] Checking available memory...
[⚠] Memory low: 7GB (minimum: 4GB, recommended: 16GB)
  → Suitable for development/testing only
  → Reduce component counts: use 1 ES node, 1 PG worker, minimal standby nodes
  → Production environments require 16GB+

[INFO] Checking critical port availability...
[✓] Port 5432 in use on specific IP (10.100.1.189:5432) - OK, can use different IP
[✓] Port 9200 in use on specific IP ([::ffff:10.100.1.179]) - OK, can use different IP
[✓] Port 9300 in use on specific IP ([::ffff:10.100.1.179]) - OK, can use different IP
[✓] Port 9092 in use on specific IP ([::ffff:10.100.1.212]) - OK, can use different IP
[✓] Port 2181 available (Zookeeper)

[INFO] Checking DNS resolution...
[✓] DNS resolution working: google.com
[✓] DNS resolution working: github.com
[✓] DNS resolution working: archive.ubuntu.com

[INFO] Checking required system commands...
[✓] Command available: curl
[✓] Command available: wget
[✓] Command available: git
[✓] Command available: sudo
[✓] Command available: systemctl
[✓] Command available: apt-get

[INFO] Checking current system load...
[✓] System load normal: 1.17 (4 CPUs)

[INFO] Checking for existing environment conflicts...
[✓] No conflicting services found for: identity-uae-main-dev

════════════════════════════════════════════════════════════════
  Pre-Flight Check Summary
════════════════════════════════════════════════════════════════
[⚠] 2 warning(s) detected

⚠️  Environment can proceed with caution
   Review warnings above and consider remediation

[INFO] Using web-provided environment: identity-sau-main-dev
[INFO] Auto-creating state directory for identity-sau-main-dev...
[ OK ] Created topology.json for identity-sau-main-dev
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=10.100.1.51)
[2026-01-03_07:27:38] Starting Terraform provisioning step
[2026-01-03_07:27:38] Service: identity
[2026-01-03_07:27:38] Zone: sau
[2026-01-03_07:27:38] Environment: dev
[2026-01-03_07:27:38] Resource: web-03
[2026-01-03_07:27:38] Terraform binary: /home/ab/bin/terraform
[2026-01-03_07:27:38] HOME: /home/www-data
[2026-01-03_07:27:38] AWS Config: /home/ab/.aws/config
[2026-01-03_07:27:38] AWS Credentials: /home/ab/.aws/credentials
[2026-01-03_07:27:38] Terraform directory: /opt/fastorder/cli/terraform/examples/citus-production
[2026-01-03_07:27:38] Running terraform init...

Initializing the backend...
Upgrading modules...
- citus_cluster in ../../modules/citus_cluster

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 5.0"...
- Using previously-installed hashicorp/aws v5.100.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
[2026-01-03_07:27:46] ✓ Terraform init succeeded
[2026-01-03_07:27:46] Running terraform validate...
Success! The configuration is valid.

[2026-01-03_07:27:52] ✓ Terraform validate succeeded
[2026-01-03_07:27:52] Running terraform plan...
module.citus_cluster.data.aws_caller_identity.current: Reading...
module.citus_cluster.data.aws_caller_identity.current: Read complete after 0s [id=464621692046]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.citus_cluster.aws_iam_instance_profile.citus will be created
  + resource "aws_iam_instance_profile" "citus" {
      + arn         = (known after apply)
      + create_date = (known after apply)
      + id          = (known after apply)
      + name        = (known after apply)
      + name_prefix = "citus-prod-"
      + path        = "/"
      + role        = (known after apply)
      + tags        = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-prod"
        }
      + tags_all    = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
        }
      + unique_id   = (known after apply)
    }

  # module.citus_cluster.aws_iam_role.citus will be created
  + resource "aws_iam_role" "citus" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRole"
                      + Effect    = "Allow"
                      + Principal = {
                          + Service = "ec2.amazonaws.com"
                        }
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = (known after apply)
      + name_prefix           = "citus-prod-"
      + path                  = "/"
      + tags                  = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-prod"
        }
      + tags_all              = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
        }
      + unique_id             = (known after apply)
    }

  # module.citus_cluster.aws_iam_role_policy.secrets_manager[0] will be created
  + resource "aws_iam_role_policy" "secrets_manager" {
      + id          = (known after apply)
      + name        = (known after apply)
      + name_prefix = "secrets-access-"
      + policy      = jsonencode(
            {
              + Statement = [
                  + {
                      + Action   = [
                          + "secretsmanager:GetSecretValue",
                          + "secretsmanager:DescribeSecret",
                        ]
                      + Effect   = "Allow"
                      + Resource = "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/db/web/ksa/main/dev/postgresqladmin/ksa/prod*"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + role        = (known after apply)
    }

  # module.citus_cluster.aws_iam_role_policy_attachment.cloudwatch will be created
  + resource "aws_iam_role_policy_attachment" "cloudwatch" {
      + id         = (known after apply)
      + policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
      + role       = (known after apply)
    }

  # module.citus_cluster.aws_iam_role_policy_attachment.ssm will be created
  + resource "aws_iam_role_policy_attachment" "ssm" {
      + id         = (known after apply)
      + policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
      + role       = (known after apply)
    }

  # module.citus_cluster.aws_instance.coordinator will be created
  + resource "aws_instance" "coordinator" {
      + ami                                  = "ami-0b2aae5f4283c0df2"
      + arn                                  = (known after apply)
      + associate_public_ip_address          = (known after apply)
      + availability_zone                    = (known after apply)
      + cpu_core_count                       = (known after apply)
      + cpu_threads_per_core                 = (known after apply)
      + disable_api_stop                     = (known after apply)
      + disable_api_termination              = (known after apply)
      + ebs_optimized                        = (known after apply)
      + enable_primary_ipv6                  = (known after apply)
      + get_password_data                    = false
      + host_id                              = (known after apply)
      + host_resource_group_arn              = (known after apply)
      + iam_instance_profile                 = (known after apply)
      + id                                   = (known after apply)
      + instance_initiated_shutdown_behavior = (known after apply)
      + instance_lifecycle                   = (known after apply)
      + instance_state                       = (known after apply)
      + instance_type                        = "r6i.2xlarge"
      + ipv6_address_count                   = (known after apply)
      + ipv6_addresses                       = (known after apply)
      + key_name                             = (known after apply)
      + monitoring                           = (known after apply)
      + outpost_arn                          = (known after apply)
      + password_data                        = (known after apply)
      + placement_group                      = (known after apply)
      + placement_partition_number           = (known after apply)
      + primary_network_interface_id         = (known after apply)
      + private_dns                          = (known after apply)
      + private_ip                           = (known after apply)
      + public_dns                           = (known after apply)
      + public_ip                            = (known after apply)
      + secondary_private_ips                = (known after apply)
      + security_groups                      = (known after apply)
      + source_dest_check                    = true
      + spot_instance_request_id             = (known after apply)
      + subnet_id                            = "subnet-0a1f5a9a74ed030cf"
      + tags                                 = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-coordinator-prod"
          + "Role"        = "coordinator"
          + "Service"     = "citus"
        }
      + tags_all                             = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-coordinator-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
          + "Role"        = "coordinator"
          + "Service"     = "citus"
        }
      + tenancy                              = (known after apply)
      + user_data                            = "2a9e41ea765dcf3b3046ee10d2f458c18f00e430"
      + user_data_base64                     = (known after apply)
      + user_data_replace_on_change          = false
      + vpc_security_group_ids               = (known after apply)

      + ebs_block_device {
          + delete_on_termination = false
          + device_name           = "/dev/sdf"
          + encrypted             = true
          + iops                  = 3000
          + kms_key_id            = (known after apply)
          + snapshot_id           = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-coordinator-prod-data"
            }
          + tags_all              = (known after apply)
          + throughput            = 125
          + volume_id             = (known after apply)
          + volume_size           = 500
          + volume_type           = "gp3"
        }

      + root_block_device {
          + delete_on_termination = false
          + device_name           = (known after apply)
          + encrypted             = true
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-coordinator-prod-root"
            }
          + tags_all              = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = 100
          + volume_type           = "gp3"
        }
    }

  # module.citus_cluster.aws_instance.workers[0] will be created
  + resource "aws_instance" "workers" {
      + ami                                  = "ami-0b2aae5f4283c0df2"
      + arn                                  = (known after apply)
      + associate_public_ip_address          = (known after apply)
      + availability_zone                    = (known after apply)
      + cpu_core_count                       = (known after apply)
      + cpu_threads_per_core                 = (known after apply)
      + disable_api_stop                     = (known after apply)
      + disable_api_termination              = (known after apply)
      + ebs_optimized                        = (known after apply)
      + enable_primary_ipv6                  = (known after apply)
      + get_password_data                    = false
      + host_id                              = (known after apply)
      + host_resource_group_arn              = (known after apply)
      + iam_instance_profile                 = (known after apply)
      + id                                   = (known after apply)
      + instance_initiated_shutdown_behavior = (known after apply)
      + instance_lifecycle                   = (known after apply)
      + instance_state                       = (known after apply)
      + instance_type                        = "r6i.2xlarge"
      + ipv6_address_count                   = (known after apply)
      + ipv6_addresses                       = (known after apply)
      + key_name                             = (known after apply)
      + monitoring                           = (known after apply)
      + outpost_arn                          = (known after apply)
      + password_data                        = (known after apply)
      + placement_group                      = (known after apply)
      + placement_partition_number           = (known after apply)
      + primary_network_interface_id         = (known after apply)
      + private_dns                          = (known after apply)
      + private_ip                           = (known after apply)
      + public_dns                           = (known after apply)
      + public_ip                            = (known after apply)
      + secondary_private_ips                = (known after apply)
      + security_groups                      = (known after apply)
      + source_dest_check                    = true
      + spot_instance_request_id             = (known after apply)
      + subnet_id                            = "subnet-0a1f5a9a74ed030cf"
      + tags                                 = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-worker-0-prod"
          + "Role"        = "worker"
          + "Service"     = "citus"
          + "WorkerIndex" = "0"
        }
      + tags_all                             = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-worker-0-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
          + "Role"        = "worker"
          + "Service"     = "citus"
          + "WorkerIndex" = "0"
        }
      + tenancy                              = (known after apply)
      + user_data                            = "7b4bd87c9982aab7fa463c8d12e99399661f8bde"
      + user_data_base64                     = (known after apply)
      + user_data_replace_on_change          = false
      + vpc_security_group_ids               = (known after apply)

      + ebs_block_device {
          + delete_on_termination = false
          + device_name           = "/dev/sdf"
          + encrypted             = true
          + iops                  = 3000
          + kms_key_id            = (known after apply)
          + snapshot_id           = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-worker-0-prod-data"
            }
          + tags_all              = (known after apply)
          + throughput            = 125
          + volume_id             = (known after apply)
          + volume_size           = 500
          + volume_type           = "gp3"
        }

      + root_block_device {
          + delete_on_termination = false
          + device_name           = (known after apply)
          + encrypted             = true
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-worker-0-prod-root"
            }
          + tags_all              = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = 100
          + volume_type           = "gp3"
        }
    }

  # module.citus_cluster.aws_instance.workers[1] will be created
  + resource "aws_instance" "workers" {
      + ami                                  = "ami-0b2aae5f4283c0df2"
      + arn                                  = (known after apply)
      + associate_public_ip_address          = (known after apply)
      + availability_zone                    = (known after apply)
      + cpu_core_count                       = (known after apply)
      + cpu_threads_per_core                 = (known after apply)
      + disable_api_stop                     = (known after apply)
      + disable_api_termination              = (known after apply)
      + ebs_optimized                        = (known after apply)
      + enable_primary_ipv6                  = (known after apply)
      + get_password_data                    = false
      + host_id                              = (known after apply)
      + host_resource_group_arn              = (known after apply)
      + iam_instance_profile                 = (known after apply)
      + id                                   = (known after apply)
      + instance_initiated_shutdown_behavior = (known after apply)
      + instance_lifecycle                   = (known after apply)
      + instance_state                       = (known after apply)
      + instance_type                        = "r6i.2xlarge"
      + ipv6_address_count                   = (known after apply)
      + ipv6_addresses                       = (known after apply)
      + key_name                             = (known after apply)
      + monitoring                           = (known after apply)
      + outpost_arn                          = (known after apply)
      + password_data                        = (known after apply)
      + placement_group                      = (known after apply)
      + placement_partition_number           = (known after apply)
      + primary_network_interface_id         = (known after apply)
      + private_dns                          = (known after apply)
      + private_ip                           = (known after apply)
      + public_dns                           = (known after apply)
      + public_ip                            = (known after apply)
      + secondary_private_ips                = (known after apply)
      + security_groups                      = (known after apply)
      + source_dest_check                    = true
      + spot_instance_request_id             = (known after apply)
      + subnet_id                            = "subnet-02c930351cde1e9c3"
      + tags                                 = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-worker-1-prod"
          + "Role"        = "worker"
          + "Service"     = "citus"
          + "WorkerIndex" = "1"
        }
      + tags_all                             = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-worker-1-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
          + "Role"        = "worker"
          + "Service"     = "citus"
          + "WorkerIndex" = "1"
        }
      + tenancy                              = (known after apply)
      + user_data                            = "7b4bd87c9982aab7fa463c8d12e99399661f8bde"
      + user_data_base64                     = (known after apply)
      + user_data_replace_on_change          = false
      + vpc_security_group_ids               = (known after apply)

      + ebs_block_device {
          + delete_on_termination = false
          + device_name           = "/dev/sdf"
          + encrypted             = true
          + iops                  = 3000
          + kms_key_id            = (known after apply)
          + snapshot_id           = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-worker-1-prod-data"
            }
          + tags_all              = (known after apply)
          + throughput            = 125
          + volume_id             = (known after apply)
          + volume_size           = 500
          + volume_type           = "gp3"
        }

      + root_block_device {
          + delete_on_termination = false
          + device_name           = (known after apply)
          + encrypted             = true
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + tags                  = {
              + "Backup"      = "Required"
              + "CostCenter"  = "Platform"
              + "Environment" = "prod"
              + "Name"        = "citus-worker-1-prod-root"
            }
          + tags_all              = (known after apply)
          + throughput            = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = 100
          + volume_type           = "gp3"
        }
    }

  # module.citus_cluster.aws_security_group.citus will be created
  + resource "aws_security_group" "citus" {
      + arn                    = (known after apply)
      + description            = "Security group for Citus cluster"
      + egress                 = [
          + {
              + cidr_blocks      = [
                  + "0.0.0.0/0",
                ]
              + description      = "Allow all outbound"
              + from_port        = 0
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "-1"
              + security_groups  = []
              + self             = false
              + to_port          = 0
            },
        ]
      + id                     = (known after apply)
      + ingress                = [
          + {
              + cidr_blocks      = [
                  + "10.0.0.0/8",
                ]
              + description      = "PgBouncer access"
              + from_port        = 6432
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 6432
            },
          + {
              + cidr_blocks      = [
                  + "10.0.0.0/8",
                ]
              + description      = "PostgreSQL access"
              + from_port        = 5432
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 5432
            },
          + {
              + cidr_blocks      = [
                  + "10.0.0.0/8",
                ]
              + description      = "SSH access"
              + from_port        = 22
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 22
            },
          + {
              + cidr_blocks      = []
              + description      = "Internal cluster communication"
              + from_port        = 0
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = true
              + to_port          = 65535
            },
        ]
      + name                   = (known after apply)
      + name_prefix            = "citus-prod-"
      + owner_id               = (known after apply)
      + revoke_rules_on_delete = false
      + tags                   = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "Name"        = "citus-prod"
          + "Service"     = "citus"
        }
      + tags_all               = {
          + "Backup"      = "Required"
          + "CostCenter"  = "Platform"
          + "Environment" = "prod"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "citus-prod"
          + "Owner"       = "Platform Team"
          + "Project"     = "FastOrder"
          + "Service"     = "citus"
        }
      + vpc_id                 = "vpc-0af7da1e7d94d62bd"
    }

Plan: 9 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + connection_string = (sensitive value)
  + coordinator_ip    = (known after apply)
  + worker_ips        = [
      + (known after apply),
      + (known after apply),
    ]

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"
[2026-01-03_07:28:00] ✓ Terraform plan succeeded
[2026-01-03_07:28:00] Generating plan JSON...
[2026-01-03_07:28:05] ✓ Terraform provisioning step completed successfully

Next step: Review the plan and apply with 'terraform apply tfplan'

[INFO] FastOrder Environment Preparation
[INFO] Service: identity
[INFO] Zone: sau
[INFO] Environment: dev
[INFO] Branch: main
[INFO] State Directory: /opt/fastorder/bash/scripts/env_app_setup/state
[INFO] Library: /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] IP: 142.93.238.16 (specified)

[INFO] Creating environment using fo-env...
[INFO] Creating new FastOrder environment (v1 topology)
[INFO] Generated environment ID: identity-sau-main-dev
[INFO] Using provided IP: 142.93.238.16
[INFO] Allocated interface: eth0:16
[INFO] Configuring network interface for VM IP: 142.93.238.16
[INFO] VM IP 142.93.238.16 is already configured on eth0:16
[CONFIG] No web configuration found for environment: identity-sau-main-dev
[CONFIG] Using defaults: ES_NODES=1, PG_WORKERS=1
[INFO] Service enabled flags: db=yes, eventbus=yes, search=yes
[ OK ] Created topology.json at /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
[ OK ] Generated overlay configurations in /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/generated/
[ OK ] Updated environments.json
[ OK ] Updated setup.json
[ OK ] Environment created successfully!
[INFO] 
[INFO] Environment Details:
[INFO]   ID: identity-sau-main-dev
[INFO]   Service: identity
[INFO]   zone: sau
[INFO]   Environment: dev
[INFO]   Branch: main
[INFO]   IP: 142.93.238.16
[INFO]   Interface: eth0:16
[INFO] 
[INFO] Configuration files:
[INFO]   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
[INFO]   Generated: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/generated/*.env
[INFO]   Overrides: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/overrides/*.env
[INFO] 
[INFO] To use this environment:
[INFO]   export ENV_ID="identity-sau-main-dev"
[INFO]   source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO]   init_environment
[ OK ] Environment preparation completed successfully!
[INFO] Creating topology from web form submission...
[INFO] Loaded from topology.json: identity-sau-main-dev
[2026-01-03 07:28:08] Loaded environment: identity-sau-main-dev
[2026-01-03 07:28:08] Service: identity, Zone: sau, Branch: main, Env: dev
[2026-01-03 07:28:08] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-03 07:28:08] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-03 07:28:08] PostgreSQL HA Nodes: 1, Citus Enabled: yes
[ OK ] Environment initialized successfully (mode: general)
[INFO] Creating topology.json from web form submission...
[INFO] DEBUG: Service enabled flags...
[INFO]   DB_ENABLED=yes
[INFO]   EVENTBUS_ENABLED=yes
[INFO]   SEARCH_ENABLED=yes
[INFO] DEBUG: Checking for form submission variables...
[INFO]   service_es_ip=10.100.1.4
[INFO]   service_es_fqdn=search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com
[INFO]   service_pg_coordinator_ip=10.100.1.14
[WARN] IP 10.100.1.4 is already allocated, allocating new IP for search
[INFO] Adding search: search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.182) [reallocated from 10.100.1.4]
[WARN] IP 10.100.1.6 is already allocated, allocating new IP for search-node-01
[INFO] Adding search-node-01: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.186) [reallocated from 10.100.1.6]
[WARN] IP 10.100.1.8 is already allocated, allocating new IP for eventbus-broker-01
[INFO] Adding eventbus-broker-01: eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com (10.100.1.199) [reallocated from 10.100.1.8]
[WARN] IP 10.100.1.10 is already allocated, allocating new IP for eventbus-connect
[INFO] Adding eventbus-connect: eventbus-identity-sau-main-dev-kafka-connect.fastorder.com (10.100.1.201) [reallocated from 10.100.1.10]
[WARN] IP 10.100.1.12 is already allocated, allocating new IP for schema-registry
[INFO] Adding schema-registry: schema-identity-sau-main-dev-kafka-registry.fastorder.com (10.100.1.202) [reallocated from 10.100.1.12]
[WARN] IP 10.100.1.14 is already allocated, allocating new IP for pg-coordinator
[INFO] Adding pg-coordinator: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com (10.100.1.203) [reallocated from 10.100.1.14]
[WARN] IP 10.100.1.16 is already allocated, allocating new IP for pgbouncer
[INFO] Adding pgbouncer: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com (10.100.1.204) [reallocated from 10.100.1.16]
[WARN] IP 10.100.1.18 is already allocated, allocating new IP for obs
[INFO] Adding obs: obs-identity-sau-main-dev.fastorder.com (10.100.1.205) [reallocated from 10.100.1.18]
[ OK ] Topology created from form data
[INFO] Applications registered:
  ✓ eventbus-broker-01: eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com (10.100.1.199)
  ✓ eventbus-connect: eventbus-identity-sau-main-dev-kafka-connect.fastorder.com (10.100.1.201)
  ✓ obs: obs-identity-sau-main-dev.fastorder.com (10.100.1.205)
  ✓ pg-coordinator: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com (10.100.1.203)
  ✓ pgbouncer: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com (10.100.1.204)
  ✓ schema-registry: schema-identity-sau-main-dev-kafka-registry.fastorder.com (10.100.1.202)
  ✓ search: search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.182)
  ✓ search-node-01: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.186)
[ OK ] Topology created from form data

[INFO] Next steps:
[INFO] 1. Review the generated topology.json and configurations
[INFO] 2. Customize overrides/*.env files if needed
[INFO] 3. Run subsequent installation steps (02-install-postgresql, etc.)

[INFO] To use this environment in other scripts:
[INFO]   export ENV_ID="$(fo-env list | tail -n1 | awk '{print $1}')"
[INFO]   source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO]   init_environment

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🚀 OBSERVABILITY CELL PROVISIONING STARTED
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Script: 02-observability-cell/run.sh
[INFO] Timestamp: 2026-01-03 07:28:19 UTC
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Ensuring correct permissions for observability deployment...
[2026-01-03 07:28:19 UTC] USER=www-data EUID=0 PID=2948403 ACTION=fsop ARGS=chmod 775 /var/log/fastorder
[2026-01-03 07:28:19 UTC] USER=www-data EUID=0 PID=2948412 ACTION=fsop ARGS=chown www-data:www-data /var/log/fastorder
[2026-01-03 07:28:19 UTC] USER=www-data EUID=0 PID=2948421 ACTION=fsop ARGS=touch /var/log/fastorder/provisioning-elevated.log
[2026-01-03 07:28:19 UTC] USER=www-data EUID=0 PID=2948430 ACTION=fsop ARGS=chmod 666 /var/log/fastorder/provisioning-elevated.log
[2026-01-03 07:28:19 UTC] USER=www-data EUID=0 PID=2948439 ACTION=fsop ARGS=chown www-data:www-data /var/log/fastorder/provisioning-elevated.log
[OK]   Log directory: /var/log/fastorder (775)
[OK]   Log file: provisioning-elevated.log (666)
[2026-01-03 07:28:19 UTC] USER=www-data EUID=0 PID=2948448 ACTION=fsop ARGS=chmod 775 /opt/fastorder/bash/scripts/env_app_setup/state
[OK]   State directory: 775
[2026-01-03 07:28:19 UTC] USER=www-data EUID=0 PID=2948457 ACTION=fsop ARGS=mkdir -p /etc/fastorder/observability/certs
[2026-01-03 07:28:19 UTC] USER=www-data EUID=0 PID=2948466 ACTION=fsop ARGS=chmod 750 /etc/fastorder/observability/certs
[OK]   Cert directory: /etc/fastorder/observability/certs (750 - secure)
[OK]   Lib scripts: executable (755)
[OK]   All deployment scripts: executable (755)
[OK]   All directories: accessible (755)
[OK]   ✅ All permissions verified and fixed
[CREDS] Using AWS credentials from: /var/www/.aws/credentials
[CREDS] Credential management library loaded (region: me-central-1)
[INFO] Using web-provided environment: identity-sau-main-dev
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
═══════════════════════════════════════════════════════════════════════════════
OBSERVABILITY CELL PROVISIONING
═══════════════════════════════════════════════════════════════════════════════
[INFO] Application Cell: identity-sau-main-dev
[INFO] Observability Cell: obs-identity-sau-main-dev
[INFO] Service: identity | Zone: sau | Env: dev

[INFO] Step 1/10: Provisioning network infrastructure...
[INFO]   Using existing IP for obs: 10.100.1.205
[INFO]   Allocated new IP for metrics: 10.100.1.206
[2026-01-03 07:28:21 UTC] USER=www-data EUID=0 PID=2948940 ACTION=fsop ARGS=cp /tmp/tmp.K4p4kYVkGr /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
[INFO]   Allocated new IP for dashboards: 10.100.1.207
[2026-01-03 07:28:21 UTC] USER=www-data EUID=0 PID=2948957 ACTION=fsop ARGS=cp /tmp/tmp.FOpY5rgZLZ /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
[INFO]   Allocated new IP for logstore: 10.100.1.208
[2026-01-03 07:28:21 UTC] USER=www-data EUID=0 PID=2948974 ACTION=fsop ARGS=cp /tmp/tmp.buX0P0wR1g /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
[INFO]   Allocated new IP for traces: 10.100.1.209
[2026-01-03 07:28:21 UTC] USER=www-data EUID=0 PID=2948991 ACTION=fsop ARGS=cp /tmp/tmp.xNiPqChZKr /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
[INFO]   Allocated new IP for alerts: 10.100.1.210
[2026-01-03 07:28:22 UTC] USER=www-data EUID=0 PID=2949009 ACTION=fsop ARGS=cp /tmp/tmp.VPfZzm6Mxe /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
[INFO]   Allocated new IP for telemetry: 10.100.1.211
[2026-01-03 07:28:22 UTC] USER=www-data EUID=0 PID=2949026 ACTION=fsop ARGS=cp /tmp/tmp.EsUxaxXDTT /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
[INFO]   Allocated observability IPs:
[INFO]     metrics: 10.100.1.206
[INFO]     alerts: 10.100.1.210
[INFO]     dashboards: 10.100.1.207
[INFO]     traces: 10.100.1.209
[INFO]     telemetry: 10.100.1.211
[INFO]     logstore: 10.100.1.208
[INFO]     proxy: 10.100.1.205
[INFO]     obs: 10.100.1.205
[ OK ] Network infrastructure allocated
[INFO] Cleaning up ports from previous environments...
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-identity-sau-main-dev
[INFO] IP Address: 10.100.1.205
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Found 9 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.205...

[INFO] Scanning 15 ports...

[INFO]   ⚠️  NodeExporter: 10.100.1.205:9100 - OCCUPIED

[WARN] Found 1 occupied port(s) out of 15 total
[WARN] Will attempt to free occupied ports...

[OK]   Port 10.100.1.205:9100 occupied but service obs-identity-sau-main-dev is running (OK - idempotent)

[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Port Cleanup Summary for obs-identity-sau-main-dev
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Total ports checked:  15
[INFO] Already free:         14
[INFO] Occupied (cleaned):   1
[OK]   Successfully freed:   1

[OK]   ✅ All ports are now FREE - ready for installation

[OK]   Port cleanup completed successfully
[INFO] Configuring IP aliases on network interface...
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING NETWORK IP ALIASES
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Interface: lo
[INFO] IP Count: 8

[INFO] Configuring: metrics → 10.100.1.206
[INFO]   IP 10.100.1.206 already configured on network interface
[INFO] Configuring: alerts → 10.100.1.210
[INFO]   IP 10.100.1.210 already configured on network interface
[INFO] Configuring: dashboards → 10.100.1.207
[INFO]   IP 10.100.1.207 already configured on network interface
[INFO] Configuring: traces → 10.100.1.209
[INFO]   IP 10.100.1.209 already configured on network interface
[INFO] Configuring: telemetry → 10.100.1.211
[INFO]   IP 10.100.1.211 already configured on network interface
[INFO] Configuring: logstore → 10.100.1.208
[INFO]   IP 10.100.1.208 already configured on network interface
[INFO] Configuring: proxy → 10.100.1.205
[INFO]   IP 10.100.1.205 already configured on network interface
[INFO] Configuring: obs → 10.100.1.205
[INFO]   IP 10.100.1.205 already configured on network interface

[OK]   ═══════════════════════════════════════════════════════════════
[OK]   ✅ All IP aliases configured successfully
[OK]   ═══════════════════════════════════════════════════════════════
[INFO] Current IP configuration on lo:
      inet 127.0.0.1/8 scope host lo
      inet 10.100.60.2/32 scope global lo
      inet 10.100.1.182/32 scope global lo
      inet 10.100.1.187/32 scope global lo
      inet 10.100.1.183/32 scope global lo
      inet 10.100.1.186/32 scope global lo
      inet 10.100.1.188/32 scope global lo
      inet 10.100.1.184/32 scope global lo
      inet 10.100.1.181/32 scope global lo
      inet 10.100.1.192/32 scope global lo:pgbouncer
      inet 10.100.1.193/32 scope global lo
      inet 10.100.1.197/32 scope global lo
      inet 10.100.1.194/32 scope global lo
      inet 10.100.1.196/32 scope global lo
      inet 10.100.1.198/32 scope global lo
      inet 10.100.1.195/32 scope global lo
      inet 10.100.1.180/32 scope global lo
      inet 10.100.1.179/32 scope global lo
      inet 10.100.1.205/32 scope global lo
      inet 10.100.1.209/32 scope global lo
      inet 10.100.1.206/32 scope global lo
      inet 10.100.1.208/32 scope global lo
      inet 10.100.1.210/32 scope global lo
      inet 10.100.1.207/32 scope global lo
      inet 10.100.1.51/32 scope global lo
      inet 10.100.1.103/32 scope global lo
      inet 10.100.1.204/32 scope global lo:pgbouncer

[OK]   IP aliases configured on network interface
[INFO] Step 2/10: Creating DNS entries...
[INFO] Configuring DNS entries in /etc/hosts...
[INFO]   Added: metrics-identity-sau-main-dev-prometheus.fastorder.com → 10.100.1.206
[INFO]   Added: alerts-identity-sau-main-dev-alertmanager.fastorder.com → 10.100.1.210
[INFO]   Added: dashboards-identity-sau-main-dev-grafana.fastorder.com → 10.100.1.207
[INFO]   Added: traces-identity-sau-main-dev-tempo.fastorder.com → 10.100.1.209
[INFO]   Added: telemetry-identity-sau-main-dev-opentelemetry.fastorder.com → 10.100.1.211
[INFO]   Added: logstore-identity-sau-main-dev-clickhouse.fastorder.com → 10.100.1.208
[INFO]   Added: observe-identity-sau-main-dev.fastorder.com → 10.100.1.205
[INFO] Adding observability integration aliases...
[INFO]   Added alias: metrics-identity-sau-main-dev.fastorder.com → 10.100.1.206
[INFO]   Added alias: alerts-identity-sau-main-dev.fastorder.com → 10.100.1.210
[INFO]   Added alias: dashboards-identity-sau-main-dev.fastorder.com → 10.100.1.207
[INFO]   Added alias: traces-identity-sau-main-dev.fastorder.com → 10.100.1.209
[INFO]   Added alias: telemetry-identity-sau-main-dev.fastorder.com → 10.100.1.211
[INFO]   Added alias: logstore-identity-sau-main-dev.fastorder.com → 10.100.1.208
[2026-01-03 07:28:23 UTC] USER=www-data EUID=0 PID=2949373 ACTION=fsop ARGS=sed -i /observe-identity-sau-main-dev.fastorder.com/d /etc/hosts
[INFO]   Added alias: observe-identity-sau-main-dev.fastorder.com → 10.100.1.205
[OK]   DNS entries created
[INFO] Step 3/10: Creating AWS Secrets Manager structure...
[INFO] Creating AWS Secrets Manager structure
[INFO]   Base path: fastorder/observability/identity/sau/dev
[INFO]   Observability Cell: obs-identity-sau-main-dev
[INFO]   Application Cell: identity-sau-main-dev
[INFO]   Exists: fastorder/observability/identity/sau/dev/metrics
[INFO]   Exists: fastorder/observability/identity/sau/dev/dashboards
[INFO]   Exists: fastorder/observability/identity/sau/dev/logstore
[INFO]   Exists: fastorder/observability/identity/sau/dev/traces
[INFO]   Exists: fastorder/observability/identity/sau/dev/telemetry
[INFO]   Exists: fastorder/observability/identity/sau/dev/alerts
[INFO] Secrets structure created successfully
[OK]   Secrets structure created
[INFO] Step 4/10: Generating mTLS certificates...
[INFO] Generating mTLS certificates for observability cell
[INFO]   Observability Cell: obs-identity-sau-main-dev
[INFO]   Components: prometheus,grafana,loki,tempo,otlp_collector,clickhouse,alertmanager
[INFO]   Creating certificate directory: /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[2026-01-03 07:28:34 UTC] USER=www-data EUID=0 PID=2949429 ACTION=fsop ARGS=mkdir -p /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[2026-01-03 07:28:34 UTC] USER=www-data EUID=0 PID=2949438 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[INFO]   Generating CA certificate for obs-identity-sau-main-dev
[2026-01-03 07:28:34 UTC] USER=www-data EUID=0 PID=2949447 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-key.pem 4096
[2026-01-03 07:28:35 UTC] USER=www-data EUID=0 PID=2949457 ACTION=fsop ARGS=openssl req -new -x509 -days 3650 -key /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-key.pem -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=obs-identity-sau-main-dev-ca
[2026-01-03 07:28:35 UTC] USER=www-data EUID=0 PID=2949466 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-key.pem
[2026-01-03 07:28:35 UTC] USER=www-data EUID=0 PID=2949475 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem
[INFO]   CA certificate created: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem
[INFO]   Generating certificate for: prometheus
[2026-01-03 07:28:35 UTC] USER=www-data EUID=0 PID=2949484 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/prometheus-key.pem 2048
[2026-01-03 07:28:36 UTC] USER=www-data EUID=0 PID=2949496 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-sau-main-dev/prometheus-key.pem -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/prometheus-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=prometheus.obs-identity-sau-main-dev
[2026-01-03 07:28:36 UTC] USER=www-data EUID=0 PID=2949505 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-sau-main-dev/prometheus-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/prometheus-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = prometheus.obs-identity-sau-main-dev
[2026-01-03 07:28:36 UTC] USER=www-data EUID=0 PID=2949514 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/prometheus-key.pem
[2026-01-03 07:28:36 UTC] USER=www-data EUID=0 PID=2949523 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/prometheus-cert.pem
[2026-01-03 07:28:36 UTC] USER=www-data EUID=0 PID=2949532 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-sau-main-dev/prometheus-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/prometheus-cert.pem
[INFO]   Generating certificate for: grafana
[2026-01-03 07:28:36 UTC] USER=www-data EUID=0 PID=2949541 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/grafana-key.pem 2048
[2026-01-03 07:28:37 UTC] USER=www-data EUID=0 PID=2949550 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-sau-main-dev/grafana-key.pem -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/grafana-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=grafana.obs-identity-sau-main-dev
[2026-01-03 07:28:37 UTC] USER=www-data EUID=0 PID=2949559 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-sau-main-dev/grafana-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/grafana-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = grafana.obs-identity-sau-main-dev
[2026-01-03 07:28:37 UTC] USER=www-data EUID=0 PID=2949568 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/grafana-key.pem
[2026-01-03 07:28:37 UTC] USER=www-data EUID=0 PID=2949577 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/grafana-cert.pem
[2026-01-03 07:28:37 UTC] USER=www-data EUID=0 PID=2949586 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-sau-main-dev/grafana-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/grafana-cert.pem
[INFO]   Generating certificate for: loki
[2026-01-03 07:28:37 UTC] USER=www-data EUID=0 PID=2949595 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/loki-key.pem 2048
[2026-01-03 07:28:37 UTC] USER=www-data EUID=0 PID=2949604 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-sau-main-dev/loki-key.pem -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/loki-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=loki.obs-identity-sau-main-dev
[2026-01-03 07:28:37 UTC] USER=www-data EUID=0 PID=2949613 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-sau-main-dev/loki-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/loki-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = loki.obs-identity-sau-main-dev
[2026-01-03 07:28:37 UTC] USER=www-data EUID=0 PID=2949622 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/loki-key.pem
[2026-01-03 07:28:37 UTC] USER=www-data EUID=0 PID=2949631 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/loki-cert.pem
[2026-01-03 07:28:37 UTC] USER=www-data EUID=0 PID=2949640 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-sau-main-dev/loki-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/loki-cert.pem
[INFO]   Generating certificate for: tempo
[2026-01-03 07:28:37 UTC] USER=www-data EUID=0 PID=2949649 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/tempo-key.pem 2048
[2026-01-03 07:28:37 UTC] USER=www-data EUID=0 PID=2949658 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-sau-main-dev/tempo-key.pem -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/tempo-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=tempo.obs-identity-sau-main-dev
[2026-01-03 07:28:37 UTC] USER=www-data EUID=0 PID=2949667 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-sau-main-dev/tempo-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/tempo-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = tempo.obs-identity-sau-main-dev
[2026-01-03 07:28:37 UTC] USER=www-data EUID=0 PID=2949676 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/tempo-key.pem
[2026-01-03 07:28:37 UTC] USER=www-data EUID=0 PID=2949685 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/tempo-cert.pem
[2026-01-03 07:28:37 UTC] USER=www-data EUID=0 PID=2949694 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-sau-main-dev/tempo-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/tempo-cert.pem
[INFO]   Generating certificate for: otlp_collector
[2026-01-03 07:28:38 UTC] USER=www-data EUID=0 PID=2949703 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/otlp_collector-key.pem 2048
[2026-01-03 07:28:38 UTC] USER=www-data EUID=0 PID=2949712 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-sau-main-dev/otlp_collector-key.pem -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/otlp_collector-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=otlp_collector.obs-identity-sau-main-dev
[2026-01-03 07:28:38 UTC] USER=www-data EUID=0 PID=2949721 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-sau-main-dev/otlp_collector-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/otlp_collector-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = otlp_collector.obs-identity-sau-main-dev
[2026-01-03 07:28:38 UTC] USER=www-data EUID=0 PID=2949730 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/otlp_collector-key.pem
[2026-01-03 07:28:38 UTC] USER=www-data EUID=0 PID=2949739 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/otlp_collector-cert.pem
[2026-01-03 07:28:38 UTC] USER=www-data EUID=0 PID=2949748 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-sau-main-dev/otlp_collector-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/otlp_collector-cert.pem
[INFO]   Generating certificate for: clickhouse
[2026-01-03 07:28:38 UTC] USER=www-data EUID=0 PID=2949757 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/clickhouse-key.pem 2048
[2026-01-03 07:28:38 UTC] USER=www-data EUID=0 PID=2949766 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-sau-main-dev/clickhouse-key.pem -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/clickhouse-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=clickhouse.obs-identity-sau-main-dev
[2026-01-03 07:28:38 UTC] USER=www-data EUID=0 PID=2949775 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-sau-main-dev/clickhouse-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/clickhouse-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = clickhouse.obs-identity-sau-main-dev
[2026-01-03 07:28:38 UTC] USER=www-data EUID=0 PID=2949784 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/clickhouse-key.pem
[2026-01-03 07:28:38 UTC] USER=www-data EUID=0 PID=2949793 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/clickhouse-cert.pem
[2026-01-03 07:28:38 UTC] USER=www-data EUID=0 PID=2949802 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-sau-main-dev/clickhouse-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/clickhouse-cert.pem
[INFO]   Generating certificate for: alertmanager
[2026-01-03 07:28:38 UTC] USER=www-data EUID=0 PID=2949811 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/alertmanager-key.pem 2048
[2026-01-03 07:28:38 UTC] USER=www-data EUID=0 PID=2949820 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-sau-main-dev/alertmanager-key.pem -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/alertmanager-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=alertmanager.obs-identity-sau-main-dev
[2026-01-03 07:28:38 UTC] USER=www-data EUID=0 PID=2949829 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-sau-main-dev/alertmanager-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/alertmanager-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = alertmanager.obs-identity-sau-main-dev
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2949838 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/alertmanager-key.pem
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2949847 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/alertmanager-cert.pem
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2949856 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-sau-main-dev/alertmanager-csr.pem
[INFO]   Certificate created: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/alertmanager-cert.pem
[INFO]   Generating PHP client certificate for metrics service...
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2949865 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/php-client-key.pem 2048
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2949874 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-sau-main-dev/php-client-key.pem -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/php-client-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Dashboard/CN=php-metrics-client.obs-identity-sau-main-dev
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2949883 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-sau-main-dev/php-client-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/php-client-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Dashboard, CN = php-metrics-client.obs-identity-sau-main-dev
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2949892 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/php-client-key.pem
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2949901 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/php-client-cert.pem
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2949910 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-identity-sau-main-dev/php-client-key.pem
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2949919 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-identity-sau-main-dev/php-client-cert.pem
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2949928 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-sau-main-dev/php-client-csr.pem
[INFO]   PHP client certificate created: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/php-client-cert.pem
[INFO]   Generating Apache client certificate for mTLS reverse proxy...
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2949937 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/apache-client-key.pem 2048
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2949946 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-identity-sau-main-dev/apache-client-key.pem -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/apache-client-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=ReverseProxy/CN=apache-proxy.obs-identity-sau-main-dev
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2949955 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-identity-sau-main-dev/apache-client-csr.pem -CA /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-identity-sau-main-dev/apache-client-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = ReverseProxy, CN = apache-proxy.obs-identity-sau-main-dev
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2949982 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/apache-client-key.pem
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2949991 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/apache-client-combined.pem
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2950000 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/apache-client-cert.pem
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2950009 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-identity-sau-main-dev/apache-client-key.pem
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2950018 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-identity-sau-main-dev/apache-client-cert.pem
[2026-01-03 07:28:39 UTC] USER=www-data EUID=0 PID=2950027 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-identity-sau-main-dev/apache-client-combined.pem
[2026-01-03 07:28:40 UTC] USER=www-data EUID=0 PID=2950036 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-identity-sau-main-dev/apache-client-csr.pem
[INFO]   Apache client certificate created: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/apache-client-cert.pem
[INFO]   Apache combined cert+key: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/apache-client-combined.pem
[INFO]   Storing mTLS certificates in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/identity/sau/main/dev/mtls/php-client-qgBJOa",
    "Name": "fastorder/observability/identity/sau/main/dev/mtls/php-client",
    "VersionId": "6650c393-e276-40e9-8c00-7f1dd9f56db7"
}
[INFO]   mTLS certificates stored in Secrets Manager: fastorder/observability/identity/sau/main/dev/mtls/php-client
[INFO] mTLS certificates generated successfully
[INFO]   Certificate directory: /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[INFO]   PHP client cert: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/php-client-cert.pem
[INFO]   PHP client key: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/php-client-key.pem
[INFO]   Apache client cert: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/apache-client-cert.pem
[INFO]   Apache combined (for SSLProxyMachineCertificateFile): /etc/fastorder/observability/certs/obs-identity-sau-main-dev/apache-client-combined.pem
[OK]   mTLS certificates generated
[INFO] Step 5/10: Deploying log storage backend...
[INFO]   Provider: clickhouse (selected)
[INFO]   Note: Deployed before telemetry (OtelCol depends on log storage)
[INFO]   FQDN: logstore-identity-sau-main-dev-clickhouse.fastorder.com
[INFO]   IP: 10.100.1.208
[INFO] Deploying log backend: clickhouse...
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] LOG STORAGE BACKEND DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: clickhouse
[INFO] Observability Cell: obs-identity-sau-main-dev
[INFO] FQDN: logstore-identity-sau-main-dev-clickhouse.fastorder.com
[INFO] IP: 10.100.1.208
[INFO] S3 Bucket: fastorder-logs-sau-dev
[INFO] Retention: 90 days
[INFO] ═══════════════════════════════════════════════════════════════

[2026-01-03 07:28:44 UTC] USER=unknown EUID=33 PID=2950098 ACTION=fsop ARGS=chmod +x /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh
/bin/chmod: changing permissions of '/opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh': Operation not permitted
[INFO] Using provider: clickhouse
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=identity, ZONE=sau, BRANCH=main, ENV=dev
[INFO] Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-identity-sau-main-dev...
[2026-01-03 07:28:44 UTC] USER=www-data EUID=0 PID=2950115 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-03 07:28:44 UTC] USER=www-data EUID=0 PID=2950124 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-03 07:28:44 UTC] USER=www-data EUID=0 PID=2950133 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-03 07:28:44 UTC] USER=www-data EUID=0 PID=2950142 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-identity-sau-main-dev
[INFO] IP Address: 10.100.1.208
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Found 9 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.208...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[INFO] Binding ClickHouse to allocated IP: 10.100.1.208
[INFO] Deploying ClickHouse for obs-identity-sau-main-dev
[INFO]   FQDN: logstore-identity-sau-main-dev-clickhouse.fastorder.com
[INFO]   Allocated IP: 10.100.1.208
[INFO]   VM IP: 10.100.1.208
[INFO]   Ports: HTTP=8123 TCP=9000 Interserver=9009
[INFO]   S3 Bucket: fastorder-logs-sau-dev (region=me-central-1)
[INFO]   Retention: 90 days
[INFO] Checking if ClickHouse is installed...
[OK]   ClickHouse already installed
[2026-01-03 07:28:44 UTC] USER=www-data EUID=0 PID=2950283 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-server-obs-identity-sau-main-dev/config.d
[2026-01-03 07:28:44 UTC] USER=www-data EUID=0 PID=2950292 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-server-obs-identity-sau-main-dev/users.d
[2026-01-03 07:28:44 UTC] USER=www-data EUID=0 PID=2950301 ACTION=fsop ARGS=mkdir -p /var/lib/clickhouse-obs-identity-sau-main-dev
[2026-01-03 07:28:44 UTC] USER=www-data EUID=0 PID=2950310 ACTION=fsop ARGS=mkdir -p /var/log/clickhouse-server-obs-identity-sau-main-dev
[2026-01-03 07:28:44 UTC] USER=www-data EUID=0 PID=2950319 ACTION=passthru ARGS=chmod 755 /etc/clickhouse-server-obs-identity-sau-main-dev
[2026-01-03 07:28:44 UTC] USER=www-data EUID=0 PID=2950328 ACTION=passthru ARGS=chmod 700 /var/lib/clickhouse-obs-identity-sau-main-dev
[2026-01-03 07:28:44 UTC] USER=www-data EUID=0 PID=2950337 ACTION=passthru ARGS=chmod 750 /var/log/clickhouse-server-obs-identity-sau-main-dev
[INFO] Found existing logs_writer credentials in Secrets Manager - reusing to maintain sync
[INFO] Found existing metrics_reader credentials in Secrets Manager - reusing to maintain sync
[INFO] TLS configuration exported for clickhouse
[INFO]   Cert: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/clickhouse-cert.pem
[INFO]   Key:  /etc/fastorder/observability/certs/obs-identity-sau-main-dev/clickhouse-key.pem
[INFO]   CA:   /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem
[INFO] Configuring certificate permissions for clickhouse (user: clickhouse)
[INFO] Initializing certificate directory for obs-identity-sau-main-dev...
[2026-01-03 07:28:48 UTC] USER=www-data EUID=0 PID=2950378 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-03 07:28:48 UTC] USER=www-data EUID=0 PID=2950387 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-03 07:28:48 UTC] USER=www-data EUID=0 PID=2950396 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-03 07:28:48 UTC] USER=www-data EUID=0 PID=2950405 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[INFO]   Setting file permissions...
[2026-01-03 07:28:48 UTC] USER=www-data EUID=0 PID=2950415 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/clickhouse-cert.pem
[2026-01-03 07:28:48 UTC] USER=www-data EUID=0 PID=2950424 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem
[2026-01-03 07:28:48 UTC] USER=www-data EUID=0 PID=2950433 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/clickhouse-key.pem
[INFO]   Setting file ownership...
[2026-01-03 07:28:48 UTC] USER=www-data EUID=0 PID=2950442 ACTION=passthru ARGS=chown root:clickhouse /etc/fastorder/observability/certs/obs-identity-sau-main-dev/clickhouse-key.pem
[2026-01-03 07:28:48 UTC] USER=www-data EUID=0 PID=2950451 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-identity-sau-main-dev/clickhouse-cert.pem /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for clickhouse
[INFO] Creating ClickHouse configuration...
[2026-01-03 07:28:48 UTC] USER=www-data EUID=0 PID=2950497 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /etc/clickhouse-server-obs-identity-sau-main-dev
[2026-01-03 07:28:48 UTC] USER=www-data EUID=0 PID=2950506 ACTION=passthru ARGS=bash -c chmod 640 /etc/clickhouse-server-obs-identity-sau-main-dev/*.xml
[OK]   ClickHouse configuration created
[INFO] Creating logs table schema...
[2026-01-03 07:28:48 UTC] USER=www-data EUID=0 PID=2950524 ACTION=passthru ARGS=sed -i s/__RETENTION_DAYS__/90/g /etc/clickhouse-server-obs-identity-sau-main-dev/logs_schema.sql
[2026-01-03 07:28:48 UTC] USER=www-data EUID=0 PID=2950533 ACTION=passthru ARGS=chmod 644 /etc/clickhouse-server-obs-identity-sau-main-dev/logs_schema.sql
[OK]   Logs schema created
[INFO] Creating systemd service...
[2026-01-03 07:28:48 UTC] USER=www-data EUID=0 PID=2950551 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /var/lib/clickhouse-obs-identity-sau-main-dev
[2026-01-03 07:28:49 UTC] USER=www-data EUID=0 PID=2950560 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /var/log/clickhouse-server-obs-identity-sau-main-dev
[2026-01-03 07:28:49 UTC] USER=www-data EUID=0 PID=2950569 ACTION=passthru ARGS=chmod 700 /var/lib/clickhouse-obs-identity-sau-main-dev
[OK]   Systemd service created
[INFO] Starting ClickHouse service...
[2026-01-03 07:28:49 UTC] USER=www-data EUID=0 PID=2950578 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-03 07:28:49 UTC] USER=www-data EUID=0 PID=2950623 ACTION=passthru ARGS=systemctl enable clickhouse-server-obs-identity-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/clickhouse-server-obs-identity-sau-main-dev.service -> /etc/systemd/system/clickhouse-server-obs-identity-sau-main-dev.service.
[2026-01-03 07:28:50 UTC] USER=www-data EUID=0 PID=2950668 ACTION=passthru ARGS=systemctl start clickhouse-server-obs-identity-sau-main-dev.service
[INFO] Waiting for ClickHouse to be ready...
[OK]   ClickHouse is ready
[INFO] Initializing database schema...
[OK]   Schema initialized
[INFO] Storing ClickHouse credentials in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/identity/sau/main/dev/clickhouse/server/logs_writer-G4ex4t",
    "Name": "fastorder/observability/identity/sau/main/dev/clickhouse/server/logs_writer",
    "VersionId": "70ccd302-1a3d-4400-bcda-b5c909642a3f"
}
[OK]   logs_writer credentials stored and verified in Secrets Manager
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/identity/sau/main/dev/clickhouse/server/metrics_reader-zM91Q1",
    "Name": "fastorder/observability/identity/sau/main/dev/clickhouse/server/metrics_reader",
    "VersionId": "b10a7d4f-5394-451e-a494-392cf7a83e4b"
}
[OK]   metrics_reader credentials stored and verified in Secrets Manager
[INFO] Validating ClickHouse deployment...
[INFO] ClickHouse version: 25.10.1.3832
[INFO] Tables created: .inner_id.897c5831-9af6-4e7e-9bd2-2aa33a028d1e
.inner_id.a514547c-19f0-46af-be7e-547d14067a3f
application_logs
error_logs_mv
iam_audit_event
metrics_all
otel_logs
request_logs_mv
security_access
[INFO] Test log inserted. Total logs: 1
[OK]   ✅ ClickHouse deployment validated

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ ClickHouse Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO]   FQDN: logstore-identity-sau-main-dev-clickhouse.fastorder.com
[INFO]   IP: 10.100.1.208
[INFO]   HTTP Port: 8123
[INFO]   Native Port: 9000
[INFO]   Database: logs
[INFO]   Retention: 90 days
[INFO]   Storage: Tiered (Local → S3: fastorder-logs-sau-dev in me-central-1)
[INFO] 
[INFO] Credentials stored in AWS Secrets Manager:
[INFO]   Writers: fastorder/observability/identity/sau/main/dev/clickhouse/server/logs_writer
[INFO]   Readers: fastorder/observability/identity/sau/main/dev/clickhouse/server/metrics_reader (for PHP metrics service)
[INFO] 
[INFO] Example queries (using credentials from Secrets Manager):
[INFO]   # Write logs:
[INFO]   clickhouse-client --host logstore-identity-sau-main-dev-clickhouse.fastorder.com --port 9000 --user logs_writer --password '***' --query 'SELECT 1'
[INFO] 
[INFO]   # Read metrics (PHP metrics service):
[INFO]   clickhouse-client --host logstore-identity-sau-main-dev-clickhouse.fastorder.com --port 9000 --user metrics_reader --password '***' --query 'SELECT * FROM system.metrics'
[INFO] 
[INFO] HTTPS Setup (run on web-03/skeleton server):
[INFO]   # Set up HTTPS reverse proxy with Let's Encrypt:
[INFO]   OBS_CELL=obs-identity-sau-main-dev BACKEND_IP=10.100.1.208 sudo bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/../https/setup-clickhouse-https.sh
[INFO] 
[INFO]   # Or add --setup-https flag when running this script
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Log Storage Backend Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: clickhouse
[INFO] FQDN: logstore-identity-sau-main-dev-clickhouse.fastorder.com
[INFO] IP: 10.100.1.208
[INFO] Retention: 90 days
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering ClickHouse in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       ClickHouse
[INFO]   Identifier:        identity-sau-main-dev-clickhouse
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.208
[INFO]   Port:              8443
[INFO]   FQDN:              logstore-identity-sau-main-dev-clickhouse.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ INVALID REQUEST
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] Response: {"success":false,"error":"Invalid JSON: Control character error, possibly incorrectly encoded"}
[ERROR] 
[ERROR] Request payload:
  {
    "env_id": "identity-sau-main-dev",
    "application": "ClickHouse",
    "identifier": "identity-sau-main-dev-clickhouse",
    "identifier_parent": "cluster",
    "ip": "10.100.1.208",
    "port": 8443,
    "fqdn": "logstore-identity-sau-main-dev-clickhouse.fastorder.com",
    "status": "running",
    "meta": {
      "role": "log_storage",
      "provider": "clickhouse",
      "version": "25.10
  1.3832",
      "http_port": 8123,
      "native_port": 9000,
      "https_port": 8443,
      "protocol": "https",
      "metrics_enabled": true,
      "metrics_port": 8123,
      "metrics_path": "/metrics",
      "health_endpoint": "https://logstore-identity-sau-main-dev-clickhouse.fastorder.com/ping",
      "retention_days": 90,
      "s3_bucket": "fastorder-logs-sau-dev"
  }
  }
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[WARN] ⚠️  Failed to register ClickHouse (service is running)
[OK]   clickhouse deployed successfully
[OK]   Log storage backend deployed
[INFO] Step 6/10: Deploying telemetry collector...
[INFO]   Provider: otlp (backend implementation - internal)
[INFO]   Endpoint: telemetry-identity-sau-main-dev-opentelemetry.fastorder.com (stable, exposed to clients)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] TELEMETRY COLLECTOR DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: otlp
[INFO] Observability Cell: obs-identity-sau-main-dev
[INFO] FQDN: telemetry-identity-sau-main-dev-opentelemetry.fastorder.com
[INFO] IP: 10.100.1.211
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: otlp
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Telemetry/provider/otlp.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=identity, ZONE=sau, BRANCH=main, ENV=dev
[INFO] Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-identity-sau-main-dev...
[2026-01-03 07:29:08 UTC] USER=www-data EUID=0 PID=2951677 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-03 07:29:08 UTC] USER=www-data EUID=0 PID=2951686 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-03 07:29:08 UTC] USER=www-data EUID=0 PID=2951695 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-03 07:29:08 UTC] USER=www-data EUID=0 PID=2951704 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-identity-sau-main-dev
[INFO] IP Address: 10.100.1.211
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-identity-sau-main-dev.service belongs to current cell (skipping)
[INFO] Service clickhouse-server-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Found 10 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.211...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[INFO] Binding to allocated IP: 10.100.1.211
[INFO] Deploying OpenTelemetry Collector for observability cell: obs-identity-sau-main-dev
[INFO] FQDN:         telemetry-identity-sau-main-dev-opentelemetry.fastorder.com
[INFO] Allocated IP: 10.100.1.211
[INFO] VM IP:        10.100.1.211
[INFO] Ports:        gRPC=4317 HTTP=4318 Metrics=8888 Prom=8889
[OK]   User 'otelcol' already exists
[INFO] Checking if OpenTelemetry Collector is installed...
[OK]   OpenTelemetry Collector already installed at /usr/local/bin/otelcol-contrib
[INFO] Creating configuration/data directories...
[2026-01-03 07:29:08 UTC] USER=www-data EUID=0 PID=2951851 ACTION=passthru ARGS=mkdir -p /etc/otelcol/obs-identity-sau-main-dev
[2026-01-03 07:29:08 UTC] USER=www-data EUID=0 PID=2951861 ACTION=passthru ARGS=mkdir -p /var/lib/otelcol/obs-identity-sau-main-dev
[2026-01-03 07:29:08 UTC] USER=www-data EUID=0 PID=2951870 ACTION=passthru ARGS=chown -R otelcol:otelcol /etc/otelcol/obs-identity-sau-main-dev /var/lib/otelcol/obs-identity-sau-main-dev
[2026-01-03 07:29:08 UTC] USER=www-data EUID=0 PID=2951879 ACTION=passthru ARGS=chmod 0750 /etc/otelcol/obs-identity-sau-main-dev
[2026-01-03 07:29:08 UTC] USER=www-data EUID=0 PID=2951888 ACTION=passthru ARGS=chmod 0750 /var/lib/otelcol/obs-identity-sau-main-dev
[INFO] Retrieving ClickHouse credentials from Secrets Manager...
[OK]   Retrieved ClickHouse credentials from Secrets Manager
[INFO] Creating OpenTelemetry Collector configuration...
[INFO] ClickHouse exporter enabled: tcp://logstore-identity-sau-main-dev-clickhouse.fastorder.com:9000
[2026-01-03 07:29:10 UTC] USER=www-data EUID=0 PID=2951927 ACTION=passthru ARGS=chown otelcol:otelcol /etc/otelcol/obs-identity-sau-main-dev/config.yaml
[2026-01-03 07:29:10 UTC] USER=www-data EUID=0 PID=2951936 ACTION=passthru ARGS=chmod 0640 /etc/otelcol/obs-identity-sau-main-dev/config.yaml
[OK]   Configuration created at /etc/otelcol/obs-identity-sau-main-dev/config.yaml
[INFO] Setting up TLS certificate permissions...
[INFO] Configuring certificate permissions for otlp_collector (user: otelcol)
[INFO] Initializing certificate directory for obs-identity-sau-main-dev...
[2026-01-03 07:29:10 UTC] USER=www-data EUID=0 PID=2951945 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-03 07:29:10 UTC] USER=www-data EUID=0 PID=2951954 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-03 07:29:10 UTC] USER=www-data EUID=0 PID=2951963 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-03 07:29:10 UTC] USER=www-data EUID=0 PID=2951972 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[INFO]   Setting file permissions...
[2026-01-03 07:29:11 UTC] USER=www-data EUID=0 PID=2951982 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/otlp_collector-cert.pem
[2026-01-03 07:29:11 UTC] USER=www-data EUID=0 PID=2951993 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem
[2026-01-03 07:29:11 UTC] USER=www-data EUID=0 PID=2952002 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/otlp_collector-key.pem
[INFO]   Setting file ownership...
[2026-01-03 07:29:11 UTC] USER=www-data EUID=0 PID=2952011 ACTION=passthru ARGS=chown root:otelcol /etc/fastorder/observability/certs/obs-identity-sau-main-dev/otlp_collector-key.pem
[2026-01-03 07:29:11 UTC] USER=www-data EUID=0 PID=2952020 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-identity-sau-main-dev/otlp_collector-cert.pem /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for otlp_collector
[OK]   Certificate permissions configured
[INFO] Creating systemd service: otelcol-obs-identity-sau-main-dev
[OK]   Systemd service created at /etc/systemd/system/otelcol-obs-identity-sau-main-dev.service
[INFO] Adding /etc/hosts entry for telemetry-identity-sau-main-dev-opentelemetry.fastorder.com -> 10.100.1.211
[2026-01-03 07:29:11 UTC] USER=www-data EUID=0 PID=2952040 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*telemetry-identity-sau-main-dev-opentelemetry.fastorder.com/10.100.1.211    telemetry-identity-sau-main-dev-opentelemetry.fastorder.com/ /etc/hosts
[OK]   Updated /etc/hosts entry to use VM_IP
[INFO] Storing OTLP configuration metadata in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/identity/sau/main/dev/otlp/collector-nBiIgn",
    "Name": "fastorder/observability/identity/sau/main/dev/otlp/collector",
    "VersionId": "579310ab-a674-4f10-9920-6dbb510bae69"
}
[OK]   Configuration metadata stored/updated in AWS Secrets Manager: fastorder/observability/identity/sau/main/dev/otlp/collector
[INFO] Enabling and starting OpenTelemetry Collector service...
[2026-01-03 07:29:13 UTC] USER=www-data EUID=0 PID=2952054 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-03 07:29:13 UTC] USER=www-data EUID=0 PID=2952099 ACTION=passthru ARGS=systemctl enable otelcol-obs-identity-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/otelcol-obs-identity-sau-main-dev.service -> /etc/systemd/system/otelcol-obs-identity-sau-main-dev.service.
[2026-01-03 07:29:14 UTC] USER=www-data EUID=0 PID=2952144 ACTION=passthru ARGS=systemctl restart otelcol-obs-identity-sau-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-01-03 07:29:17 UTC] USER=www-data EUID=0 PID=2952170 ACTION=passthru ARGS=systemctl is-active --quiet otelcol-obs-identity-sau-main-dev.service
[OK]   ✅ OpenTelemetry Collector is running
[OK]   ✅ gRPC endpoint listening on port 4317
[OK]   ✅ HTTP endpoint listening on port 4318
[OK]   ✅ Prometheus metrics endpoint listening on port 8889
[INFO] Service logs (last 10 lines):
[2026-01-03 07:29:17 UTC] USER=www-data EUID=0 PID=2952185 ACTION=passthru ARGS=journalctl -u otelcol-obs-identity-sau-main-dev.service -n 10 --no-pager
Jan 03 07:29:15 web-03 otelcol-obs-identity-sau-main-dev[2952151]: 2026-01-03T07:29:15.541Z        info        internal/resourcedetection.go:125        began detecting resource information        {"kind": "processor", "name": "resourcedetection", "pipeline": "traces"}
Jan 03 07:29:15 web-03 otelcol-obs-identity-sau-main-dev[2952151]: 2026-01-03T07:29:15.543Z        info        system/system.go:201        This attribute changed from int to string. Temporarily switch back to int using the feature gate.        {"kind": "processor", "name": "resourcedetection", "pipeline": "traces", "attribute": "host.cpu.family", "feature gate": "processor.resourcedetection.hostCPUModelAndFamilyAsString"}
Jan 03 07:29:15 web-03 otelcol-obs-identity-sau-main-dev[2952151]: 2026-01-03T07:29:15.545Z        info        system/system.go:220        This attribute changed from int to string. Temporarily switch back to int using the feature gate.        {"kind": "processor", "name": "resourcedetection", "pipeline": "traces", "attribute": "host.cpu.model.id", "feature gate": "processor.resourcedetection.hostCPUModelAndFamilyAsString"}
Jan 03 07:29:15 web-03 otelcol-obs-identity-sau-main-dev[2952151]: 2026-01-03T07:29:15.545Z        info        internal/resourcedetection.go:139        detected resource information        {"kind": "processor", "name": "resourcedetection", "pipeline": "traces", "resource": {"host.name":"web-03","os.type":"linux"}}
Jan 03 07:29:15 web-03 otelcol-obs-identity-sau-main-dev[2952151]: 2026-01-03T07:29:15.652Z        info        prometheusreceiver@v0.91.0/metrics_receiver.go:231        Scrape job added        {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "jobName": "otel-collector"}
Jan 03 07:29:15 web-03 otelcol-obs-identity-sau-main-dev[2952151]: 2026-01-03T07:29:15.652Z        info        prometheusreceiver@v0.91.0/metrics_receiver.go:240        Starting discovery manager        {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
Jan 03 07:29:15 web-03 otelcol-obs-identity-sau-main-dev[2952151]: 2026-01-03T07:29:15.653Z        info        prometheusreceiver@v0.91.0/metrics_receiver.go:282        Starting scrape manager        {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
Jan 03 07:29:15 web-03 otelcol-obs-identity-sau-main-dev[2952151]: 2026-01-03T07:29:15.654Z        info        otlpreceiver@v0.91.0/otlp.go:83        Starting GRPC server        {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "10.100.1.211:4317"}
Jan 03 07:29:15 web-03 otelcol-obs-identity-sau-main-dev[2952151]: 2026-01-03T07:29:15.655Z        info        otlpreceiver@v0.91.0/otlp.go:101        Starting HTTP server        {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "10.100.1.211:4318"}
Jan 03 07:29:15 web-03 otelcol-obs-identity-sau-main-dev[2952151]: 2026-01-03T07:29:15.655Z        info        service@v0.91.0/service.go:171        Everything is ready. Begin running and processing data.

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Telemetry Collector Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: otlp
[INFO] FQDN: telemetry-identity-sau-main-dev-opentelemetry.fastorder.com
[INFO] IP: 10.100.1.211
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering OpenTelemetry Collector in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       OpenTelemetry Collector
[INFO]   Identifier:        identity-sau-main-dev-opentelemetry
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.211
[INFO]   Port:              4317
[INFO]   FQDN:              telemetry-identity-sau-main-dev-opentelemetry.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: f0ffe8a2-dfee-46fa-b427-20c818c9fa66
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✅ OpenTelemetry Collector registered in dashboard
[INFO] Setting up OpenTelemetry Collector metrics collection timer...
[2026-01-03 07:29:18 UTC] USER=www-data EUID=0 PID=2952242 ACTION=passthru ARGS=mv /tmp/otelcol-metrics-identity-sau-main-dev.service /etc/systemd/system/
[2026-01-03 07:29:18 UTC] USER=www-data EUID=0 PID=2952251 ACTION=passthru ARGS=mv /tmp/otelcol-metrics-identity-sau-main-dev.timer /etc/systemd/system/
[2026-01-03 07:29:18 UTC] USER=www-data EUID=0 PID=2952260 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-03 07:29:18 UTC] USER=www-data EUID=0 PID=2952305 ACTION=passthru ARGS=systemctl enable otelcol-metrics-identity-sau-main-dev.timer
[2026-01-03 07:29:19 UTC] USER=www-data EUID=0 PID=2952350 ACTION=passthru ARGS=systemctl start otelcol-metrics-identity-sau-main-dev.timer
[OK]   ✅ Metrics collection timer installed and started
[OK]   Telemetry collector (otlp) deployed successfully

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Step 7/10: METRICS BACKEND DEPLOYMENT
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Provider: prometheus
[INFO]   OBS Cell: obs-identity-sau-main-dev
[INFO]   FQDN: metrics-identity-sau-main-dev-prometheus.fastorder.com
[INFO]   IP: 10.100.1.206
[INFO]   Script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/deploy-metrics.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📊 METRICS DEPLOYMENT WRAPPER STARTED
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Script: deploy-metrics.sh
[INFO] Timestamp: 2026-01-03 07:29:19 UTC
[INFO] Arguments: --provider prometheus --obs-cell obs-identity-sau-main-dev --fqdn metrics-identity-sau-main-dev-prometheus.fastorder.com --ip 10.100.1.206

[INFO] ═══════════════════════════════════════════════════════════════
[INFO] METRICS DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: prometheus
[INFO] Observability Cell: obs-identity-sau-main-dev
[INFO] FQDN: metrics-identity-sau-main-dev-prometheus.fastorder.com
[INFO] IP: 10.100.1.206
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: prometheus
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/provider/prometheus.sh

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/provider/prometheus.sh
[INFO]   OBS_CELL: obs-identity-sau-main-dev
[INFO]   FQDN: metrics-identity-sau-main-dev-prometheus.fastorder.com
[INFO]   IP: 10.100.1.206
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🚀 PROMETHEUS DEPLOYMENT STARTED
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Script: prometheus.sh
[INFO] Timestamp: 2026-01-03 07:29:19 UTC
[INFO] Arguments: --obs-cell obs-identity-sau-main-dev --fqdn metrics-identity-sau-main-dev-prometheus.fastorder.com --ip 10.100.1.206

[INFO] Parsed: SERVICE=identity, ZONE=sau, BRANCH=main, ENV=dev
[INFO] Step 1/12: Sourcing centralized libraries...
[INFO]   Library directory: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/lib
[INFO]   Sourcing port_allocator.sh...
[OK]     ✓ port_allocator.sh loaded
[INFO]   Sourcing cert_permissions.sh...
[OK]     ✓ cert_permissions.sh loaded
[INFO]   Sourcing port_cleanup.sh...
[OK]     ✓ port_cleanup.sh loaded
[OK]   Step 1/12: Libraries sourced successfully

[INFO] Step 2/12: Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-identity-sau-main-dev...
[2026-01-03 07:29:19 UTC] USER=www-data EUID=0 PID=2952376 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-03 07:29:19 UTC] USER=www-data EUID=0 PID=2952385 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-03 07:29:19 UTC] USER=www-data EUID=0 PID=2952394 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-03 07:29:19 UTC] USER=www-data EUID=0 PID=2952403 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-identity-sau-main-dev
[INFO] IP Address: 10.100.1.206
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-identity-sau-main-dev.service belongs to current cell (skipping)
[INFO] Service clickhouse-server-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-identity-sau-main-dev.service belongs to current cell (skipping)
[INFO] Service otelcol-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Found 11 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.206...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[OK]   Step 2/12: Port cleanup completed

[INFO] Step 3/12: Allocating ports...
[OK]   Step 3/12: Ports allocated

[INFO] Step 4/12: Setting up configuration...
[INFO]   Observability cell: obs-identity-sau-main-dev
[INFO]   FQDN: metrics-identity-sau-main-dev-prometheus.fastorder.com
[INFO]   IP: 10.100.1.206
[INFO]   Prometheus Port: 9090
[INFO] Step 5/12: Checking if Prometheus is installed...
[OK]   Prometheus already installed at /usr/local/bin/prometheus
[OK]   Step 5/12: Prometheus binary ready

[INFO] Step 5.1/12: Creating configuration directories early (required for Node Exporter config)...
[INFO]   Config: /etc/prometheus/obs-identity-sau-main-dev
[INFO]   Data: /var/lib/prometheus/obs-identity-sau-main-dev
[INFO]   Rules: /etc/prometheus/obs-identity-sau-main-dev/rules
[2026-01-03 07:29:19 UTC] USER=www-data EUID=0 PID=2952554 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-identity-sau-main-dev
[2026-01-03 07:29:20 UTC] USER=www-data EUID=0 PID=2952563 ACTION=passthru ARGS=mkdir -p /var/lib/prometheus/obs-identity-sau-main-dev
[2026-01-03 07:29:20 UTC] USER=www-data EUID=0 PID=2952572 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-identity-sau-main-dev/rules
[2026-01-03 07:29:20 UTC] USER=www-data EUID=0 PID=2952581 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-identity-sau-main-dev/targets
[OK]   Step 5.1/12: Directories created early

[INFO] Step 6/12: Setting up Node Exporter...
[INFO] Checking if Node Exporter is installed...
[OK]   Node Exporter already installed at /usr/local/bin/node_exporter
[INFO] Creating Node Exporter TLS web config...
[INFO] Creating Node Exporter systemd service with TLS...
[2026-01-03 07:29:20 UTC] USER=www-data EUID=0 PID=2952608 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-03 07:29:20 UTC] USER=www-data EUID=0 PID=2952653 ACTION=passthru ARGS=systemctl enable node_exporter-obs-identity-sau-main-dev.service
[2026-01-03 07:29:21 UTC] USER=www-data EUID=0 PID=2952700 ACTION=passthru ARGS=systemctl restart node_exporter-obs-identity-sau-main-dev.service
[OK]   Step 6/12: Node Exporter ready

[INFO] Step 7/12: Creating configuration directories...
[INFO]   Config: /etc/prometheus/obs-identity-sau-main-dev
[INFO]   Data: /var/lib/prometheus/obs-identity-sau-main-dev
[INFO]   Rules: /etc/prometheus/obs-identity-sau-main-dev/rules
[2026-01-03 07:29:21 UTC] USER=www-data EUID=0 PID=2952714 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-identity-sau-main-dev
[2026-01-03 07:29:21 UTC] USER=www-data EUID=0 PID=2952723 ACTION=passthru ARGS=mkdir -p /var/lib/prometheus/obs-identity-sau-main-dev
[2026-01-03 07:29:21 UTC] USER=www-data EUID=0 PID=2952732 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-identity-sau-main-dev/rules
[OK]   Step 7/12: Directories created

[INFO] Step 8/12: Creating Prometheus configuration...
[INFO] Generated FQDNs:
[INFO]   Prometheus:   metrics-identity-sau-main-dev-prometheus.fastorder.com
[INFO]   Alertmanager: alerts-identity-sau-main-dev-alertmanager.fastorder.com
[INFO]   Grafana:      dashboards-identity-sau-main-dev-grafana.fastorder.com
[INFO]   Otelcol:      telemetry-identity-sau-main-dev-opentelemetry.fastorder.com
[OK]   Step 8/12: Configuration created at /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml

[INFO] Step 9/12: Creating TLS/HTTPS web config...
[OK]   Step 9/12: Web config created at /etc/prometheus/obs-identity-sau-main-dev/web-config.yml
[INFO]   TLS cert: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/prometheus-cert.pem
[INFO]   TLS key: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/prometheus-key.pem
[INFO]   CA cert: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem

[INFO] Creating basic alerting rules...
[OK]   Alerting rules created
[2026-01-03 07:29:21 UTC] USER=www-data EUID=0 PID=2952768 ACTION=passthru ARGS=mkdir -p /etc/prometheus/obs-identity-sau-main-dev/targets
[2026-01-03 07:29:21 UTC] USER=www-data EUID=0 PID=2952777 ACTION=passthru ARGS=bash -c cat > '/etc/prometheus/obs-identity-sau-main-dev/targets/.placeholder.yml' << 'EOF'
# Placeholder file to prevent file_sd_configs warning
# Application targets will be added here automatically
[]
EOF
[INFO] Step 10/12: Creating systemd service...
[INFO]   Service: prometheus-obs-identity-sau-main-dev
[INFO] Binding to: 10.100.1.206:9090
[OK]   Step 10/12: Systemd service created at /etc/systemd/system/prometheus-obs-identity-sau-main-dev.service

[INFO] Step 11/12: Configuring certificate permissions...
[INFO]   Looking for certificates in: /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[OK]     ✓ All certificate files exist
[INFO] Configuring certificate permissions for prometheus (user: root)
[INFO] Initializing certificate directory for obs-identity-sau-main-dev...
[2026-01-03 07:29:21 UTC] USER=www-data EUID=0 PID=2952796 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-03 07:29:21 UTC] USER=www-data EUID=0 PID=2952805 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-03 07:29:21 UTC] USER=www-data EUID=0 PID=2952814 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-03 07:29:21 UTC] USER=www-data EUID=0 PID=2952823 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[INFO]   Setting file permissions...
[2026-01-03 07:29:21 UTC] USER=www-data EUID=0 PID=2952833 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/prometheus-cert.pem
[2026-01-03 07:29:21 UTC] USER=www-data EUID=0 PID=2952843 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem
[2026-01-03 07:29:21 UTC] USER=www-data EUID=0 PID=2952852 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/prometheus-key.pem
[INFO]   Setting file ownership...
[2026-01-03 07:29:22 UTC] USER=www-data EUID=0 PID=2952861 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-identity-sau-main-dev/prometheus-key.pem
[2026-01-03 07:29:22 UTC] USER=www-data EUID=0 PID=2952870 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-identity-sau-main-dev/prometheus-cert.pem /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for prometheus
[OK]   Step 11/12: Certificate permissions configured

[INFO] Adding /etc/hosts entry for metrics-identity-sau-main-dev-prometheus.fastorder.com -> 10.100.1.206
[2026-01-03 07:29:22 UTC] USER=www-data EUID=0 PID=2952881 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*metrics-identity-sau-main-dev-prometheus.fastorder.com/10.100.1.206    metrics-identity-sau-main-dev-prometheus.fastorder.com/ /etc/hosts
[OK]   Updated /etc/hosts entry to use VM_IP
[INFO] Validating Prometheus configuration...
Checking /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml is valid prometheus config file syntax

Checking /etc/prometheus/obs-identity-sau-main-dev/rules/basic_alerts.yml
  SUCCESS: 4 rules found

[OK]   ✅ Configuration is valid
[INFO] Storing Prometheus configuration in AWS Secrets Manager...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/identity/sau/main/dev/prometheus/server-W39WdW",
    "Name": "fastorder/observability/identity/sau/main/dev/prometheus/server",
    "VersionId": "c0591c56-b7ad-41eb-aecd-ca9c0a8e68a2"
}
[OK]   Configuration stored in AWS Secrets Manager
[INFO] Step 12/12: Starting Prometheus service...
[INFO]   Reloading systemd daemon...
[2026-01-03 07:29:24 UTC] USER=www-data EUID=0 PID=2952904 ACTION=passthru ARGS=systemctl daemon-reload
[OK]     ✓ Systemd daemon reloaded
[INFO]   Enabling service...
[2026-01-03 07:29:25 UTC] USER=www-data EUID=0 PID=2952949 ACTION=passthru ARGS=systemctl enable prometheus-obs-identity-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus-obs-identity-sau-main-dev.service -> /etc/systemd/system/prometheus-obs-identity-sau-main-dev.service.
[OK]     ✓ Service enabled
[INFO]   Starting service...
[2026-01-03 07:29:25 UTC] USER=www-data EUID=0 PID=2952994 ACTION=passthru ARGS=systemctl restart prometheus-obs-identity-sau-main-dev.service
[OK]     ✓ Service start command issued

[INFO] Validating Prometheus deployment...
[2026-01-03 07:29:28 UTC] USER=www-data EUID=0 PID=2953015 ACTION=passthru ARGS=systemctl is-active --quiet prometheus-obs-identity-sau-main-dev.service
[OK]   ✅ Prometheus is running
[OK]   ✅ Prometheus web interface listening on port 9090
[OK]   ✅ Prometheus health check passed (HTTPS)
[INFO] ═══════════════════════════════════════════════════════════════
[OK]   Prometheus Web UI: https://metrics-identity-sau-main-dev-prometheus.fastorder.com:9090
[OK]   Targets: https://metrics-identity-sau-main-dev-prometheus.fastorder.com:9090/targets
[OK]   Alerts: https://metrics-identity-sau-main-dev-prometheus.fastorder.com:9090/alerts
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service logs (last 10 lines):
[2026-01-03 07:29:31 UTC] USER=www-data EUID=0 PID=2953028 ACTION=passthru ARGS=journalctl -u prometheus-obs-identity-sau-main-dev.service -n 10 --no-pager
Jan 03 07:29:26 web-03 prometheus-obs-identity-sau-main-dev[2953001]: ts=2026-01-03T07:29:26.337Z caller=tls_config.go:274 level=info component=web msg="Listening on" address=10.100.1.206:9090
Jan 03 07:29:26 web-03 prometheus-obs-identity-sau-main-dev[2953001]: ts=2026-01-03T07:29:26.337Z caller=head.go:761 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
Jan 03 07:29:26 web-03 prometheus-obs-identity-sau-main-dev[2953001]: ts=2026-01-03T07:29:26.337Z caller=head.go:798 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=242.648µs wal_replay_duration=10.804738ms wbl_replay_duration=221ns total_replay_duration=11.096478ms
Jan 03 07:29:26 web-03 prometheus-obs-identity-sau-main-dev[2953001]: ts=2026-01-03T07:29:26.338Z caller=tls_config.go:310 level=info component=web msg="TLS is enabled." http2=true address=10.100.1.206:9090
Jan 03 07:29:26 web-03 prometheus-obs-identity-sau-main-dev[2953001]: ts=2026-01-03T07:29:26.344Z caller=main.go:1045 level=info fs_type=EXT4_SUPER_MAGIC
Jan 03 07:29:26 web-03 prometheus-obs-identity-sau-main-dev[2953001]: ts=2026-01-03T07:29:26.344Z caller=main.go:1048 level=info msg="TSDB started"
Jan 03 07:29:26 web-03 prometheus-obs-identity-sau-main-dev[2953001]: ts=2026-01-03T07:29:26.344Z caller=main.go:1230 level=info msg="Loading configuration file" filename=/etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
Jan 03 07:29:26 web-03 prometheus-obs-identity-sau-main-dev[2953001]: ts=2026-01-03T07:29:26.350Z caller=main.go:1267 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/obs-identity-sau-main-dev/prometheus.yml totalDuration=5.7617ms db_storage=2.727µs remote_storage=2.239µs web_handler=1.064µs query_engine=2.165µs scrape=1.847236ms scrape_sd=113.719µs notify=48.855µs notify_sd=22.188µs rules=2.864263ms tracing=10.106µs
Jan 03 07:29:26 web-03 prometheus-obs-identity-sau-main-dev[2953001]: ts=2026-01-03T07:29:26.350Z caller=main.go:1009 level=info msg="Server is ready to receive web requests."
Jan 03 07:29:26 web-03 prometheus-obs-identity-sau-main-dev[2953001]: ts=2026-01-03T07:29:26.350Z caller=manager.go:1012 level=info component="rule manager" msg="Starting rule manager..."
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Provider script completed with exit code: 0
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Metrics Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: prometheus
[INFO] FQDN: metrics-identity-sau-main-dev-prometheus.fastorder.com
[INFO] IP: 10.100.1.206
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering Prometheus in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Prometheus
[INFO]   Identifier:        identity-sau-main-dev-prometheus
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.206
[INFO]   Port:              9090
[INFO]   FQDN:              metrics-identity-sau-main-dev-prometheus.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: b687453a-0f92-417d-acdc-547bd9f48358
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   Prometheus registered in dashboard

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Step 7/10: METRICS DEPLOYMENT RESULT
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Exit code: 0
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ Metrics backend (prometheus) deployed successfully
[INFO] Step 8/10: Deploying traces backend...
[INFO]   Provider: tempo (selected)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] TRACES DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: tempo
[INFO] Observability Cell: obs-identity-sau-main-dev
[INFO] FQDN: traces-identity-sau-main-dev-tempo.fastorder.com
[INFO] IP: 10.100.1.209
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: tempo
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Traces/provider/tempo.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=identity, ZONE=sau, BRANCH=main, ENV=dev
[INFO] Checking and cleaning ports before installation...
[INFO] Initializing certificate directory for obs-identity-sau-main-dev...
[2026-01-03 07:29:31 UTC] USER=www-data EUID=0 PID=2953082 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-03 07:29:31 UTC] USER=www-data EUID=0 PID=2953091 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-03 07:29:31 UTC] USER=www-data EUID=0 PID=2953100 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-03 07:29:31 UTC] USER=www-data EUID=0 PID=2953109 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Checking and cleaning ports for observability cell: obs-identity-sau-main-dev
[INFO] IP Address: 10.100.1.209
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Checking for conflicting observability services...
[INFO] Service clickhouse-server-obs-identity-sau-main-dev.service belongs to current cell (skipping)
[INFO] Service clickhouse-server-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service otelcol-obs-identity-sau-main-dev.service belongs to current cell (skipping)
[INFO] Service otelcol-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service prometheus-obs-identity-sau-main-dev.service belongs to current cell (skipping)
[INFO] Service prometheus-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service grafana-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service tempo-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Service alertmanager-obs-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[INFO] Found 12 observability service(s) (all belong to current cell)
[INFO] Checking for remaining processes on IP 10.100.1.209...

[INFO] Scanning 15 ports...


[OK]   ✅ All 15 ports are FREE - ready for installation

[OK]   Port cleanup successful on attempt 1
[INFO] Binding Tempo to allocated IP: 10.100.1.209
[INFO] Deploying Grafana Tempo for observability cell: obs-identity-sau-main-dev
[INFO] FQDN: traces-identity-sau-main-dev-tempo.fastorder.com
[INFO] Allocated IP: 10.100.1.209
[INFO] VM IP: 10.100.1.209
[INFO] Ports: HTTP=3200 gRPC=9095, OTLP gRPC=4317, OTLP HTTP=4318
[INFO] Checking if Grafana Tempo is installed...
[OK]   Grafana Tempo already installed at /usr/local/bin/tempo
[INFO] Preparing configuration and data directories...
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953266 ACTION=passthru ARGS=mkdir -p /etc/tempo/obs-identity-sau-main-dev
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953275 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-identity-sau-main-dev
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953284 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-identity-sau-main-dev/wal
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953293 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-identity-sau-main-dev/blocks
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953302 ACTION=passthru ARGS=chown -R tempo:tempo /etc/tempo/obs-identity-sau-main-dev /var/lib/tempo/obs-identity-sau-main-dev
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953311 ACTION=passthru ARGS=chmod 750 /etc/tempo/obs-identity-sau-main-dev /var/lib/tempo/obs-identity-sau-main-dev
[INFO] Creating Grafana Tempo configuration...
[INFO] TLS configuration exported for tempo
[INFO]   Cert: /etc/fastorder/observability/certs/obs-identity-sau-main-dev/tempo-cert.pem
[INFO]   Key:  /etc/fastorder/observability/certs/obs-identity-sau-main-dev/tempo-key.pem
[INFO]   CA:   /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem
[INFO] Setting up certificate permissions for Tempo...
[INFO] Configuring certificate permissions for tempo (user: tempo)
[INFO] Initializing certificate directory for obs-identity-sau-main-dev...
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953326 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953335 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953344 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953353 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[OK]   Certificate directory initialized: /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[INFO]   Setting file permissions...
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953363 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/tempo-cert.pem
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953372 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953381 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/tempo-key.pem
[INFO]   Setting file ownership...
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953390 ACTION=passthru ARGS=chown root:tempo /etc/fastorder/observability/certs/obs-identity-sau-main-dev/tempo-key.pem
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953399 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-identity-sau-main-dev/tempo-cert.pem /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem
[INFO]   Permission configuration completed
[INFO]   (Verification skipped - running via wrapper, trust chmod/chown success)
[OK]   ✅ Certificate permissions configured successfully for tempo
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953418 ACTION=passthru ARGS=chown tempo:tempo /etc/tempo/obs-identity-sau-main-dev/config.yaml
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953427 ACTION=passthru ARGS=chmod 640 /etc/tempo/obs-identity-sau-main-dev/config.yaml
[OK]   Configuration created at /etc/tempo/obs-identity-sau-main-dev/config.yaml
[INFO] Creating systemd service: tempo-obs-identity-sau-main-dev
[OK]   Systemd service created
[INFO] Adding /etc/hosts entry for traces-identity-sau-main-dev-tempo.fastorder.com -> 10.100.1.209
[2026-01-03 07:29:32 UTC] USER=www-data EUID=0 PID=2953446 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*traces-identity-sau-main-dev-tempo.fastorder.com/10.100.1.209    traces-identity-sau-main-dev-tempo.fastorder.com/ /etc/hosts
[OK]   Updated /etc/hosts entry to use VM_IP
[INFO] Storing Tempo configuration in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/identity/sau/main/dev/tempo/server-rvT8sR",
    "Name": "fastorder/observability/identity/sau/main/dev/tempo/server",
    "VersionId": "2ae8b23d-6c7a-426f-a25b-1dc66dce9eb1"
}
[OK]   Tempo configuration stored/updated in AWS Secrets Manager: fastorder/observability/identity/sau/main/dev/tempo/server
[WARN] Port cleanup library not found, skipping automatic cleanup
[INFO] Adding iptables redirect for Tempo internal communication (optional)...
[2026-01-03 07:29:34 UTC] USER=www-data EUID=0 PID=2953463 ACTION=passthru ARGS=iptables -t nat -A OUTPUT -p tcp -d 127.0.0.1 --dport 9095 -j DNAT --to-destination 10.100.1.209:9095
ERROR: passthru not allowed: iptables
[WARN] Could not add iptables redirect (iptables not allowed in wrapper)
[WARN] Tempo will still work - clients should connect to 10.100.1.209:9095 directly
[INFO] Enabling and starting Grafana Tempo service...
[2026-01-03 07:29:34 UTC] USER=www-data EUID=0 PID=2953471 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-03 07:29:35 UTC] USER=www-data EUID=0 PID=2953516 ACTION=passthru ARGS=systemctl enable tempo-obs-identity-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/tempo-obs-identity-sau-main-dev.service -> /etc/systemd/system/tempo-obs-identity-sau-main-dev.service.
[2026-01-03 07:29:36 UTC] USER=www-data EUID=0 PID=2953563 ACTION=passthru ARGS=systemctl restart tempo-obs-identity-sau-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-01-03 07:29:39 UTC] USER=www-data EUID=0 PID=2953583 ACTION=passthru ARGS=systemctl is-active --quiet tempo-obs-identity-sau-main-dev.service
[OK]   ✅ Grafana Tempo is running
[OK]   ✅ HTTP endpoint listening on port 3200
[OK]   ✅ OTLP gRPC endpoint listening on port 4317
[OK]   ✅ OTLP HTTP endpoint listening on port 4318
[INFO] Service logs (last 10 lines):
[2026-01-03 07:29:39 UTC] USER=www-data EUID=0 PID=2953598 ACTION=passthru ARGS=journalctl -u tempo-obs-identity-sau-main-dev.service -n 10 --no-pager
Jan 03 07:29:36 web-03 tempo-obs-identity-sau-main-dev[2953570]: level=info ts=2026-01-03T07:29:36.518217189Z caller=ingester.go:364 msg="beginning wal replay"
Jan 03 07:29:36 web-03 tempo-obs-identity-sau-main-dev[2953570]: level=warn ts=2026-01-03T07:29:36.522912881Z caller=wal.go:94 msg="unowned file entry ignored during wal replay" file=blocks err=null
Jan 03 07:29:36 web-03 tempo-obs-identity-sau-main-dev[2953570]: level=info ts=2026-01-03T07:29:36.523101232Z caller=ingester.go:402 msg="wal replay complete"
Jan 03 07:29:36 web-03 tempo-obs-identity-sau-main-dev[2953570]: level=info ts=2026-01-03T07:29:36.523694352Z caller=ingester.go:416 msg="reloading local blocks" tenants=0
Jan 03 07:29:36 web-03 tempo-obs-identity-sau-main-dev[2953570]: level=info ts=2026-01-03T07:29:36.527019708Z caller=lifecycler.go:624 msg="not loading tokens from file, tokens file path is empty"
Jan 03 07:29:36 web-03 tempo-obs-identity-sau-main-dev[2953570]: ts=2026-01-03T07:29:36Z level=info msg="Starting GRPC server" component=tempo endpoint=10.100.1.209:4317
Jan 03 07:29:36 web-03 tempo-obs-identity-sau-main-dev[2953570]: ts=2026-01-03T07:29:36Z level=info msg="Starting HTTP server" component=tempo endpoint=10.100.1.209:4318
Jan 03 07:29:36 web-03 tempo-obs-identity-sau-main-dev[2953570]: level=info ts=2026-01-03T07:29:36.527918699Z caller=lifecycler.go:649 msg="instance not found in ring, adding with no tokens" ring=ingester
Jan 03 07:29:36 web-03 tempo-obs-identity-sau-main-dev[2953570]: level=info ts=2026-01-03T07:29:36.528415831Z caller=lifecycler.go:493 msg="auto-joining cluster after timeout" ring=ingester
Jan 03 07:29:36 web-03 tempo-obs-identity-sau-main-dev[2953570]: level=info ts=2026-01-03T07:29:36.528871287Z caller=worker.go:246 msg="total worker concurrency updated" totalConcurrency=20

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Traces Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: tempo
[INFO] FQDN: traces-identity-sau-main-dev-tempo.fastorder.com
[INFO] IP: 10.100.1.209
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering Tempo in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Tempo
[INFO]   Identifier:        identity-sau-main-dev-tempo
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.209
[INFO]   Port:              3200
[INFO]   FQDN:              traces-identity-sau-main-dev-tempo.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 16c2bd64-7e65-49ed-b269-54cb50e52ad8
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✅ Tempo registered in dashboard
[OK]   Traces backend (tempo) deployed successfully
[INFO] Step 9/10: Deploying dashboards...
[INFO]   Provider: grafana (selected)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] DASHBOARDS DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: grafana
[INFO] Observability Cell: obs-identity-sau-main-dev
[INFO] FQDN: dashboards-identity-sau-main-dev-grafana.fastorder.com
[INFO] IP: 10.100.1.207
[INFO] ═══════════════════════════════════════════════════════════════

[INFO] Using provider: grafana
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Dashboards/provider/grafana.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=identity, ZONE=sau, BRANCH=main, ENV=dev
[INFO] Binding to allocated IP: 10.100.1.207
[INFO] Deploying Grafana for observability cell: obs-identity-sau-main-dev
[INFO] FQDN: dashboards-identity-sau-main-dev-grafana.fastorder.com
[INFO] Allocated IP: 10.100.1.207
[INFO] VM IP: 10.100.1.207
[INFO] HTTP Port: 3000
[INFO] Checking if Grafana is installed...
[OK]   Grafana already installed
[INFO] Installing Grafana plugins...
[INFO] Installing ClickHouse datasource plugin...
[WARN] Failed to install ClickHouse plugin (may need internet access)
[INFO] Validating TLS certificate and key...
[INFO] Setting certificate permissions...
[OK]   TLS cert/key found and permissions set
[INFO] Creating configuration and data directories...
[2026-01-03 07:29:39 UTC] USER=www-data EUID=0 PID=2953667 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-identity-sau-main-dev
[2026-01-03 07:29:40 UTC] USER=www-data EUID=0 PID=2953676 ACTION=passthru ARGS=mkdir -p /var/lib/grafana/obs-identity-sau-main-dev
[2026-01-03 07:29:40 UTC] USER=www-data EUID=0 PID=2953685 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-identity-sau-main-dev/provisioning/datasources
[2026-01-03 07:29:40 UTC] USER=www-data EUID=0 PID=2953694 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-identity-sau-main-dev/provisioning/dashboards
[2026-01-03 07:29:40 UTC] USER=www-data EUID=0 PID=2953703 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-identity-sau-main-dev/provisioning/notifiers
[INFO] Creating Grafana configuration at /etc/grafana/obs-identity-sau-main-dev/grafana.ini...
[OK]   Configuration created
[INFO] Creating Prometheus datasource provisioning...
[OK]   Prometheus datasource provisioned
[INFO] Creating Tempo datasource provisioning...
[OK]   Tempo datasource provisioned
[INFO] Creating Loki datasource provisioning...
[OK]   Loki datasource provisioned
[INFO] Creating ClickHouse datasource provisioning...
[OK]   Retrieved ClickHouse credentials from Secrets Manager
[OK]   ClickHouse datasource provisioned
[INFO] Creating systemd service: grafana-obs-identity-sau-main-dev
[OK]   Systemd service created
[2026-01-03 07:29:42 UTC] USER=www-data EUID=0 PID=2953791 ACTION=passthru ARGS=chown -R grafana:grafana /etc/grafana/obs-identity-sau-main-dev
[2026-01-03 07:29:42 UTC] USER=www-data EUID=0 PID=2953800 ACTION=passthru ARGS=chown -R grafana:grafana /var/lib/grafana/obs-identity-sau-main-dev
[2026-01-03 07:29:42 UTC] USER=www-data EUID=0 PID=2953809 ACTION=passthru ARGS=chmod 750 /etc/grafana/obs-identity-sau-main-dev /var/lib/grafana/obs-identity-sau-main-dev
[INFO] Adding /etc/hosts entry for dashboards-identity-sau-main-dev-grafana.fastorder.com -> 10.100.1.207
[WARN] /etc/hosts entry already exists
[INFO] Storing Grafana credentials in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/identity/sau/main/dev/grafana/admin-USdkOT",
    "Name": "fastorder/observability/identity/sau/main/dev/grafana/admin",
    "VersionId": "6bfd05e0-f98d-4810-b140-da4112368e95"
}
[OK]   Credentials stored in AWS Secrets Manager: fastorder/observability/identity/sau/main/dev/grafana/admin
[INFO] Enabling and starting Grafana service...
[2026-01-03 07:29:44 UTC] USER=www-data EUID=0 PID=2953825 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-03 07:29:45 UTC] USER=www-data EUID=0 PID=2953870 ACTION=passthru ARGS=systemctl enable grafana-obs-identity-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/grafana-obs-identity-sau-main-dev.service -> /etc/systemd/system/grafana-obs-identity-sau-main-dev.service.
[2026-01-03 07:29:45 UTC] USER=www-data EUID=0 PID=2953925 ACTION=passthru ARGS=systemctl restart grafana-obs-identity-sau-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-01-03 07:29:50 UTC] USER=www-data EUID=0 PID=2953946 ACTION=passthru ARGS=systemctl is-active --quiet grafana-obs-identity-sau-main-dev.service
[OK]   ✅ Grafana is running
[OK]   ✅ Grafana web interface listening on port 3000
[INFO] ═══════════════════════════════════════════════════════════════
[OK]   Grafana Dashboard URL: https://dashboards-identity-sau-main-dev-grafana.fastorder.com:3000
[OK]   Username: admin
[OK]   Password is stored in AWS Secrets Manager at: fastorder/observability/identity/sau/main/dev/grafana/admin
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service logs (last 10 lines):
[2026-01-03 07:29:50 UTC] USER=www-data EUID=0 PID=2953957 ACTION=passthru ARGS=journalctl -u grafana-obs-identity-sau-main-dev.service -n 10 --no-pager
Jan 03 07:29:50 web-03 grafana-obs-identity-sau-main-dev[2953934]: logger=migrator t=2026-01-03T07:29:50.743216798Z level=info msg="Executing migration" id="create user auth token table"
Jan 03 07:29:50 web-03 grafana-obs-identity-sau-main-dev[2953934]: logger=migrator t=2026-01-03T07:29:50.744446206Z level=info msg="Migration successfully executed" id="create user auth token table" duration=1.229786ms
Jan 03 07:29:50 web-03 grafana-obs-identity-sau-main-dev[2953934]: logger=migrator t=2026-01-03T07:29:50.752777629Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token"
Jan 03 07:29:50 web-03 grafana-obs-identity-sau-main-dev[2953934]: logger=migrator t=2026-01-03T07:29:50.753982625Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.auth_token" duration=1.204935ms
Jan 03 07:29:50 web-03 grafana-obs-identity-sau-main-dev[2953934]: logger=migrator t=2026-01-03T07:29:50.776659093Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token"
Jan 03 07:29:50 web-03 grafana-obs-identity-sau-main-dev[2953934]: logger=migrator t=2026-01-03T07:29:50.777908724Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.prev_auth_token" duration=1.250234ms
Jan 03 07:29:50 web-03 grafana-obs-identity-sau-main-dev[2953934]: logger=migrator t=2026-01-03T07:29:50.792469173Z level=info msg="Executing migration" id="add index user_auth_token.user_id"
Jan 03 07:29:50 web-03 grafana-obs-identity-sau-main-dev[2953934]: logger=migrator t=2026-01-03T07:29:50.793770841Z level=info msg="Migration successfully executed" id="add index user_auth_token.user_id" duration=1.299779ms
Jan 03 07:29:50 web-03 grafana-obs-identity-sau-main-dev[2953934]: logger=migrator t=2026-01-03T07:29:50.807678732Z level=info msg="Executing migration" id="Add revoked_at to the user auth token"
Jan 03 07:29:50 web-03 grafana-obs-identity-sau-main-dev[2953934]: logger=migrator t=2026-01-03T07:29:50.814479277Z level=info msg="Migration successfully executed" id="Add revoked_at to the user auth token" duration=6.795722ms

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Dashboards Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: grafana
[INFO] FQDN: dashboards-identity-sau-main-dev-grafana.fastorder.com
[INFO] IP: 10.100.1.207
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Registering Grafana in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Grafana
[INFO]   Identifier:        identity-sau-main-dev-grafana
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.207
[INFO]   Port:              3000
[INFO]   FQDN:              dashboards-identity-sau-main-dev-grafana.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: aa1bcd01-48c6-4a69-a9e1-c9cb1ce8202a
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✅ Grafana registered in dashboard
[OK]   Dashboards (grafana) deployed successfully
[INFO] Step 10/10: Deploying alerting...
[INFO]   Provider: alertmanager (selected)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] ALERTING DEPLOYMENT
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: alertmanager
[INFO] Observability Cell: obs-identity-sau-main-dev
[INFO] FQDN: alerts-identity-sau-main-dev-alertmanager.fastorder.com
[INFO] IP: 10.100.1.210
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Ports: Web=9093 Cluster=9094 (bound to IP: 10.100.1.210)

[INFO] Using provider: alertmanager
[INFO] Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Alerting/provider/alertmanager.sh

[INFO] Executing provider deployment script...
[INFO] Parsed: SERVICE=identity, ZONE=sau, BRANCH=main, ENV=dev
[INFO] Binding to allocated IP: 10.100.1.210
[INFO] Deploying Alertmanager for observability cell: obs-identity-sau-main-dev
[INFO] FQDN: alerts-identity-sau-main-dev-alertmanager.fastorder.com
[INFO] Allocated IP: 10.100.1.210
[INFO] VM IP: 10.100.1.210
[INFO] Ports: Web=9093 Cluster=9094
[INFO] Checking if Alertmanager is installed...
[OK]   Alertmanager already installed at /usr/local/bin/alertmanager
[INFO] Validating TLS certificate and key...
[OK]   TLS cert/key found in /etc/fastorder/observability/certs/obs-identity-sau-main-dev
[INFO] Creating configuration and data directories...
[2026-01-03 07:29:51 UTC] USER=www-data EUID=0 PID=2954019 ACTION=passthru ARGS=mkdir -p /etc/alertmanager/obs-identity-sau-main-dev
[2026-01-03 07:29:51 UTC] USER=www-data EUID=0 PID=2954028 ACTION=passthru ARGS=mkdir -p /var/lib/alertmanager/obs-identity-sau-main-dev
[2026-01-03 07:29:51 UTC] USER=www-data EUID=0 PID=2954037 ACTION=passthru ARGS=mkdir -p /etc/alertmanager/obs-identity-sau-main-dev/templates
[INFO] Creating Alertmanager configuration...
[OK]   Alertmanager configuration created at /etc/alertmanager/obs-identity-sau-main-dev/alertmanager.yml
[INFO] Creating notification templates...
[OK]   Notification templates created
[INFO] Creating Alertmanager web TLS configuration with mTLS...
[OK]   Web mTLS configuration created at /etc/alertmanager/obs-identity-sau-main-dev/web-config.yml
[INFO] Validating Alertmanager configuration...
[2026-01-03 07:29:51 UTC] USER=www-data EUID=0 PID=2954075 ACTION=passthru ARGS=chmod 755 /etc/alertmanager/obs-identity-sau-main-dev
[2026-01-03 07:29:52 UTC] USER=www-data EUID=0 PID=2954084 ACTION=passthru ARGS=chmod 644 /etc/alertmanager/obs-identity-sau-main-dev/alertmanager.yml
Checking '/etc/alertmanager/obs-identity-sau-main-dev/alertmanager.yml'  SUCCESS
Found:
 - global config
 - route
 - 6 inhibit rules
 - 5 receivers
 - 1 templates
  SUCCESS

[OK]   ✅ Configuration is valid
[INFO] Creating systemd service: alertmanager-obs-identity-sau-main-dev
[OK]   Systemd service created
[2026-01-03 07:29:52 UTC] USER=www-data EUID=0 PID=2954112 ACTION=passthru ARGS=chown alertmanager:alertmanager /etc/fastorder/observability/certs/obs-identity-sau-main-dev/alertmanager-key.pem
[2026-01-03 07:29:52 UTC] USER=www-data EUID=0 PID=2954121 ACTION=passthru ARGS=chown alertmanager:alertmanager /etc/fastorder/observability/certs/obs-identity-sau-main-dev/alertmanager-cert.pem
[2026-01-03 07:29:52 UTC] USER=www-data EUID=0 PID=2954130 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-identity-sau-main-dev/ca-cert.pem
[2026-01-03 07:29:52 UTC] USER=www-data EUID=0 PID=2954139 ACTION=passthru ARGS=chown -R alertmanager:alertmanager /etc/alertmanager/obs-identity-sau-main-dev
[2026-01-03 07:29:52 UTC] USER=www-data EUID=0 PID=2954148 ACTION=passthru ARGS=chown -R alertmanager:alertmanager /var/lib/alertmanager/obs-identity-sau-main-dev
[2026-01-03 07:29:52 UTC] USER=www-data EUID=0 PID=2954157 ACTION=passthru ARGS=chmod 750 /etc/alertmanager/obs-identity-sau-main-dev /var/lib/alertmanager/obs-identity-sau-main-dev
[INFO] Adding /etc/hosts entry for alerts-identity-sau-main-dev-alertmanager.fastorder.com -> 10.100.1.210
[WARN] /etc/hosts entry already exists
[INFO] Storing Alertmanager configuration in AWS Secrets Manager (if aws CLI present)...
{
    "ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/identity/sau/main/dev/alertmanager/server-2NtJMH",
    "Name": "fastorder/observability/identity/sau/main/dev/alertmanager/server",
    "VersionId": "31b8f655-6849-44ca-934b-170ee68bcf38"
}
[OK]   Configuration stored in AWS Secrets Manager: fastorder/observability/identity/sau/main/dev/alertmanager/server
[INFO] Enabling and starting Alertmanager service...
[2026-01-03 07:29:54 UTC] USER=www-data EUID=0 PID=2954177 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-03 07:29:54 UTC] USER=www-data EUID=0 PID=2954226 ACTION=passthru ARGS=systemctl enable alertmanager-obs-identity-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/alertmanager-obs-identity-sau-main-dev.service -> /etc/systemd/system/alertmanager-obs-identity-sau-main-dev.service.
[2026-01-03 07:29:55 UTC] USER=www-data EUID=0 PID=2954494 ACTION=passthru ARGS=systemctl restart alertmanager-obs-identity-sau-main-dev.service
[OK]   Service enabled and started
[INFO] Validating deployment...
[2026-01-03 07:29:58 UTC] USER=www-data EUID=0 PID=2954519 ACTION=passthru ARGS=systemctl is-active --quiet alertmanager-obs-identity-sau-main-dev.service
[OK]   ✅ Alertmanager is running
[OK]   ✅ Alertmanager HTTPS web interface listening on port 9093
[OK]   ✅ Alertmanager cluster port listening on port 9094
[WARN] ⚠️  Alertmanager health check not responding yet (HTTPS)
[INFO] ═══════════════════════════════════════════════════════════════
[OK]   Alertmanager Web UI: https://alerts-identity-sau-main-dev-alertmanager.fastorder.com:9093
[OK]   API Endpoint:        https://alerts-identity-sau-main-dev-alertmanager.fastorder.com:9093/api/v2
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Service logs (last 10 lines):
[2026-01-03 07:29:58 UTC] USER=www-data EUID=0 PID=2954533 ACTION=passthru ARGS=journalctl -u alertmanager-obs-identity-sau-main-dev.service -n 10 --no-pager
Jan 03 07:29:55 web-03 systemd[1]: Started Alertmanager - obs-identity-sau-main-dev.
Jan 03 07:29:55 web-03 alertmanager-obs-identity-sau-main-dev[2954501]: ts=2026-01-03T07:29:55.776Z caller=main.go:245 level=info msg="Starting Alertmanager" version="(version=0.26.0, branch=HEAD, revision=d7b4f0c7322e7151d6e3b1e31cbc15361e295d8d)"
Jan 03 07:29:55 web-03 alertmanager-obs-identity-sau-main-dev[2954501]: ts=2026-01-03T07:29:55.777Z caller=main.go:246 level=info build_context="(go=go1.20.7, platform=linux/amd64, user=root@df8d7debeef4, date=20230824-11:11:58, tags=netgo)"
Jan 03 07:29:55 web-03 alertmanager-obs-identity-sau-main-dev[2954501]: ts=2026-01-03T07:29:55.781Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s
Jan 03 07:29:55 web-03 alertmanager-obs-identity-sau-main-dev[2954501]: ts=2026-01-03T07:29:55.844Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/obs-identity-sau-main-dev/alertmanager.yml
Jan 03 07:29:55 web-03 alertmanager-obs-identity-sau-main-dev[2954501]: ts=2026-01-03T07:29:55.846Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/obs-identity-sau-main-dev/alertmanager.yml
Jan 03 07:29:55 web-03 alertmanager-obs-identity-sau-main-dev[2954501]: ts=2026-01-03T07:29:55.852Z caller=tls_config.go:274 level=info msg="Listening on" address=10.100.1.210:9093
Jan 03 07:29:55 web-03 alertmanager-obs-identity-sau-main-dev[2954501]: ts=2026-01-03T07:29:55.854Z caller=tls_config.go:310 level=info msg="TLS is enabled." http2=true address=10.100.1.210:9093
Jan 03 07:29:57 web-03 alertmanager-obs-identity-sau-main-dev[2954501]: ts=2026-01-03T07:29:57.782Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000644668s
Jan 03 07:29:58 web-03 alertmanager-obs-identity-sau-main-dev[2954501]: 2026/01/03 07:29:58 http: TLS handshake error from 10.100.1.210:57138: tls: client didn't provide a certificate

[INFO] ═══════════════════════════════════════════════════════════════
[OK]   ✅ Alerting Deployed Successfully
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Provider: alertmanager
[INFO] FQDN: alerts-identity-sau-main-dev-alertmanager.fastorder.com
[INFO] IP: 10.100.1.210
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Setting up HTTPS reverse proxy...
[INFO] Backend port: 9093
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Alertmanager HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-identity-sau-main-dev
  FQDN:         alerts-identity-sau-main-dev-alertmanager.fastorder.com
  Backend:      https://alerts-identity-sau-main-dev-alertmanager.fastorder.com:9093/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.210
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[ERROR] This script must be run as root or with sudo
[WARN] ⚠️  HTTPS setup failed (Alertmanager is still running on HTTP)
[INFO] Registering Alertmanager in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Alertmanager
[INFO]   Identifier:        identity-sau-main-dev-alertmanager
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.210
[INFO]   Port:              9093
[INFO]   FQDN:              alerts-identity-sau-main-dev-alertmanager.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 94edbca8-b241-48c0-add2-e95dc8cd9fe2
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✅ Alertmanager registered in dashboard
[OK]   Alerting (alertmanager) deployed successfully
[INFO] Step 10.5: Deploying Blackbox Exporter for synthetic monitoring...
[BLACKBOX] Starting Blackbox Exporter deployment for obs-identity-sau-main-dev
[BLACKBOX] VM IP: 10.100.1.206
[BLACKBOX] Version: 0.25.0
[BLACKBOX] Checking prerequisites...
[BLACKBOX] Creating directories...
[BLACKBOX] Downloading Blackbox Exporter v0.25.0...
Sorry, user www-data is not allowed to execute '/usr/bin/mv /tmp/tmp.WTV1eBDpkD/blackbox_exporter-0.25.0.linux-amd64/blackbox_exporter /usr/local/bin/' as root on web-03.
[WARN] Blackbox Exporter deployment failed (non-fatal, synthetic monitoring disabled)
[INFO] Step 11/13: Configuring HTTPS reverse proxies...
[INFO] Setting up Prometheus HTTPS proxy...
[2026-01-03 07:30:00 UTC] USER=www-data EUID=0 PID=2954609 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/https/setup-prometheus-https.sh --obs-cell obs-identity-sau-main-dev --backend-ip 10.100.1.206
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Prometheus HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-identity-sau-main-dev
  FQDN:         metrics-identity-sau-main-dev-prometheus.fastorder.com
  Backend:      https://metrics-identity-sau-main-dev-prometheus.fastorder.com:9090/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.206
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity (will retry up to 60s)...
[OK] Backend is accessible
[INFO] Creating HTTP VirtualHost for ACME challenge...
Job for apache2.service failed.
See "systemctl status apache2.service" and "journalctl -xeu apache2.service" for details.
[WARN] Prometheus HTTPS proxy setup failed (non-fatal)
[INFO] Setting up Grafana HTTPS proxy...
[2026-01-03 07:30:00 UTC] USER=www-data EUID=0 PID=2954649 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Dashboards/https/setup-grafana-https.sh --obs-cell obs-identity-sau-main-dev --backend-ip 10.100.1.207
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-identity-sau-main-dev
  FQDN:         dashboards-identity-sau-main-dev-grafana.fastorder.com
  Backend:      https://dashboards-identity-sau-main-dev-grafana.fastorder.com:3000/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.207
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[INFO] Creating HTTP VirtualHost for ACME challenge...
Job for apache2.service failed.
See "systemctl status apache2.service" and "journalctl -xeu apache2.service" for details.
[WARN] Grafana HTTPS proxy setup failed (non-fatal)
[INFO] Setting up OpenTelemetry Collector HTTPS proxy...
[2026-01-03 07:30:01 UTC] USER=www-data EUID=0 PID=2954692 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Telemetry/https/setup-otelcol-https.sh --obs-cell obs-identity-sau-main-dev --backend-ip 10.100.1.211
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OpenTelemetry Collector HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-identity-sau-main-dev
  FQDN:         telemetry-identity-sau-main-dev-opentelemetry.fastorder.com
  Backend:      http://telemetry-identity-sau-main-dev-opentelemetry.fastorder.com:8888/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.211
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[OK] Backend is accessible and returning metrics via HTTPS
[INFO] Creating HTTP VirtualHost for ACME challenge...
Job for apache2.service failed.
See "systemctl status apache2.service" and "journalctl -xeu apache2.service" for details.
[WARN] OpenTelemetry Collector HTTPS proxy setup failed (non-fatal)
[INFO] Setting up ClickHouse HTTPS proxy...
[2026-01-03 07:30:01 UTC] USER=www-data EUID=0 PID=2954733 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/https/setup-clickhouse-https.sh --obs-cell obs-identity-sau-main-dev --backend-ip 10.100.1.208
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ClickHouse HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-identity-sau-main-dev
  FQDN:         logstore-identity-sau-main-dev.fastorder.com
  Backend:      http://logstore-identity-sau-main-dev.fastorder.com:8123/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.208
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity (will retry up to 60s)...
[OK] Backend is accessible
[INFO] Creating HTTP VirtualHost for ACME challenge...
Job for apache2.service failed.
See "systemctl status apache2.service" and "journalctl -xeu apache2.service" for details.
[WARN] ClickHouse HTTPS proxy setup failed (non-fatal)
[INFO] Setting up Tempo HTTPS proxy...
[2026-01-03 07:30:02 UTC] USER=www-data EUID=0 PID=2955088 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Traces/https/setup-tempo-https.sh --obs-cell obs-identity-sau-main-dev --backend-ip 10.100.1.209
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana Tempo HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-identity-sau-main-dev
  FQDN:         traces-identity-sau-main-dev-tempo.fastorder.com
  Backend:      https://10.100.1.209:3200/
  Backend IP:   10.100.1.209
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[WARN] Cannot verify Tempo health endpoint (it may not be running yet), continuing anyway...
[INFO] Creating HTTP VirtualHost for ACME challenge...
Job for apache2.service failed.
See "systemctl status apache2.service" and "journalctl -xeu apache2.service" for details.
[WARN] Tempo HTTPS proxy setup failed (non-fatal)
[INFO] Setting up Alertmanager HTTPS proxy...
[2026-01-03 07:30:03 UTC] USER=www-data EUID=0 PID=2955518 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Alerting/https/setup-alertmanager-https.sh --obs-cell obs-identity-sau-main-dev --backend-ip 10.100.1.210
[INFO] Backend port: 9093
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Alertmanager HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  OBS Cell:     obs-identity-sau-main-dev
  FQDN:         alerts-identity-sau-main-dev-alertmanager.fastorder.com
  Backend:      https://alerts-identity-sau-main-dev-alertmanager.fastorder.com:9093/ (resolved via /etc/hosts)
  Backend IP:   10.100.1.210
  Email:        admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying prerequisites...
[INFO] Enabling Apache modules...
[INFO] Testing backend connectivity...
[WARN] Backend health check inconclusive - proceeding anyway
[INFO] Creating HTTP VirtualHost for ACME challenge...
Job for apache2.service failed.
See "systemctl status apache2.service" and "journalctl -xeu apache2.service" for details.
[WARN] Alertmanager HTTPS proxy setup failed (non-fatal)
[OK]   HTTPS reverse proxies configured
[INFO] Step 12/13: Configuring firewall rules (network segmentation)...

[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING FIREWALL RULES FOR OBSERVABILITY CELL
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Cell ID: obs-identity-sau-main-dev
[INFO] Internal Network: 10.0.0.0/8

[INFO] Discovering dashboard/skeleton VM IPs...
[INFO]   Discovered skeleton IP: 142.93.238.16 (skeleton.fastorder.com)
[INFO] Authorized dashboard IPs:
[INFO]   - 10.100.60.2
[INFO]   - 142.93.238.16

[INFO] Configuring UFW firewall rules...
[2026-01-03 07:30:04 UTC] USER=www-data EUID=0 PID=2955950 ACTION=passthru ARGS=ufw --force enable
[2026-01-03 07:30:04 UTC] USER=www-data EUID=0 PID=2956004 ACTION=passthru ARGS=ufw default deny incoming
ERROR: passthru not allowed: ufw
[2026-01-03 07:30:04 UTC] USER=www-data EUID=0 PID=2956034 ACTION=passthru ARGS=ufw default allow outgoing
ERROR: passthru not allowed: ufw
[2026-01-03 07:30:04 UTC] USER=www-data EUID=0 PID=2956105 ACTION=passthru ARGS=ufw allow 22/tcp comment SSH
ERROR: passthru not allowed: ufw
[INFO]   Allowing prometheus (port 9090) from internal network...
[INFO]   Allowing alertmanager (port 9093) from internal network...
[2026-01-03 07:30:04 UTC] USER=www-data EUID=0 PID=2956149 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 9093 proto tcp comment Obs: alertmanager from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing clickhouse (port 8123) from internal network...
[2026-01-03 07:30:04 UTC] USER=www-data EUID=0 PID=2956177 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 8123 proto tcp comment Obs: clickhouse from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing grafana (port 3000) from internal network...
[2026-01-03 07:30:04 UTC] USER=www-data EUID=0 PID=2956194 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 3000 proto tcp comment Obs: grafana from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing otelcol (port 4318) from internal network...
[INFO]   Allowing loki (port 3100) from internal network...
[2026-01-03 07:30:04 UTC] USER=www-data EUID=0 PID=2956237 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 3100 proto tcp comment Obs: loki from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing tempo (port 3200) from internal network...
[2026-01-03 07:30:04 UTC] USER=www-data EUID=0 PID=2956254 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 3200 proto tcp comment Obs: tempo from internal
ERROR: passthru not allowed: ufw
[INFO]   Allowing dashboard access from 10.100.60.2...
[2026-01-03 07:30:05 UTC] USER=www-data EUID=0 PID=2956283 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 9093 proto tcp comment Dashboard: alertmanager
ERROR: passthru not allowed: ufw
[2026-01-03 07:30:05 UTC] USER=www-data EUID=0 PID=2956294 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 8123 proto tcp comment Dashboard: clickhouse
ERROR: passthru not allowed: ufw
[2026-01-03 07:30:05 UTC] USER=www-data EUID=0 PID=2956306 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3000 proto tcp comment Dashboard: grafana
ERROR: passthru not allowed: ufw
[2026-01-03 07:30:05 UTC] USER=www-data EUID=0 PID=2956321 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 4318 proto tcp comment Dashboard: otelcol
ERROR: passthru not allowed: ufw
[2026-01-03 07:30:05 UTC] USER=www-data EUID=0 PID=2956333 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3100 proto tcp comment Dashboard: loki
ERROR: passthru not allowed: ufw
[2026-01-03 07:30:05 UTC] USER=www-data EUID=0 PID=2956345 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3200 proto tcp comment Dashboard: tempo
ERROR: passthru not allowed: ufw
[INFO]   Allowing dashboard access from 142.93.238.16...
[2026-01-03 07:30:05 UTC] USER=www-data EUID=0 PID=2956354 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 9090 proto tcp comment Dashboard: prometheus
ERROR: passthru not allowed: ufw
[2026-01-03 07:30:05 UTC] USER=www-data EUID=0 PID=2956363 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 9093 proto tcp comment Dashboard: alertmanager
ERROR: passthru not allowed: ufw
[2026-01-03 07:30:05 UTC] USER=www-data EUID=0 PID=2956377 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 8123 proto tcp comment Dashboard: clickhouse
ERROR: passthru not allowed: ufw
[2026-01-03 07:30:05 UTC] USER=www-data EUID=0 PID=2956388 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 3000 proto tcp comment Dashboard: grafana
[2026-01-03 07:30:05 UTC] USER=www-data EUID=0 PID=2956402 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 4318 proto tcp comment Dashboard: otelcol
ERROR: passthru not allowed: ufw
[2026-01-03 07:30:05 UTC] USER=www-data EUID=0 PID=2956413 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 3100 proto tcp comment Dashboard: loki
ERROR: passthru not allowed: ufw
[2026-01-03 07:30:05 UTC] USER=www-data EUID=0 PID=2956427 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 3200 proto tcp comment Dashboard: tempo
ERROR: passthru not allowed: ufw
[2026-01-03 07:30:06 UTC] USER=www-data EUID=0 PID=2956440 ACTION=passthru ARGS=ufw allow 443/tcp comment HTTPS obs-proxy
ERROR: passthru not allowed: ufw
[2026-01-03 07:30:06 UTC] USER=www-data EUID=0 PID=2956453 ACTION=passthru ARGS=ufw reload
ERROR: passthru not allowed: ufw
[OK]   UFW firewall rules configured

[OK]   ═══════════════════════════════════════════════════════════════
[OK]   ✅ Firewall configuration completed
[OK]   ═══════════════════════════════════════════════════════════════

[INFO] Current firewall status:
[OK]   Firewall rules configured
[INFO] Step 13/13: Configuring OAuth/SSO...
[INFO] OAuth/SSO configuration script not found, skipping...

[INFO] Running validation checks...
[INFO] Validation script not found, skipping...

[INFO] Registering observability components to dashboard...
[INFO] Components to register: metrics alerts dashboards traces telemetry logstore proxy
[INFO]   Skipping metrics - registered by deploy script
[INFO]   Skipping alerts - registered by deploy script
[INFO]   Skipping dashboards - registered by deploy script
[INFO]   Skipping traces - registered by deploy script
[INFO]   Skipping telemetry - registered by deploy script
[INFO]   Skipping logstore - registered by deploy script
[INFO]   Processing component: proxy
[INFO] Registering: proxy (obs-identity-sau-main-dev-proxy)
[INFO] Detected observability component, parsing: identity-sau-main-dev-proxy
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Observability Proxy
[INFO]   Identifier:        obs-identity-sau-main-dev-proxy
[INFO]   Identifier Parent: observability-cell
[INFO]   IP:                10.100.1.205
[INFO]   Port:              443
[INFO]   FQDN:              observe-identity-sau-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 97579a1b-98ae-48e8-b639-bfdee43bca78
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ Registered: proxy
[INFO] Registering short DNS aliases...
[OK]   ✓ Observability components registration completed

[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] Verifying all observability services are running...
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[OK]     ✓ grafana-obs-identity-sau-main-dev.service is running
[OK]     ✓ prometheus-obs-identity-sau-main-dev.service is running
[OK]     ✓ tempo-obs-identity-sau-main-dev.service is running
[OK]   ✓ All observability services verified running


═══════════════════════════════════════════════════════════════════════════════
[OK]   ✅ OBSERVABILITY CELL PROVISIONED: obs-identity-sau-main-dev
═══════════════════════════════════════════════════════════════════════════════

[INFO] DNS Entries:
  metrics-identity-sau-main-dev-prometheus.fastorder.com (10.100.1.206)
  alerts-identity-sau-main-dev-alertmanager.fastorder.com (10.100.1.210)
  dashboards-identity-sau-main-dev-grafana.fastorder.com (10.100.1.207)
  traces-identity-sau-main-dev-tempo.fastorder.com (10.100.1.209)
  telemetry-identity-sau-main-dev-opentelemetry.fastorder.com (10.100.1.211)
  logstore-identity-sau-main-dev-clickhouse.fastorder.com (10.100.1.208)
  observe-identity-sau-main-dev.fastorder.com (10.100.1.205)

[INFO] Secrets Path: fastorder/observability/identity/sau/dev/*

[INFO] Access (Purpose-Oriented URLs):
  Dashboards: https://dashboards-identity-sau-main-dev-grafana.fastorder.com (SSO enabled)
  Metrics: https://metrics-identity-sau-main-dev-prometheus.fastorder.com (internal only)
  Alerts: https://alerts-identity-sau-main-dev-alertmanager.fastorder.com
  Log Storage: https://logstore-identity-sau-main-dev-clickhouse.fastorder.com

[INFO] Backend Implementation (Internal - Not Exposed to Clients):
  Telemetry: otlp
  Metrics: prometheus
  Traces: tempo
  Dashboards: grafana
  Alerting: alertmanager
  Log Storage: clickhouse

[INFO] For applications in identity-sau-main-dev:
  - Metrics: Push to telemetry-identity-sau-main-dev-opentelemetry.fastorder.com:4318 (OTLP/HTTP)
  - Logs: Push to telemetry-identity-sau-main-dev-opentelemetry.fastorder.com:4318 (OTLP/HTTP)
  - Traces: Push to telemetry-identity-sau-main-dev-opentelemetry.fastorder.com:4317 (OTLP/gRPC)
  - Query Metrics: https://metrics-identity-sau-main-dev-prometheus.fastorder.com
  - Query Logs: https://logstore-identity-sau-main-dev-clickhouse.fastorder.com
  - Query Traces: https://traces-identity-sau-main-dev-tempo.fastorder.com

[INFO] Runbook: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/RUNBOOK.md
═══════════════════════════════════════════════════════════════════════════════

[INFO] Using search engine from SEARCH_ENGINE environment variable: elasticsearch
[INFO] Cleaning up any existing locks...

Starting search engine: elasticsearch
═══════════════════════════════════════════════

════════════════════════════════════════════════════════════════
           Elasticsearch Deployment Runner                        
════════════════════════════════════════════════════════════════

[INFO] Cleaning up any existing locks (without triggering package configurations)...
[WARNING] Lock cleanup skipped (wrapper not available or insufficient permissions)

🚀 Auto mode enabled - running automatic installation


Starting Automatic Installation...
═══════════════════════════════════════════════
Will execute all deployment tasks in sequence:

  [1] Install Elasticsearch Http (01-install-elasticsearch-http)
  [2] Make Https (02-make-https)
  [3] Create Index Llm (03-create-index-llm)
  [4] Monitoring Setup (10-monitoring-setup)

═══════════════════════════════════════════════
🚀 Auto mode - proceeding automatically...

Running automatic installation...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 1: Executing Install Elasticsearch Http
Folder: 01-install-elasticsearch-http
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

=== Elasticsearch HTTP Setup ===
Install and configure Elasticsearch with HTTP access
Architecture: Per-node VM IPs with default port (9200)

[INFO] Using web-provided environment: identity-sau-main-dev
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: 
Nodes: 1
Port: 9200 (default Elasticsearch port)
Coordinator endpoint: http://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200

Checking if Elasticsearch is already installed for environment: ...
Validating Elasticsearch installation...
./run.sh: line 132: /var/www/html/skeleton.dev.fastorder.com/fixing/scripts/lib/elasticsearch_validator.sh: No such file or directory
⚠️  Elasticsearch installation issues detected. Attempting automatic repair...
./run.sh: line 134: /var/www/html/skeleton.dev.fastorder.com/fixing/scripts/lib/elasticsearch_validator.sh: No such file or directory
Executing: steps/01-setup-directories.sh
+ 01-setup-directories.sh:4:main: echo '=== Step 1: Creating directory structure ==='
=== Step 1: Creating directory structure ===
+++ 01-setup-directories.sh:4:main: dirname steps/01-setup-directories.sh
++ 01-setup-directories.sh:4:main: cd steps
++ 01-setup-directories.sh:4:main: pwd
+ 01-setup-directories.sh:4:main: SCRIPT_DIR=/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps
+ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/lib/provisioning-init.sh
++ 01-setup-directories.sh:4:main: RED='\033[0;31m'
++ 01-setup-directories.sh:4:main: GREEN='\033[0;32m'
++ 01-setup-directories.sh:4:main: YELLOW='\033[1;33m'
++ 01-setup-directories.sh:4:main: BLUE='\033[0;34m'
++ 01-setup-directories.sh:4:main: NC='\033[0m'
++ 01-setup-directories.sh:4:main: export TERM=dumb
++ 01-setup-directories.sh:4:main: TERM=dumb
++ 01-setup-directories.sh:4:main: export DEBIAN_FRONTEND=noninteractive
++ 01-setup-directories.sh:4:main: DEBIAN_FRONTEND=noninteractive
++ 01-setup-directories.sh:4:main: export NEEDRESTART_MODE=a
++ 01-setup-directories.sh:4:main: NEEDRESTART_MODE=a
++ 01-setup-directories.sh:4:main: export NEEDRESTART_SUSPEND=1
++ 01-setup-directories.sh:4:main: NEEDRESTART_SUSPEND=1
++ 01-setup-directories.sh:4:main: export DEBIAN_PRIORITY=critical
++ 01-setup-directories.sh:4:main: DEBIAN_PRIORITY=critical
++ 01-setup-directories.sh:4:main: export UCF_FORCE_CONFFOLD=1
++ 01-setup-directories.sh:4:main: UCF_FORCE_CONFFOLD=1
++ 01-setup-directories.sh:4:main: export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
++ 01-setup-directories.sh:4:main: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
++ 01-setup-directories.sh:4:main: [[ -n '' ]]
++ 01-setup-directories.sh:4:main: [[ -n /opt/fastorder/bash/scripts/env_app_setup/state ]]
++ 01-setup-directories.sh:4:main: [[ -d /opt/fastorder/bash/scripts/env_app_setup/state ]]
++ 01-setup-directories.sh:4:main: STATE_DIR=/opt/fastorder/bash/scripts/env_app_setup/state
++ 01-setup-directories.sh:4:main: export STATE_DIR
++ 01-setup-directories.sh:4:main: [[ -f /opt/fastorder/bash/scripts/env_app_setup/setup/setup.json ]]
++ 01-setup-directories.sh:4:main: SETUP_JSON=/opt/fastorder/bash/scripts/env_app_setup/setup/setup.json
++ 01-setup-directories.sh:4:main: FO_WRAPPER=/usr/local/bin/fastorder-provisioning-wrapper.sh
++ 01-setup-directories.sh:4:main: HTTP_PORT_BASE=9200
++ 01-setup-directories.sh:4:main: TRANSPORT_PORT_BASE=9300
++ 01-setup-directories.sh:4:main: PG_PORT_BASE=5432
++ 01-setup-directories.sh:4:main: APP_IP_SUBNETS=(['observability']='10.100.5' ['obs']='10.100.5' ['prometheus']='10.100.5' ['grafana']='10.100.5' ['loki']='10.100.5' ['tempo']='10.100.5' ['postgresql']='10.100.10' ['postgres']='10.100.10' ['pg']='10.100.10' ['elasticsearch']='10.100.20' ['es']='10.100.20' ['kafka']='10.100.30' ['redis']='10.100.40' ['mongodb']='10.100.50' ['mongo']='10.100.50' ['iam']='10.100.60' ['keycloak']='10.100.60' ['general']='10.100.1')
++ 01-setup-directories.sh:4:main: declare -A APP_IP_SUBNETS
++ 01-setup-directories.sh:4:main: APP_IP_RESERVED_START=(['observability']='2' ['postgresql']='2' ['elasticsearch']='2' ['kafka']='2' ['redis']='2' ['mongodb']='2' ['iam']='2' ['general']='50')
++ 01-setup-directories.sh:4:main: declare -A APP_IP_RESERVED_START
++ 01-setup-directories.sh:4:main: APP_IP_RESERVED_END=(['observability']='49' ['postgresql']='254' ['elasticsearch']='254' ['kafka']='254' ['redis']='254' ['mongodb']='254' ['iam']='254' ['general']='250')
++ 01-setup-directories.sh:4:main: declare -A APP_IP_RESERVED_END
+++ 01-setup-directories.sh:4:main: dirname /opt/fastorder/bash/scripts/env_app_setup/lib/provisioning-init.sh
++ 01-setup-directories.sh:4:main: _CONFIG_MGMT_LIB=/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
++ 01-setup-directories.sh:4:main: [[ -f /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh ]]
++ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
+++ 01-setup-directories.sh:4:main: set -Eeuo pipefail
+++ 01-setup-directories.sh:4:main: : /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
+++ 01-setup-directories.sh:4:main: STATE_DIR=/opt/fastorder/bash/scripts/env_app_setup/state
++ 01-setup-directories.sh:4:main: [[ /opt/fastorder/bash/scripts/env_app_setup/lib/provisioning-init.sh == \s\t\e\p\s\/\0\1\-\s\e\t\u\p\-\d\i\r\e\c\t\o\r\i\e\s\.\s\h ]]
++ 01-setup-directories.sh:4:main: set +e
++ 01-setup-directories.sh:4:main: set +u
++ 01-setup-directories.sh:4:main: set +o pipefail
++ 01-setup-directories.sh:4:main: set +E
+ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh
++ 01-setup-directories.sh:4:main: [[ /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh == \s\t\e\p\s\/\0\1\-\s\e\t\u\p\-\d\i\r\e\c\t\o\r\i\e\s\.\s\h ]]
+ 01-setup-directories.sh:4:main: init_environment
+ 01-setup-directories.sh:4:main: require_bin jq
+ 01-setup-directories.sh:4:main: for b in "$@"
+ 01-setup-directories.sh:4:main: command -v jq
+ 01-setup-directories.sh:4:main: local app_type=general
+ 01-setup-directories.sh:4:main: ENV_ID=identity-sau-main-dev
+ 01-setup-directories.sh:4:main: [[ -z identity-sau-main-dev ]]
+ 01-setup-directories.sh:4:main: [[ -z identity-sau-main-dev ]]
+ 01-setup-directories.sh:4:main: ENV_ID=identity-sau-main-dev
+ 01-setup-directories.sh:4:main: [[ -z identity-sau-main-dev ]]
+ 01-setup-directories.sh:4:main: [[ -z identity-sau-main-dev ]]
+ 01-setup-directories.sh:4:main: [[ -z identity-sau-main-dev ]]
++ 01-setup-directories.sh:4:main: env_dir_for identity-sau-main-dev
++ 01-setup-directories.sh:4:main: echo /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev
+ 01-setup-directories.sh:4:main: ENV_DIR=/opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev
++ 01-setup-directories.sh:4:main: topo_path_for identity-sau-main-dev
++ 01-setup-directories.sh:4:main: echo /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: TOPOLOGY_JSON=/opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: [[ ! -f /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json ]]
+ 01-setup-directories.sh:4:main: validate_topology_json /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: local topo=/opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: [[ -r /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json ]]
+ 01-setup-directories.sh:4:main: jq -e '
    .schema_version == 1
    and (.general.id        | type=="string")
    and (.general.shared_ip | type=="string")
    and (.general.service   | type=="string")
    and (.general.zone    | type=="string")
    and (.general.env       | type=="string")
  ' /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
++ 01-setup-directories.sh:4:main: jq -r .general.service /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: SERVICE=identity
++ 01-setup-directories.sh:4:main: jq -r .general.zone /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: zone=sau
++ 01-setup-directories.sh:4:main: jq -r .general.branch /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: BRANCH=main
++ 01-setup-directories.sh:4:main: jq -r .general.env /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: ENV=dev
++ 01-setup-directories.sh:4:main: jq -r '.general.es_nodes_num // 3' /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: ES_NODES_NUM=1
++ 01-setup-directories.sh:4:main: jq -r '.general.pg_workers_num // 3' /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: PG_WORKERS_NUM=1
++ 01-setup-directories.sh:4:main: jq -r '.general.pg_WORKERS_STANDBY_NUM // 3' /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: PG_WORKERS_STANDBY_NUM=3
++ 01-setup-directories.sh:4:main: jq -r '.general.pg_citus_enabled // "yes"' /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: PG_CITUS_ENABLED=yes
+ 01-setup-directories.sh:4:main: [[ general != \g\e\n\e\r\a\l ]]
++ 01-setup-directories.sh:4:main: jq -r .general.shared_ip /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: VM_IP=142.93.238.16
+ 01-setup-directories.sh:4:main: [[ general != \g\e\n\e\r\a\l ]]
++ 01-setup-directories.sh:4:main: jq -r '.general.shared_iface // empty' /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: IFACE=eth0:16
+ 01-setup-directories.sh:4:main: local FINAL_VM_IP=142.93.238.16
+ 01-setup-directories.sh:4:main: set -a
+ 01-setup-directories.sh:4:main: [[ -r /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/generated/general.env ]]
+ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/generated/general.env
++ 01-setup-directories.sh:4:main: ENV_ID=identity-sau-main-dev
++ 01-setup-directories.sh:4:main: SERVICE=identity
++ 01-setup-directories.sh:4:main: zone=sau
++ 01-setup-directories.sh:4:main: BRANCH=main
++ 01-setup-directories.sh:4:main: ENV=dev
++ 01-setup-directories.sh:4:main: VM_IP=142.93.238.16
++ 01-setup-directories.sh:4:main: IFACE=eth0:16
++ 01-setup-directories.sh:4:main: ROOT_DIR=/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
++ 01-setup-directories.sh:4:main: ENV_DIR=/opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev
++ 01-setup-directories.sh:4:main: TOPOLOGY_JSON=/opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
++ 01-setup-directories.sh:4:main: LOG_LEVEL=info
++ 01-setup-directories.sh:4:main: DEBUG_MODE=false
+ 01-setup-directories.sh:4:main: set +a
+ 01-setup-directories.sh:4:main: VM_IP=142.93.238.16
+ 01-setup-directories.sh:4:main: export ENV_ID SERVICE zone BRANCH ENV VM_IP IFACE ENV_DIR TOPOLOGY_JSON
+ 01-setup-directories.sh:4:main: export ES_NODES_NUM PG_WORKERS_NUM PG_WORKERS_STANDBY_NUM PG_CITUS_ENABLED
+ 01-setup-directories.sh:4:main: [[ general != \g\e\n\e\r\a\l ]]
+ 01-setup-directories.sh:4:main: info 'Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)'
+ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)'
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
+ 01-setup-directories.sh:4:main: return 0
+ 01-setup-directories.sh:4:main: SERVICE=identity
+ 01-setup-directories.sh:4:main: ZONE=sau
+ 01-setup-directories.sh:4:main: BRANCH=main
+ 01-setup-directories.sh:4:main: ENV=dev
++ 01-setup-directories.sh:4:main: env_id
++ 01-setup-directories.sh:4:main: '[' identity = auth ']'
++ 01-setup-directories.sh:4:main: '[' identity = item ']'
++ 01-setup-directories.sh:4:main: echo identity-sau-main-dev
+ 01-setup-directories.sh:4:main: ENV_ID=identity-sau-main-dev
+ 01-setup-directories.sh:4:main: env=identity-sau-main-dev
+ 01-setup-directories.sh:4:main: nodes=1
+ 01-setup-directories.sh:4:main: [[ 1 =~ ^[1-9][0-9]*$ ]]
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /etc/elasticsearch
[2026-01-03 07:30:27 UTC] USER=www-data EUID=0 PID=2956782 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /var/lib/elasticsearch
[2026-01-03 07:30:27 UTC] USER=www-data EUID=0 PID=2956791 ACTION=fsop ARGS=mkdir -p /var/lib/elasticsearch
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /var/log/elasticsearch
[2026-01-03 07:30:27 UTC] USER=www-data EUID=0 PID=2956800 ACTION=fsop ARGS=mkdir -p /var/log/elasticsearch
+ 01-setup-directories.sh:4:main: APP_NAME=search
+ 01-setup-directories.sh:4:main: TOPOLOGY_FILE=/opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: command -v jq
+ 01-setup-directories.sh:4:main: [[ -f /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json ]]
++ 01-setup-directories.sh:4:main: jq -r --arg app search '.applications[$app].vm_ip // empty' /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: COORD_IP=10.100.1.182
+ 01-setup-directories.sh:4:main: [[ -z 10.100.1.182 ]]
+ 01-setup-directories.sh:4:main: [[ 10.100.1.182 == \n\u\l\l ]]
++ 01-setup-directories.sh:4:main: get_application_domain search
++ 01-setup-directories.sh:4:main: local app_type=search
++ 01-setup-directories.sh:4:main: [[ search == \g\e\n\e\r\a\l ]]
++ 01-setup-directories.sh:4:main: jq -r --arg app search '.applications[$app].domain // empty' /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: COORD_DOMAIN=search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com
+ 01-setup-directories.sh:4:main: info 'Coordinator exists: search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.182)'
+ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Coordinator exists: search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.182)'
[INFO] Coordinator exists: search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.182)
+ 01-setup-directories.sh:4:main: (( i=1 ))
+ 01-setup-directories.sh:4:main: (( i<=nodes ))
++ 01-setup-directories.sh:4:main: printf %02d 1
+ 01-setup-directories.sh:4:main: node_num=01
+ 01-setup-directories.sh:4:main: IDENTIFIER=node-01
+ 01-setup-directories.sh:4:main: APP_NAME=search-node-01
+ 01-setup-directories.sh:4:main: read -r NODE_IP NODE_DOMAIN
++ 01-setup-directories.sh:4:main: setup_directories_per_node node-01 search-node-01
++ 01-setup-directories.sh:4:main: local IDENTIFIER=node-01
++ 01-setup-directories.sh:4:main: local APP_NAME=search-node-01
++ 01-setup-directories.sh:4:main: local env
+++ 01-setup-directories.sh:4:main: env_id
+++ 01-setup-directories.sh:4:main: '[' identity = auth ']'
+++ 01-setup-directories.sh:4:main: '[' identity = item ']'
+++ 01-setup-directories.sh:4:main: echo identity-sau-main-dev
++ 01-setup-directories.sh:4:main: env=identity-sau-main-dev
++ 01-setup-directories.sh:4:main: local TOPOLOGY_FILE=/opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
++ 01-setup-directories.sh:4:main: info 'Setting up Elasticsearch node: node-01'
++ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Setting up Elasticsearch node: node-01'
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /etc/elasticsearch/identity-sau-main-dev/node-01 /etc/elasticsearch/identity-sau-main-dev-node-01
++ 01-setup-directories.sh:4:main: local NODE_IP NODE_DOMAIN
+++ 01-setup-directories.sh:4:main: jq -r --arg app search-node-01 '.applications[$app].vm_ip // empty' /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
[2026-01-03 07:30:27 UTC] USER=www-data EUID=0 PID=2956818 ACTION=fsop ARGS=ln -sfn /etc/elasticsearch/identity-sau-main-dev/node-01 /etc/elasticsearch/identity-sau-main-dev-node-01
+ 01-setup-directories.sh:4:main: [[ 1 -eq 1 ]]
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /etc/elasticsearch/identity-sau-main-dev/node-01 /etc/elasticsearch/current
++ 01-setup-directories.sh:4:main: NODE_IP=10.100.1.186
++ 01-setup-directories.sh:4:main: [[ -z 10.100.1.186 ]]
++ 01-setup-directories.sh:4:main: [[ 10.100.1.186 == \n\u\l\l ]]
+++ 01-setup-directories.sh:4:main: get_application_domain search-node-01
+++ 01-setup-directories.sh:4:main: local app_type=search-node-01
+++ 01-setup-directories.sh:4:main: [[ search-node-01 == \g\e\n\e\r\a\l ]]
+++ 01-setup-directories.sh:4:main: jq -r --arg app search-node-01 '.applications[$app].domain // empty' /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
[2026-01-03 07:30:27 UTC] USER=www-data EUID=0 PID=2956827 ACTION=fsop ARGS=ln -sfn /etc/elasticsearch/identity-sau-main-dev/node-01 /etc/elasticsearch/current
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /var/lib/elasticsearch/identity-sau-main-dev/node-01 /var/lib/elasticsearch/current
++ 01-setup-directories.sh:4:main: NODE_DOMAIN=search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com
++ 01-setup-directories.sh:4:main: info 'Using existing node-01: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.186)'
++ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Using existing node-01: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.186)'
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh: line 13: printf: write error: Broken pipe
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /etc/elasticsearch/identity-sau-main-dev/node-01/certs
[2026-01-03 07:30:27 UTC] USER=www-data EUID=0 PID=2956838 ACTION=fsop ARGS=ln -sfn /var/lib/elasticsearch/identity-sau-main-dev/node-01 /var/lib/elasticsearch/current
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /var/log/elasticsearch/identity-sau-main-dev/node-01 /var/log/elasticsearch/current
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /var/lib/elasticsearch/identity-sau-main-dev/node-01/tmp
[2026-01-03 07:30:27 UTC] USER=www-data EUID=0 PID=2956855 ACTION=fsop ARGS=ln -sfn /var/log/elasticsearch/identity-sau-main-dev/node-01 /var/log/elasticsearch/current
+ 01-setup-directories.sh:4:main: (( i++ ))
+ 01-setup-directories.sh:4:main: (( i<=nodes ))
+ 01-setup-directories.sh:4:main: success 'Directory structure created for '\''identity-sau-main-dev'\'' with 1 node(s).'
+ 01-setup-directories.sh:4:main: printf '[ OK ] %s\n' 'Directory structure created for '\''identity-sau-main-dev'\'' with 1 node(s).'
[ OK ] Directory structure created for 'identity-sau-main-dev' with 1 node(s).
Executing: steps/02-install-dependencies.sh
=== Step 2: Installing/Validating Elasticsearch (latest) ===
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /var/log/elasticsearch/identity-sau-main-dev/node-01
++ 01-setup-directories.sh:4:main: id -u elasticsearch
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-sau-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/identity-sau-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chown -R elasticsearch:elasticsearch /var/log/elasticsearch/identity-sau-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chmod 0750 /etc/elasticsearch/identity-sau-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chmod 0750 /var/lib/elasticsearch/identity-sau-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chmod 0750 /var/log/elasticsearch/identity-sau-main-dev/node-01
++ 01-setup-directories.sh:4:main: info 'Created dirs for identity-sau-main-dev/node-01 @ 10.100.1.186'
++ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Created dirs for identity-sau-main-dev/node-01 @ 10.100.1.186'
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh: line 13: printf: write error: Broken pipe
++ 01-setup-directories.sh:4:main: printf '%s\n' 10.100.1.186
/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh: line 58: printf: write error: Broken pipe
++ 01-setup-directories.sh:4:main: printf '%s\n' search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com
/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh: line 59: printf: write error: Broken pipe
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] Cleaning dpkg/apt locks...
[2026-01-03 07:30:28 UTC] USER=www-data EUID=0 PID=2956955 ACTION=cleanup-dpkg-locks ARGS=
steps/02-install-dependencies.sh: line 16: 2956953 Killed                  command sudo -n "$WRAP" cleanup-dpkg-locks
[2026-01-03 07:30:28 UTC] USER=www-data EUID=0 PID=2956964 ACTION=fsop ARGS=mkdir -p /etc/apt/keyrings
[2026-01-03 07:30:28 UTC] USER=www-data EUID=0 PID=2956973 ACTION=fsop ARGS=chmod 0755 /etc/apt/keyrings
[INFO] apt-get update…
[2026-01-03 07:30:28 UTC] USER=www-data EUID=0 PID=2956983 ACTION=pkg ARGS=update
Hit:1 http://apt.postgresql.org/pub/repos/apt jammy-pgdg InRelease
Hit:2 https://packages.confluent.io/deb/7.6 stable InRelease
Hit:3 https://artifacts.elastic.co/packages/8.x/apt stable InRelease
Hit:4 https://packages.microsoft.com/repos/azure-cli jammy InRelease
Hit:5 https://apt.grafana.com stable InRelease
Hit:6 https://mirrors.edge.kernel.org/ubuntu jammy InRelease
Hit:7 https://mirrors.edge.kernel.org/ubuntu jammy-backports InRelease
Hit:8 https://mirrors.edge.kernel.org/ubuntu jammy-security InRelease
Hit:9 https://mirrors.edge.kernel.org/ubuntu jammy-updates InRelease
Hit:10 https://deb.nodesource.com/node_22.x nodistro InRelease
Hit:11 https://packages.clickhouse.com/deb stable InRelease
Hit:12 https://ppa.launchpadcontent.net/ondrej/php/ubuntu jammy InRelease
Hit:13 https://repos.citusdata.com/community/ubuntu jammy InRelease
Reading package lists...
[INFO] Installed version : 8.19.9
[INFO] Candidate version : 8.19.9
✅ Elasticsearch already at latest (or only) available version.
✅ Elasticsearch installation validated.
🎉 Dependencies installed and up-to-date.
Executing: steps/03-create-env-configs.sh
=== Step 3: Creating environment configurations (master + nodes, TLS, units) ===
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Configuring env: identity-sau-main-dev (nodes: 1, http: 9200, transport: 9300)
Using heap size: 1024m per node
[2026-01-03 07:30:49 UTC] USER=www-data EUID=0 PID=2957967 ACTION=fsop ARGS=chown root:root /etc/default/elasticsearch
[2026-01-03 07:30:49 UTC] USER=www-data EUID=0 PID=2957976 ACTION=fsop ARGS=chmod 0644 /etc/default/elasticsearch
[2026-01-03 07:30:49 UTC] USER=www-data EUID=0 PID=2957994 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/jvm.options
[2026-01-03 07:30:49 UTC] USER=www-data EUID=0 PID=2958003 ACTION=fsop ARGS=chmod 0644 /etc/elasticsearch/jvm.options
[2026-01-03 07:30:49 UTC] USER=www-data EUID=0 PID=2958021 ACTION=fsop ARGS=mkdir -p /etc/systemd/system.conf.d /etc/systemd/user.conf.d
[2026-01-03 07:30:49 UTC] USER=www-data EUID=0 PID=2958048 ACTION=passthru ARGS=systemctl daemon-reload
Current max_map_count: 262144
Current swappiness:   1
[2026-01-03 07:30:50 UTC] USER=www-data EUID=0 PID=2958125 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/log4j2.properties
[2026-01-03 07:30:50 UTC] USER=www-data EUID=0 PID=2958134 ACTION=fsop ARGS=chmod 0644 /etc/elasticsearch/log4j2.properties
[2026-01-03 07:30:50 UTC] USER=www-data EUID=0 PID=2958143 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-sau-main-dev/template
[2026-01-03 07:30:50 UTC] USER=www-data EUID=0 PID=2958152 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/identity-sau-main-dev /etc/elasticsearch/identity-sau-main-dev/template
[2026-01-03 07:30:50 UTC] USER=www-data EUID=0 PID=2958161 ACTION=fsop ARGS=chmod 0755 /etc/elasticsearch/identity-sau-main-dev
[2026-01-03 07:30:50 UTC] USER=www-data EUID=0 PID=2958170 ACTION=fsop ARGS=cp /etc/elasticsearch/jvm.options /etc/elasticsearch/identity-sau-main-dev/template/jvm.options
[INFO] 🌐 Registering general environment domain: identity-sau-main-dev.fastorder.com
[INFO]   Allocated VM IP: 10.100.1.50 for general environment
[INFO]   Configuring VM IP 10.100.1.50 on network interface...
[WARNING]   VM IP may already be configured or need manual setup
[WARNING]   Warning: VM IP 10.100.1.50 not found on network interfaces
[ OK ] ✅ Registered general domain identity-sau-main-dev.fastorder.com -> 10.100.1.50
[ OK ] ✅ DNS resolution verified for identity-sau-main-dev.fastorder.com
[INFO] → Configuring identity-sau-main-dev-node-01 (10.100.1.186) roles=[ master, data, data_hot, data_content, ingest ]
[2026-01-03 07:30:51 UTC] USER=www-data EUID=0 PID=2958262 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-sau-main-dev/node-01/certs /var/lib/elasticsearch/identity-sau-main-dev/node-01/tmp /var/log/elasticsearch/identity-sau-main-dev/node-01
[2026-01-03 07:30:51 UTC] USER=www-data EUID=0 PID=2958271 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-sau-main-dev/node-01
[2026-01-03 07:30:51 UTC] USER=www-data EUID=0 PID=2958280 ACTION=fsop ARGS=chmod 0750 /etc/elasticsearch/identity-sau-main-dev/node-01 /var/lib/elasticsearch/identity-sau-main-dev/node-01 /var/log/elasticsearch/identity-sau-main-dev/node-01
[2026-01-03 07:30:51 UTC] USER=www-data EUID=0 PID=2958289 ACTION=fsop ARGS=cp /etc/elasticsearch/identity-sau-main-dev/template/jvm.options /etc/elasticsearch/identity-sau-main-dev/node-01/jvm.options
[2026-01-03 07:30:51 UTC] USER=www-data EUID=0 PID=2958298 ACTION=fsop ARGS=sed -i s/^-Xms.*/-Xms1024m/ /etc/elasticsearch/identity-sau-main-dev/node-01/jvm.options
[2026-01-03 07:30:51 UTC] USER=www-data EUID=0 PID=2958307 ACTION=fsop ARGS=sed -i s/^-Xmx.*/-Xmx1024m/ /etc/elasticsearch/identity-sau-main-dev/node-01/jvm.options
[2026-01-03 07:30:51 UTC] USER=www-data EUID=0 PID=2958326 ACTION=fsop ARGS=cp /etc/elasticsearch/log4j2.properties /etc/elasticsearch/identity-sau-main-dev/node-01/log4j2.properties
[2026-01-03 07:30:51 UTC] USER=www-data EUID=0 PID=2958364 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-sau-main-dev/node-01
[2026-01-03 07:30:51 UTC] USER=www-data EUID=0 PID=2958373 ACTION=fsop ARGS=chmod 0644 /etc/elasticsearch/identity-sau-main-dev/node-01/elasticsearch.yml
[2026-01-03 07:30:51 UTC] USER=www-data EUID=0 PID=2958391 ACTION=fsop ARGS=chmod 0644 /etc/default/elasticsearch-identity-sau-main-dev-node-01
[2026-01-03 07:30:51 UTC] USER=www-data EUID=0 PID=2958402 ACTION=fsop ARGS=sed -i /[[:space:]]search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com\([[:space:]]\|$\)/d /etc/hosts
[INFO]   → Also added short domain: search-identity-sau-main-dev.fastorder.com
[INFO] ✔ Created configuration for identity-sau-main-dev/node-01 (roles=single-node)
[2026-01-03 07:30:51 UTC] USER=www-data EUID=0 PID=2958432 ACTION=fsop ARGS=sed -i /[[:space:]]search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com\([[:space:]]\|$\)/d /etc/hosts
[INFO] ✔ Registered master domain search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com -> 10.100.1.186 (points to node-01)
[INFO] Cleaning up legacy non-templated elasticsearch-*.service units (if any)...
[INFO] No legacy units found.
[INFO] Base template exists: elasticsearch@.service
[ OK ] Created unit: elasticsearch@identity-sau-main-dev-node-01.service
[2026-01-03 07:30:52 UTC] USER=www-data EUID=0 PID=2958484 ACTION=passthru ARGS=systemctl daemon-reload

[ OK ] Environment configurations (master + nodes with TLS) created successfully!
[INFO] Environment: identity-sau-main-dev
[INFO] Nodes: 1
[INFO] HTTP Port: 9200
[INFO] Transport Port: 9300
[INFO] Heap Size: 1024m per node
[INFO] Master: search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.182)
[INFO]   node-01: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.186)
[INFO] Systemd units prepared (not started). Start sequence runs in Step 7.
Executing: steps/04-start-clusters.sh
=== Step 7: Starting Elasticsearch clusters (with waits) ===
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Starting Elasticsearch cluster for environment: identity-sau-main-dev (1 nodes)
[INFO] === Ensuring VM IP services are started ===
[WARNING] VM IP service vm-ip-10-100-1-186.service not found - IP might not persist
[INFO] Manually configuring IP: 10.100.1.186
[2026-01-03 07:30:56 UTC] USER=www-data EUID=0 PID=2958576 ACTION=configure-network-interface ARGS=lo:search01 10.100.1.186
✓ lo:search01 <- 10.100.1.186
[INFO] Cleaning up any existing Elasticsearch processes and lock files...
[2026-01-03 07:30:56 UTC] USER=www-data EUID=0 PID=2958627 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@identity-sau-main-dev-node-01.service
[INFO] Stopping Elasticsearch services for environment: identity-sau-main-dev ...
[INFO] No active Elasticsearch services found for environment: identity-sau-main-dev
[INFO] Removing lock files from: /var/lib/elasticsearch/identity-sau-main-dev/node-01
[2026-01-03 07:30:56 UTC] USER=www-data EUID=0 PID=2958697 ACTION=fsop ARGS=find /var/lib/elasticsearch/identity-sau-main-dev/node-01 -name *.lock -delete
[2026-01-03 07:30:56 UTC] USER=www-data EUID=0 PID=2958759 ACTION=fsop ARGS=find /var/lib/elasticsearch/identity-sau-main-dev/node-01 -name node.lock -delete
[2026-01-03 07:30:56 UTC] USER=www-data EUID=0 PID=2958816 ACTION=fsop ARGS=find /var/lib/elasticsearch/identity-sau-main-dev/node-01 -name _state -type d -exec rm -rf {} +
[2026-01-03 07:30:56 UTC] USER=www-data EUID=0 PID=2958839 ACTION=fsop ARGS=find /tmp -name *elasticsearch*identity-sau-main-dev-node-01* -delete
[ OK ] Cleanup completed for environment: identity-sau-main-dev
[INFO] Checking for port conflicts before starting Elasticsearch...
[INFO] Checking for port conflicts on 10.100.1.182:9200 and 10.100.1.182:9300...
[ OK ] ✓ Ports 9200 and 9300 are available on 10.100.1.182
[INFO] Ensuring correct ownership of Elasticsearch directories...
[2026-01-03 07:30:58 UTC] USER=www-data EUID=0 PID=2958881 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch
[2026-01-03 07:30:59 UTC] USER=www-data EUID=0 PID=2958890 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /var/lib/elasticsearch
[2026-01-03 07:31:02 UTC] USER=www-data EUID=0 PID=2958913 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /var/log/elasticsearch
[ OK ] Directory ownership fixed
[INFO] === Starting Elasticsearch Nodes ===
[INFO] Starting 1 node(s) for cluster
▶ Starting elasticsearch@identity-sau-main-dev-node-01.service (search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200)
[2026-01-03 07:31:02 UTC] USER=www-data EUID=0 PID=2958926 ACTION=passthru ARGS=systemctl is-enabled --quiet elasticsearch@identity-sau-main-dev-node-01.service
[2026-01-03 07:31:03 UTC] USER=www-data EUID=0 PID=2958980 ACTION=passthru ARGS=systemctl start elasticsearch@identity-sau-main-dev-node-01.service
⏳ Waiting for TCP search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200 to be accessible (timeout 360s)...
✅ Port 9200 is accessible on search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com.
⏳ Waiting for ES HTTP readiness on http://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200 (timeout 300s)...
[ OK ] ES HTTP ready on search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[ OK ] elasticsearch@identity-sau-main-dev-node-01.service is up and answering HTTP on search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Node 1 started successfully
[INFO] Cluster with 1 node(s) started successfully
⏳ Waiting for the cluster to elect master and settle...
⏳ Waiting for cluster health=green via search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200 (timeout 300s)...
[ OK ] Cluster is GREEN (nodes="number_of_nodes") on search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
[ OK ] Cluster identity-sau-main-dev is healthy and green!

[INFO] === Final Status Check ===
[2026-01-03 07:31:49 UTC] USER=www-data EUID=0 PID=2959348 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@identity-sau-main-dev-node-01.service
[ OK ] elasticsearch@identity-sau-main-dev-node-01.service is ACTIVE (search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200)
  └── HTTP responding on search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200 ✓
[ OK ] All 1 node(s) in environment 'identity-sau-main-dev' are running successfully!
[INFO] Node endpoints:
  - http://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200


[ OK ] Elasticsearch cluster started successfully!
[INFO] Environment: identity-sau-main-dev
[INFO] Nodes: 1
[INFO] Cluster endpoints:
  - http://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === Quick Cluster Information ===
Cluster Name: fastorder-identity-sau-main-dev
Node Name: identity-sau-main-dev-node-01
Version: 8.19.9
Architecture: 1 node(s), each on default port 9200

Cluster with 1 node(s) started successfully (each on port 9200)
Executing: steps/05-verify-setup.sh
=== Step 8: Verifying setup (with retries) ===
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Verifying environment: identity-sau-main-dev (1 nodes, Single-node)
Main HTTP endpoint: http://search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
Testing network connectivity to search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200...
✓ Domain connection available
Testing HTTP response...
[ OK ] ✓ identity-sau-main-dev is responding on search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200

[INFO] === Cluster Health ===
{
  "cluster_name" : "fastorder-identity-sau-main-dev",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
[ OK ] Cluster status: GREEN ("number_of_nodes" nodes)

[INFO] === Cluster Nodes ===
ip           heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.100.1.186           53          98  38    4.67    3.51    32.52 dhims     *      identity-sau-main-dev-node-01

[INFO] === Single-Node Service Verification ===
Testing coordinator service (search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200)...
  ✓ Coordinator HTTP responding on search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
    Name: identity-sau-main-dev-node-01, Version: 8.19.9

[INFO] === Cluster State Summary ===
Using jq for formatted output:
jq parsing failed

[ OK ] === Verification Summary ===
[INFO] Environment: identity-sau-main-dev
[INFO] Nodes configured: 1
[INFO] Main endpoint: http://search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
[INFO] Service endpoint: http://search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200

[INFO] === Final Connectivity Test ===
  ✓ Coordinator: search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200

[ OK ] Single-node cluster is responding successfully!
[ OK ] Elasticsearch cluster 'identity-sau-main-dev' verification completed successfully!
Executing: steps/06-confirm-working.sh
=== Step 9: Comprehensive Cluster Verification (gated) ===
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
========================================
🔍 Verifying Environment: identity-sau-main-dev (1 nodes)
========================================
Domain: search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com
Environment: identity-sau-main-dev
Nodes: 1

[INFO] Testing network connectivity...
Setup type: Single-node
Testing endpoint: search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
[ OK ] ✓ Using domain: search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com


📡 Coordinator Service (elasticsearch@identity-sau-main-dev-node-01.service)
Endpoint: search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
--------------------------------
[2026-01-03 07:31:51 UTC] USER=www-data EUID=0 PID=2959504 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@identity-sau-main-dev-node-01.service
✅ Service: ACTIVE
⏳ Waiting for TCP search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200 to be accessible (timeout 5s)...
✅ Port 9200 is accessible on search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com.
✅ Port: LISTENING on search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
✅ HTTP: RESPONDING on search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
   Node name: identity-sau-main-dev-node-01

========================================
🏥 Cluster Health Check
========================================
Cluster Name: fastorder-identity-sau-main-dev
Nodes Count: "number_of_nodes"
Status: green
[ OK ] ✅ Cluster status: GREEN (healthy)

Full cluster health:
{
  "cluster_name" : "fastorder-identity-sau-main-dev",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

========================================
📊 Final Verification Results
========================================
[ OK ] ✅ Comprehensive verification PASSED!
[ OK ] Environment 'identity-sau-main-dev' with 1 nodes is fully operational

📋 QUICK DIAGNOSTIC COMMANDS:
----------------------------------------
# Test cluster endpoints:
curl http://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200

# Check cluster health:
curl http://search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200/_cluster/health?pretty

# Check nodes info:
curl http://search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200/_cat/nodes?v

# Check all Elasticsearch ports:
sudo ss -tlnp | grep java

# Check systemd service status:
sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status elasticsearch@identity-sau-main-dev-node-01.service

# View recent logs:
sudo journalctl -u elasticsearch@identity-sau-main-dev-node-01.service -f

[INFO] Environment: identity-sau-main-dev
[INFO] Nodes: 1
[INFO] Port: 9200 (default Elasticsearch port)
[INFO] Coordinator endpoint: http://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200

=== Elasticsearch HTTP Setup completed successfully! ===
Environment:  (1 nodes)
Port: 9200 (default Elasticsearch port)

✅ Coordinator endpoint: http://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200

Quick test commands:
  curl http://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
  curl http://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty


✓ Step 1 completed successfully!

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 2: Executing Make Https
Folder: 02-make-https
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

=== Elasticsearch HTTPS Setup ===
Configure HTTPS/SSL for Elasticsearch cluster
[INFO] Using web-provided environment: identity-sau-main-dev
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: 
Nodes: 1
Node: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.186)
Port: 9200 (default port)

Executing: steps/01-generate-ssl-certificates.sh
==================================================================
STEP 1: Generate SSL certificates for Elasticsearch transport
==================================================================
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Domain: identity-sau-main-dev.fastorder.com
Environment: identity-sau-main-dev
Nodes: 1
Per-node VM IPs and domains:
  Node 1: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.186)
Port: 9200 (default port for all nodes)

=== Generating SSL certificates for ES transport ===
[INFO] Generating certificates for environment: identity-sau-main-dev (1 nodes)
[INFO] Configuring certificates for 1 node(s)
[INFO] Certificate storage: /etc/fastorder/elasticsearch/certs/identity-sau-main-dev
[2026-01-03 07:31:53 UTC] USER=www-data EUID=0 PID=2959597 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/temp-2959574
[2026-01-03 07:31:53 UTC] USER=www-data EUID=0 PID=2959606 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/temp-2959574
[2026-01-03 07:31:53 UTC] USER=www-data EUID=0 PID=2959615 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/temp-2959574
[2026-01-03 07:31:53 UTC] USER=www-data EUID=0 PID=2959634 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/temp-2959574/instances.yml
[INFO] Creating certificate instances configuration...
  Adding node: identity-sau-main-dev-node-01 (search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com, 10.100.1.186)

[INFO] Certificate instances configuration:
instances:
  - name: identity-sau-main-dev-node-01
    dns: [ "identity-sau-main-dev-node-01", "localhost", "search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com" ]
    ip:  [ "10.100.1.186", "127.0.0.1" ]

[INFO] Creating Certificate Authority for identity-sau-main-dev...
[2026-01-03 07:31:53 UTC] USER=www-data EUID=0 PID=2959659 ACTION=fsop ARGS=mkdir -p /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01/certs
[2026-01-03 07:31:53 UTC] USER=www-data EUID=0 PID=2959668 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/fastorder/elasticsearch/certs/identity-sau-main-dev
[2026-01-03 07:31:53 UTC] USER=www-data EUID=0 PID=2959677 ACTION=fsop ARGS=chmod -R 755 /etc/fastorder/elasticsearch/certs/identity-sau-main-dev
[2026-01-03 07:31:53 UTC] USER=www-data EUID=0 PID=2959686 ACTION=fsop ARGS=rm -f /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01/certs/identity-sau-main-dev-ca.zip
yes: standard output: Broken pipe
[ OK ] ✓ CA certificate created

[INFO] Creating node certificates for identity-sau-main-dev...
yes: standard output: Broken pipe
[ OK ] ✓ Node certificates created

[INFO] Distributing certificates...
  Configuring certificates for node 1 (identity-sau-main-dev-node-01)...
[2026-01-03 07:32:00 UTC] USER=www-data EUID=0 PID=2960007 ACTION=fsop ARGS=cp /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01/certs/ca/ca.crt /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01/ca.crt
[2026-01-03 07:32:00 UTC] USER=www-data EUID=0 PID=2960022 ACTION=fsop ARGS=cp /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01/certs/nodes/identity-sau-main-dev-node-01.crt /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01/
[2026-01-03 07:32:00 UTC] USER=www-data EUID=0 PID=2960031 ACTION=fsop ARGS=cp /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01/certs/nodes/identity-sau-main-dev-node-01.key /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01/
[2026-01-03 07:32:00 UTC] USER=www-data EUID=0 PID=2960040 ACTION=fsop ARGS=chmod 644 /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01/identity-sau-main-dev-node-01.crt
[2026-01-03 07:32:00 UTC] USER=www-data EUID=0 PID=2960049 ACTION=fsop ARGS=chmod 600 /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01/identity-sau-main-dev-node-01.key
[ OK ]   ✓ Certificates copied for identity-sau-main-dev-node-01
[2026-01-03 07:32:00 UTC] USER=www-data EUID=0 PID=2960058 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01
[2026-01-03 07:32:00 UTC] USER=www-data EUID=0 PID=2960067 ACTION=fsop ARGS=find /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01/certs -type f -name *.key -exec chmod 600 {} ;
[2026-01-03 07:32:00 UTC] USER=www-data EUID=0 PID=2960078 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01/certs
[2026-01-03 07:32:00 UTC] USER=www-data EUID=0 PID=2960087 ACTION=fsop ARGS=rm -rf /etc/elasticsearch/temp-2959574
[ OK ] ✓ Certificates ready for environment: identity-sau-main-dev

[ OK ] ✓ SSL certificate generation completed successfully!
[INFO] Environment: identity-sau-main-dev
[INFO] Nodes configured: 1
[INFO] Per-node VM IPs and domains (each with default port 9200):
  Node 1: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.186)
[INFO] Certificate directory: /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01/certs

[INFO] === Certificate Summary ===
CA Certificate: /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01/certs/ca/ca.crt
Node Certificates:
  - identity-sau-main-dev-node-01: /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01/

[INFO] === Verification Commands ===
# Verify CA certificate:
openssl x509 -in /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01/certs/ca/ca.crt -text -noout

# Verify node certificates:
openssl x509 -in /etc/fastorder/elasticsearch/certs/identity-sau-main-dev/node-01/identity-sau-main-dev-node-01.crt -text -noout

[INFO] Next: Configure transport SSL in Elasticsearch configuration files
Executing: steps/02-enable-security-transport.sh
==================================================================
STEP 2: Enable security with transport SSL
==================================================================
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Nodes: 1
Node: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.186)

[INFO] === Single-Node Setup ===
[INFO] Enabling security (xpack.security.enabled: true)
[2026-01-03 07:32:01 UTC] USER=www-data EUID=0 PID=2960144 ACTION=fsop ARGS=sed -i /^xpack.security.enabled:/d /etc/elasticsearch/identity-sau-main-dev/node-01/elasticsearch.yml
[INFO] Disabling transport SSL (not needed for single-node)
[2026-01-03 07:32:01 UTC] USER=www-data EUID=0 PID=2960163 ACTION=fsop ARGS=sed -i /^xpack.security.transport.ssl.enabled:/d /etc/elasticsearch/identity-sau-main-dev/node-01/elasticsearch.yml

[ OK ] ==================================================================
[ OK ] Security and Transport SSL Configuration Complete
[ OK ] ==================================================================
[INFO] Environment: identity-sau-main-dev
[INFO] Nodes: 1
[INFO] Security enabled: true
[INFO] Transport SSL enabled: false (not required for single-node)

[INFO] === Next Step ===
Restart services to apply security configuration (step 04)
Executing: steps/03-http-ssl.sh
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
==================================================================
Direct HTTPS Configuration (native TLS, PEM)
==================================================================
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Nodes: 1
Node: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.186)
Port: 9200 (default port)
Domain: identity-sau-main-dev.fastorder.com

[INFO] === Single-Node Direct HTTPS Setup ===
🔐 Generated PKCS12 password: iFbh2YfB... (32 chars)
[INFO] Using first node configuration: /etc/elasticsearch/identity-sau-main-dev/node-01
[2026-01-03 07:32:03 UTC] USER=www-data EUID=0 PID=2960226 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-sau-main-dev/node-01
[2026-01-03 07:32:03 UTC] USER=www-data EUID=0 PID=2960235 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-sau-main-dev/node-01
[INFO] Checking prerequisites...
[ OK ] ✓ Prerequisites verified
[INFO] Setting up temporary directories...
[2026-01-03 07:32:03 UTC] USER=www-data EUID=0 PID=2960246 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs
[2026-01-03 07:32:03 UTC] USER=www-data EUID=0 PID=2960255 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs
[2026-01-03 07:32:03 UTC] USER=www-data EUID=0 PID=2960264 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-sau-main-dev
[2026-01-03 07:32:03 UTC] USER=www-data EUID=0 PID=2960273 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-sau-main-dev/node-01
[2026-01-03 07:32:03 UTC] USER=www-data EUID=0 PID=2960282 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs
[2026-01-03 07:32:03 UTC] USER=www-data EUID=0 PID=2960291 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out
[ OK ] ✓ Directories created

[INFO] Building certificate instances configuration...
[INFO] Certificate instances configuration:
instances:
  - name: "identity-sau-main-dev-http"
    dns:  [ "localhost", "web-03", "search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com", "identity-sau-main-dev-node-01.fastorder.com", "search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com", "search-identity-sau-main-dev.fastorder.com", "identity-sau-main-dev-node-01.local" ]
    ip:   [ "10.100.1.186", "127.0.0.1", "::1" ]
[ OK ] ✓ Instances configuration created

[INFO] Generating HTTP Certificate Authority...
[2026-01-03 07:32:03 UTC] USER=www-data EUID=0 PID=2960309 ACTION=fsop ARGS=rm -f /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/http-ca.zip /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out/http-certs.zip
[2026-01-03 07:32:03 UTC] USER=www-data EUID=0 PID=2960318 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs
[2026-01-03 07:32:03 UTC] USER=www-data EUID=0 PID=2960327 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs
[2026-01-03 07:32:06 UTC] USER=www-data EUID=0 PID=2960380 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/http-ca.zip
Archive:  /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/http-ca.zip
  inflating: /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/ca/ca.crt  
  inflating: /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/ca/ca.key  
[ OK ] ✓ HTTP CA generated successfully

[INFO] Generating per-node HTTP certificates...
[2026-01-03 07:32:06 UTC] USER=www-data EUID=0 PID=2960392 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out
[2026-01-03 07:32:06 UTC] USER=www-data EUID=0 PID=2960402 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out
[2026-01-03 07:32:09 UTC] USER=www-data EUID=0 PID=2960450 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out/http-certs.zip
[2026-01-03 07:32:09 UTC] USER=www-data EUID=0 PID=2960459 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out/http
Archive:  /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out/http-certs.zip
  inflating: /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out/http/identity-sau-main-dev-http/identity-sau-main-dev-http.crt  
  inflating: /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out/http/identity-sau-main-dev-http/identity-sau-main-dev-http.key  
[ OK ] ✓ HTTP certificates generated successfully

[INFO] Installing certificates and configuring services...
[INFO] Configuring main service for single-node HTTPS...
[2026-01-03 07:32:09 UTC] USER=www-data EUID=0 PID=2960471 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-sau-main-dev/node-01/certs
[2026-01-03 07:32:09 UTC] USER=www-data EUID=0 PID=2960480 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-sau-main-dev/node-01/certs
[2026-01-03 07:32:09 UTC] USER=www-data EUID=0 PID=2960489 ACTION=fsop ARGS=cp /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out/http/identity-sau-main-dev-http/identity-sau-main-dev-http.crt /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http.crt
[2026-01-03 07:32:09 UTC] USER=www-data EUID=0 PID=2960498 ACTION=fsop ARGS=cp /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out/http/identity-sau-main-dev-http/identity-sau-main-dev-http.key /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http.key
[2026-01-03 07:32:09 UTC] USER=www-data EUID=0 PID=2960507 ACTION=fsop ARGS=cp /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/ca/ca.crt /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt
[2026-01-03 07:32:09 UTC] USER=www-data EUID=0 PID=2960516 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/identity-sau-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-01-03 07:32:09 UTC] USER=www-data EUID=0 PID=2960525 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-sau-main-dev/node-01/certs
[2026-01-03 07:32:09 UTC] USER=www-data EUID=0 PID=2960534 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http.key
[2026-01-03 07:32:09 UTC] USER=www-data EUID=0 PID=2960543 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http.crt /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt /etc/elasticsearch/identity-sau-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-01-03 07:32:09 UTC] USER=www-data EUID=0 PID=2960552 ACTION=fsop ARGS=sed -i -E -e /^\s*xpack\.security\.http\.ssl(\..*)?\s*:/d /etc/elasticsearch/identity-sau-main-dev/node-01/elasticsearch.yml
[2026-01-03 07:32:09 UTC] USER=www-data EUID=0 PID=2960561 ACTION=fsop ARGS=sed -i /^# --- BEGIN direct HTTPS (managed, PEM) ---$/,/^# --- END direct HTTPS (managed, PEM) ---$/d /etc/elasticsearch/identity-sau-main-dev/node-01/elasticsearch.yml
[2026-01-03 07:32:09 UTC] USER=www-data EUID=0 PID=2960570 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/identity-sau-main-dev/node-01/certs/fastorder_ra_root.crt
YAML: /etc/elasticsearch/identity-sau-main-dev/node-01/elasticsearch.yml
[ OK ]   ✓ Main service configured with HTTPS
[2026-01-03 07:32:09 UTC] USER=www-data EUID=0 PID=2960589 ACTION=fsop ARGS=rm -rf /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients
[2026-01-03 07:32:09 UTC] USER=www-data EUID=0 PID=2960598 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients
[2026-01-03 07:32:09 UTC] USER=www-data EUID=0 PID=2960607 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients
Archive:  /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client.zip
   creating: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/
  inflating: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.crt  
  inflating: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.key  
[INFO] Creating P12 keystore for es-client...
[2026-01-03 07:32:12 UTC] USER=www-data EUID=0 PID=2960652 ACTION=fsop ARGS=mv /tmp/es-client-2960197.p12 /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.p12
[ OK ] ✓ Created P12 keystore: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-01-03 07:32:12 UTC] USER=www-data EUID=0 PID=2960661 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients
[2026-01-03 07:32:12 UTC] USER=www-data EUID=0 PID=2960670 ACTION=fsop ARGS=chmod 640 /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.key
[2026-01-03 07:32:12 UTC] USER=www-data EUID=0 PID=2960679 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-01-03 07:32:12 UTC] USER=www-data EUID=0 PID=2960688 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.crt /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt
[INFO] Saving keystore passwords to secrets vault...
🔑 Configuring AWS credentials for secrets vault...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[INFO] 🔐 Vaulting search passwords to remote backend...
✅ Passwords vaulted to remote backend
✓ Keystore passwords saved to secrets vault: search/identity-sau-main-dev/keystore-passwords

[INFO] === Installing CA Certificate for Users ===
[INFO] HOME not set, skipping user CA installation

✓ Direct HTTPS configuration completed for environment: identity-sau-main-dev
[INFO] All services now serve HTTPS using PEM certificates
[INFO] Network binding: 10.100.1.186
[INFO] HTTPS endpoint: https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === Certificate Summary ===
CA Certificate: /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/ca/ca.crt
Certificate Directory: /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs
Main service certificates: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/

[INFO] === Next Steps ===
1. Restart Elasticsearch services to apply HTTPS configuration
2. Test HTTPS connectivity: curl https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
3. Verify certificates: openssl s_client -connect search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === SSL Certificate Information ===
• CA certificate installed for user - curl will work without -k flag
• Each certificate includes node-specific domain, VM IP, and localhost in Subject Alternative Names
• Certificates are valid for 3 years from generation date

[WARNING] Important: You'll need to restart Elasticsearch services for HTTPS to take effect
Executing: steps/04-restart-systemd-services.sh
==================================================================
STEP 4 (STRICT): Restart systemd services and verify secure health
==================================================================
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: identity-sau-main-dev
Nodes: 1
Per-node endpoints (all use default port 9200):
  Node 1: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.186)

[INFO] Building service list for environment: identity-sau-main-dev (1 nodes)
  - elasticsearch@identity-sau-main-dev-node-01.service (port 9200)

[INFO] Will restart 1 service(s) for environment: identity-sau-main-dev
[2026-01-03 07:32:18 UTC] USER=www-data EUID=0 PID=2960777 ACTION=passthru ARGS=systemctl daemon-reload

[INFO] === Ensuring VM IPs are configured correctly ===
[INFO] Configuring 10.100.1.186 on eth0:1 for node-01...
[2026-01-03 07:32:19 UTC] USER=www-data EUID=0 PID=2960827 ACTION=configure-network-interface ARGS=eth0:1 10.100.1.186
✓ eth0:1 <- 10.100.1.186

[INFO] === Ensuring transport SSL certificates for all nodes ===
[INFO] ✓ Transport certificates already exist for node-01

[INFO] === Restarting Services ===
↻ Restarting elasticsearch@identity-sau-main-dev-node-01.service ...
[2026-01-03 07:32:19 UTC] USER=www-data EUID=0 PID=2960837 ACTION=passthru ARGS=systemctl restart elasticsearch@identity-sau-main-dev-node-01.service
[2026-01-03 07:32:23 UTC] USER=www-data EUID=0 PID=2960904 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@identity-sau-main-dev-node-01.service
[ OK ] elasticsearch@identity-sau-main-dev-node-01.service is active
[INFO] Waiting 10s for Elasticsearch to start listening on ports...

[INFO] === Waiting for STRICT Secure Cluster Health ===
[INFO] Waiting for port 9200 on 10.100.1.186 (timeout 120s)...
[INFO] Waiting for cluster to form and be ready for write operations...
✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
[INFO] Cluster stable and ready for operations (20 consecutive healthy responses over 40s)

✓ Retrieved password from AWS Secrets Manager
[INFO] Testing cluster at: https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Using SSL CA certificate: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt
[INFO] Using client cert/key for mTLS
[INFO] Using client cert/key for mTLS: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.key
[INFO]   ⏳ waiting for secure cluster health (require 200) at https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200 (timeout 30s)...
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] 401 (auth required) — creds OK at TLS, waiting for health 200…
[WARNING] Cluster did not become healthy (secure 200) within 30s
[WARNING] Initial authentication failed - password may not be set in Elasticsearch yet
[WARNING] Running password setup to set/reset Elasticsearch password...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
╔════════════════════════════════════════════════════════════╗
║   Elasticsearch Password Management via AWS Secrets MGR   ║
╚════════════════════════════════════════════════════════════╝

Environment: identity-sau-main-dev
User:        elastic
Identifier:  node-01
AWS Secret:  fastorder/search/identity/sau/main/dev/elasticsearch/node-01

Using configuration path: /etc/elasticsearch/identity-sau-main-dev/node-01 (IDENTIFIER: node-01)
Node domain: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com
HTTP port: 9200 (default Elasticsearch port)
[INFO] xpack.security.enabled already true → no restart.
[INFO] No restart needed.
[2026-01-03 07:34:10 UTC] USER=www-data EUID=0 PID=2961980 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-sau-main-dev/node-01
[2026-01-03 07:34:10 UTC] USER=www-data EUID=0 PID=2962007 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/identity-sau-main-dev/node-01/users /etc/elasticsearch/identity-sau-main-dev/node-01/users_roles
[2026-01-03 07:34:10 UTC] USER=www-data EUID=0 PID=2962016 ACTION=fsop ARGS=chmod 660 /etc/elasticsearch/identity-sau-main-dev/node-01/users /etc/elasticsearch/identity-sau-main-dev/node-01/users_roles
✓ users/users_roles present and writable
[2026-01-03 07:34:10 UTC] USER=www-data EUID=0 PID=2962025 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/identity-sau-main-dev/node-01/elasticsearch.keystore
[2026-01-03 07:34:10 UTC] USER=www-data EUID=0 PID=2962034 ACTION=fsop ARGS=chmod 660 /etc/elasticsearch/identity-sau-main-dev/node-01/elasticsearch.keystore
✓ Keystore exists: /etc/elasticsearch/identity-sau-main-dev/node-01/elasticsearch.keystore
HTTPS is enabled in configuration
✓ Found HTTP CA certificate: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt
✓ Using client certificates for mTLS
Waiting for Elasticsearch to be reachable at https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200...
✓ Elasticsearch is reachable (HTTP 401)

ES_PATH_CONF: /etc/elasticsearch/identity-sau-main-dev/node-01
HTTP URL:    https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200

Running HTTP reset (online, --batch)…
Note: Using HTTPS - tools will read SSL config from elasticsearch.yml
Command output:
Password for the [elastic] user successfully reset.
New value: JgZeXRo9EN2a7bqSBJrm
Exit status: 0
✓ HTTP reset succeeded for elastic
Storing credentials in AWS Secrets Manager: fastorder/search/identity/sau/main/dev/elasticsearch/node-01
ℹ️  Setting Elasticsearch credentials in vault: fastorder/search/identity/sau/main/dev/elasticsearch/node-01
ℹ️  Setting secret in AWS Secrets Manager: fastorder/search/identity/sau/main/dev/elasticsearch/node-01
✅ Secret updated: fastorder/search/identity/sau/main/dev/elasticsearch/node-01
✅ Elasticsearch credentials set in vault: fastorder/search/identity/sau/main/dev/elasticsearch/node-01
✓ Password stored in AWS Secrets Manager: fastorder/search/identity/sau/main/dev/elasticsearch/node-01
✓ Cache cleared for: fastorder/search/identity/sau/main/dev/elasticsearch/node-01

✓ Done. Password stored in AWS Secrets Manager: fastorder/search/identity/sau/main/dev/elasticsearch/node-01

Usage Examples:
  # Retrieve password using AWS CLI
  aws secretsmanager get-secret-value --secret-id fastorder/search/identity/sau/main/dev/elasticsearch/node-01 --region ${AWS_REGION:-me-central-1}

  # Using fastctl
  fastctl secrets get fastorder/search/identity/sau/main/dev/elasticsearch/node-01

  # Test connection
  curl -u elastic:$(fastctl secrets get fastorder/search/identity/sau/main/dev/elasticsearch/node-01 --field password) https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health
✓ Retrieved password from AWS Secrets Manager
[INFO] Retrying authentication with new password...
[INFO] Using client cert/key for mTLS: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.key
[INFO]   ⏳ waiting for secure cluster health (require 200) at https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200 (timeout 300s)...
[ OK ] Cluster health OK: green

==================================================================
[ OK ] All services restarted successfully!
[ OK ] Cluster is healthy, HTTPS-secure, and responding with 200
[INFO] Environment: identity-sau-main-dev
[INFO] Services: 1
[INFO] Endpoint: https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === Manual verification (copy/paste) ===
curl -u 'elastic:JgZeXRo9EN2a7bqSBJrm' \
  --cacert '/etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt' \
  --cert   '/etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.crt' \
  --key    '/etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.key' \
  'https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty'

[INFO] === Quick checks ===
curl -u 'elastic:JgZeXRo9EN2a7bqSBJrm' --cacert '/etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt' --cert '/etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.crt' --key '/etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.key' https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cat/nodes?v
curl -u 'elastic:JgZeXRo9EN2a7bqSBJrm' --cacert '/etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt' --cert '/etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.crt' --key '/etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.key' https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/
Executing: steps/05-test-elastic.sh
==================================================================
STEP 5: Test Elasticsearch Cluster
==================================================================
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: identity-sau-main-dev
Nodes: 1
Node: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.186)
Port: 9200 (default port)

[INFO] Using centralized test suite: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/02-make-https/lib/elasticsearch-test-suite.sh
[INFO] Executing centralized test suite with args: -v -t all --env identity-sau-main-dev -u elastic
[INFO] Using CURRENT_ENV_ID from environment: identity-sau-main-dev
[INFO] Loaded from topology.json: identity-sau-main-dev
[2026-01-03 07:34:27] Loaded environment: identity-sau-main-dev
[2026-01-03 07:34:27] Service: identity, Zone: sau, Branch: main, Env: dev
[2026-01-03 07:34:27] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-03 07:34:27] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-03 07:34:27] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
ℹ Using CURRENT_ENV_ID from environment: identity-sau-main-dev
ℹ Loaded from topology.json: identity-sau-main-dev
[2026-01-03 07:34:28] Loaded environment: identity-sau-main-dev
[2026-01-03 07:34:28] Service: identity, Zone: sau, Branch: main, Env: dev
[2026-01-03 07:34:28] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-03 07:34:28] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-03 07:34:28] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
ℹ Project root: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch
ℹ Environment:  identity-sau-main-dev
ℹ Nodes count:  1
ℹ Endpoint:      https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
ℹ Using CA:       /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt
ℹ Using mTLS:     /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.key
╔════════════════════════════════════════════╗
║    Elasticsearch Centralized Test Suite    ║
╚════════════════════════════════════════════╝

=== Authentication Test ===
✓ Loaded credentials for user elastic from AWS Secrets Manager
Curl (local): curl --cacert /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt --cert /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.crt --key /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.key  -u 'elastic:********' 'https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty'
✓ Local authentication successful (HTTP 200).
{
  "cluster_name" : "fastorder-identity-sau-main-dev",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 3,
  "active_shards" : 3,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Executing: steps/06-final-testing.sh
==================================================================
STEP 6: Final Testing and Verification
==================================================================
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: identity-sau-main-dev
Nodes: 1
Node: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.186)
Port: 9200 (default port)

[INFO] Using centralized test suite: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/02-make-https/lib/elasticsearch-test-suite.sh
[INFO] Using CURRENT_ENV_ID from environment: identity-sau-main-dev
[INFO] Loaded from topology.json: identity-sau-main-dev
[2026-01-03 07:34:31] Loaded environment: identity-sau-main-dev
[2026-01-03 07:34:31] Service: identity, Zone: sau, Branch: main, Env: dev
[2026-01-03 07:34:31] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-03 07:34:31] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-03 07:34:31] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
ℹ Using CURRENT_ENV_ID from environment: identity-sau-main-dev
ℹ Loaded from topology.json: identity-sau-main-dev
[2026-01-03 07:34:32] Loaded environment: identity-sau-main-dev
[2026-01-03 07:34:32] Service: identity, Zone: sau, Branch: main, Env: dev
[2026-01-03 07:34:32] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-03 07:34:32] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-03 07:34:32] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
ℹ Project root: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch
ℹ Environment:  identity-sau-main-dev
ℹ Nodes count:  1
ℹ Endpoint:      https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
ℹ Using CA:       /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt
ℹ Using mTLS:     /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.key
╔════════════════════════════════════════════╗
║    Elasticsearch Centralized Test Suite    ║
╚════════════════════════════════════════════╝

=== Authentication Test ===
✓ Loaded credentials for user elastic from AWS Secrets Manager
Curl (local): curl --cacert /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt --cert /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.crt --key /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.key  -u 'elastic:********' 'https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty'
✓ Local authentication successful (HTTP 200).
{
  "cluster_name" : "fastorder-identity-sau-main-dev",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 3,
  "active_shards" : 3,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Executing: steps/07-set-passwords.sh
==================================================================
STEP 7: Setting cluster passwords (bootstrap via alias)
==================================================================
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Using HTTPS with CA: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt (host: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com)
[INFO] Using centralized password setter: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/02-make-https/steps/../lib/elasticsearch-set-password.sh
[ OK ] Elastic password already valid (HTTP 200) via search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com; nothing to do.
Executing: steps/08-create-app-user.sh
==================================================================
STEP 8: Create Application User and Roles (cluster-scoped)
==================================================================
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: identity-sau-main-dev
Nodes: 1

[INFO] Using HTTPS with CA: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt (host: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com)
[ OK ] Retrieved elastic password from Vault (cluster scope).
[INFO] Configuration:
[INFO]   App User         : app_user
[INFO]   Read-only Role   : app_ro
[INFO]   Read-write Role  : app_rw
[INFO]   Index Patterns   : app-*,cdc-*,identity_sau_*,*_account_router
[INFO]   Endpoint         : https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] Creating read-only role: app_ro
[ OK ] ✓ Role app_ro ensured
[INFO] Creating read-write role: app_rw
[ OK ] ✓ Role app_rw ensured
[INFO] Creating/Updating application user: app_user
[ OK ] ✓ User app_user ensured
ℹ️  Setting Elasticsearch credentials in vault: fastorder/search/identity/sau/main/dev/elasticsearch/node-01/app_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/search/identity/sau/main/dev/elasticsearch/node-01/app_user
✅ Secret updated: fastorder/search/identity/sau/main/dev/elasticsearch/node-01/app_user
✅ Elasticsearch credentials set in vault: fastorder/search/identity/sau/main/dev/elasticsearch/node-01/app_user
[ OK ] ✓ Stored app_user password under 'node-01/app_user'
ℹ️  Setting Elasticsearch credentials in vault: fastorder/search/identity/sau/main/dev/elasticsearch/cluster/app_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/search/identity/sau/main/dev/elasticsearch/cluster/app_user
✅ Secret updated: fastorder/search/identity/sau/main/dev/elasticsearch/cluster/app_user
✅ Elasticsearch credentials set in vault: fastorder/search/identity/sau/main/dev/elasticsearch/cluster/app_user
[ OK ] ✓ Stored app_user password under 'cluster/app_user'
[INFO] Testing authentication for app_user...
[ OK ] ✓ Authentication test passed for app_user

[ OK ] ✓ Application user and roles created successfully!
[INFO] User    : app_user
[INFO] Roles   : app_ro, app_rw
[INFO] Patterns: app-*,cdc-*,identity_sau_*,*_account_router
[INFO] Endpoint: https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
Executing: steps/09-config.sh
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Nodes: 1
Node: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.186)
Port: 9200

✓ Auto mode: Cloud IMDS detected → MODE=role
[INFO] Mode: role

[INFO] AWS Region: me-central-1
[INFO] MODE=role → will purge any static S3 keys from each node keystore

[2026-01-03 07:36:08 UTC] USER=www-data EUID=0 PID=2963671 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-sau-main-dev/node-01
[INFO] • node-01 keystore cleared (role-based auth)
[2026-01-03 07:36:17 UTC] USER=www-data EUID=0 PID=2963780 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-03 07:36:18 UTC] USER=www-data EUID=0 PID=2963825 ACTION=passthru ARGS=systemctl restart elasticsearch@identity-sau-main-dev-node-01.service
✓ ✓ restarted elasticsearch@identity-sau-main-dev-node-01.service

⏳ Waiting for HTTPS readiness on https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Waiting HTTP readiness at https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/ (200/401/302)…
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
[OK] Ready: 401
⏳ Waiting for cluster health (green|yellow)
[INFO] Waiting health (green|yellow) at https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health…
[OK] 401 pre-auth received; security enabled.
✓ ✓ identity-sau-main-dev is responding via search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com

✓ ✓ AWS S3 configuration completed for environment: identity-sau-main-dev (1 nodes)
[INFO] Mode: role
[INFO] Region: me-central-1
Executing: steps/0ld-03-http-ssl.sh
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
==================================================================
STEP 5: HTTP SSL Configuration (Optional)
==================================================================
Environment: identity-sau-main-dev
Nodes: 1
Node: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.186)
Port: 9200 (default port)

[ OK ] 🚀 Auto mode/Default installation: Selecting Direct HTTPS configuration (option 1)

[ OK ] Configuring Direct HTTPS (Elasticsearch native SSL)...
──────────────────────────────────────────────────────────
[INFO] Environment: identity-sau-main-dev (1 nodes)
[INFO] Node: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.186)
[INFO] Port: 9200 (default port)
==================================================================
Direct HTTPS Configuration (native TLS, PEM)
==================================================================
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Nodes: 1
Node: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.186)
Port: 9200 (default port)
Domain: identity-sau-main-dev.fastorder.com

[INFO] === Single-Node Direct HTTPS Setup ===
🔐 Generated PKCS12 password: wrFpsVuc... (32 chars)
[INFO] Using first node configuration: /etc/elasticsearch/identity-sau-main-dev/node-01
[2026-01-03 07:37:00 UTC] USER=www-data EUID=0 PID=2964460 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-sau-main-dev/node-01
[2026-01-03 07:37:00 UTC] USER=www-data EUID=0 PID=2964469 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-sau-main-dev/node-01
[INFO] Checking prerequisites...
[ OK ] ✓ Prerequisites verified
[INFO] Setting up temporary directories...
[2026-01-03 07:37:00 UTC] USER=www-data EUID=0 PID=2964480 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs
[2026-01-03 07:37:00 UTC] USER=www-data EUID=0 PID=2964489 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs
[2026-01-03 07:37:00 UTC] USER=www-data EUID=0 PID=2964498 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-sau-main-dev
[2026-01-03 07:37:00 UTC] USER=www-data EUID=0 PID=2964507 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-sau-main-dev/node-01
[2026-01-03 07:37:00 UTC] USER=www-data EUID=0 PID=2964516 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs
[2026-01-03 07:37:00 UTC] USER=www-data EUID=0 PID=2964525 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out
[ OK ] ✓ Directories created

[INFO] Building certificate instances configuration...
[INFO] Certificate instances configuration:
instances:
  - name: "identity-sau-main-dev-http"
    dns:  [ "localhost", "web-03", "search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com", "identity-sau-main-dev-node-01.fastorder.com", "search-identity-sau-main-dev-elasticsearch-coordinator.fastorder.com", "search-identity-sau-main-dev.fastorder.com", "identity-sau-main-dev-node-01.local" ]
    ip:   [ "10.100.1.186", "127.0.0.1", "::1" ]
[ OK ] ✓ Instances configuration created

[INFO] Generating HTTP Certificate Authority...
[2026-01-03 07:37:00 UTC] USER=www-data EUID=0 PID=2964543 ACTION=fsop ARGS=rm -f /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/http-ca.zip /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out/http-certs.zip
[2026-01-03 07:37:00 UTC] USER=www-data EUID=0 PID=2964552 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs
[2026-01-03 07:37:00 UTC] USER=www-data EUID=0 PID=2964561 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs
[2026-01-03 07:37:05 UTC] USER=www-data EUID=0 PID=2964616 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/http-ca.zip
Archive:  /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/http-ca.zip
  inflating: /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/ca/ca.crt  
  inflating: /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/ca/ca.key  
[ OK ] ✓ HTTP CA generated successfully

[INFO] Generating per-node HTTP certificates...
[2026-01-03 07:37:05 UTC] USER=www-data EUID=0 PID=2964628 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out
[2026-01-03 07:37:05 UTC] USER=www-data EUID=0 PID=2964637 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out
[2026-01-03 07:37:08 UTC] USER=www-data EUID=0 PID=2964697 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out/http-certs.zip
[2026-01-03 07:37:08 UTC] USER=www-data EUID=0 PID=2964706 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out/http
Archive:  /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out/http-certs.zip
  inflating: /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out/http/identity-sau-main-dev-http/identity-sau-main-dev-http.crt  
  inflating: /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out/http/identity-sau-main-dev-http/identity-sau-main-dev-http.key  
[ OK ] ✓ HTTP certificates generated successfully

[INFO] Installing certificates and configuring services...
[INFO] Configuring main service for single-node HTTPS...
[2026-01-03 07:37:08 UTC] USER=www-data EUID=0 PID=2964718 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-sau-main-dev/node-01/certs
[2026-01-03 07:37:08 UTC] USER=www-data EUID=0 PID=2964727 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/identity-sau-main-dev/node-01/certs
[2026-01-03 07:37:08 UTC] USER=www-data EUID=0 PID=2964736 ACTION=fsop ARGS=cp /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out/http/identity-sau-main-dev-http/identity-sau-main-dev-http.crt /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http.crt
[2026-01-03 07:37:08 UTC] USER=www-data EUID=0 PID=2964745 ACTION=fsop ARGS=cp /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/out/http/identity-sau-main-dev-http/identity-sau-main-dev-http.key /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http.key
[2026-01-03 07:37:08 UTC] USER=www-data EUID=0 PID=2964754 ACTION=fsop ARGS=cp /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/ca/ca.crt /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt
[2026-01-03 07:37:08 UTC] USER=www-data EUID=0 PID=2964763 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/identity-sau-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-01-03 07:37:08 UTC] USER=www-data EUID=0 PID=2964772 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/identity-sau-main-dev/node-01/certs
[2026-01-03 07:37:08 UTC] USER=www-data EUID=0 PID=2964781 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http.key
[2026-01-03 07:37:08 UTC] USER=www-data EUID=0 PID=2964790 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http.crt /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt /etc/elasticsearch/identity-sau-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-01-03 07:37:08 UTC] USER=www-data EUID=0 PID=2964799 ACTION=fsop ARGS=sed -i -E -e /^\s*xpack\.security\.http\.ssl(\..*)?\s*:/d /etc/elasticsearch/identity-sau-main-dev/node-01/elasticsearch.yml
[2026-01-03 07:37:08 UTC] USER=www-data EUID=0 PID=2964808 ACTION=fsop ARGS=sed -i /^# --- BEGIN direct HTTPS (managed, PEM) ---$/,/^# --- END direct HTTPS (managed, PEM) ---$/d /etc/elasticsearch/identity-sau-main-dev/node-01/elasticsearch.yml
[2026-01-03 07:37:08 UTC] USER=www-data EUID=0 PID=2964817 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/identity-sau-main-dev/node-01/certs/fastorder_ra_root.crt
YAML: /etc/elasticsearch/identity-sau-main-dev/node-01/elasticsearch.yml
[ OK ]   ✓ Main service configured with HTTPS
[2026-01-03 07:37:09 UTC] USER=www-data EUID=0 PID=2964836 ACTION=fsop ARGS=rm -rf /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients
[2026-01-03 07:37:09 UTC] USER=www-data EUID=0 PID=2964845 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients
[2026-01-03 07:37:09 UTC] USER=www-data EUID=0 PID=2964854 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients
Archive:  /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client.zip
   creating: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/
  inflating: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.crt  
  inflating: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.key  
[INFO] Creating P12 keystore for es-client...
[2026-01-03 07:37:12 UTC] USER=www-data EUID=0 PID=2964900 ACTION=fsop ARGS=mv /tmp/es-client-2964431.p12 /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.p12
[ OK ] ✓ Created P12 keystore: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-01-03 07:37:12 UTC] USER=www-data EUID=0 PID=2964909 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients
[2026-01-03 07:37:12 UTC] USER=www-data EUID=0 PID=2964918 ACTION=fsop ARGS=chmod 640 /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.key
[2026-01-03 07:37:12 UTC] USER=www-data EUID=0 PID=2964927 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-01-03 07:37:12 UTC] USER=www-data EUID=0 PID=2964936 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.crt /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt
[INFO] Saving keystore passwords to secrets vault...
🔑 Configuring AWS credentials for secrets vault...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[INFO] 🔐 Vaulting search passwords to remote backend...
✅ Passwords vaulted to remote backend
✓ Keystore passwords saved to secrets vault: search/identity-sau-main-dev/keystore-passwords

[INFO] === Installing CA Certificate for Users ===
[INFO] HOME not set, skipping user CA installation

✓ Direct HTTPS configuration completed for environment: identity-sau-main-dev
[INFO] All services now serve HTTPS using PEM certificates
[INFO] Network binding: 10.100.1.186
[INFO] HTTPS endpoint: https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === Certificate Summary ===
CA Certificate: /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs/ca/ca.crt
Certificate Directory: /etc/elasticsearch/identity-sau-main-dev/node-01/http-certs
Main service certificates: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/

[INFO] === Next Steps ===
1. Restart Elasticsearch services to apply HTTPS configuration
2. Test HTTPS connectivity: curl https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
3. Verify certificates: openssl s_client -connect search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200

[INFO] === SSL Certificate Information ===
• CA certificate installed for user - curl will work without -k flag
• Each certificate includes node-specific domain, VM IP, and localhost in Subject Alternative Names
• Certificates are valid for 3 years from generation date

[WARNING] Important: You'll need to restart Elasticsearch services for HTTPS to take effect
[ OK ] ✓ Direct HTTPS configuration completed successfully

[ OK ] ==================================================================
[ OK ] HTTP SSL Configuration Complete
[ OK ] ==================================================================
[INFO] Environment: identity-sau-main-dev
[INFO] Nodes: 1
[INFO] Configuration applied to port: 9200 (default port for all nodes)

[INFO] === Next Steps ===
1. Verify Elasticsearch is running: systemctl status elasticsearch@identity-sau-main-dev-node-01.service
2. Test cluster health: curl https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health
3. Check SSL certificate: openssl s_client -connect search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
=== HTTPS Setup completed successfully! ===
Environment:  (1 nodes)
Domain: .fastorder.com
HTTPS endpoint: https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
Node IP: 10.100.1.186

✓ Step 2 completed successfully!

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 3: Executing Create Index Llm
Folder: 03-create-index-llm
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

==================================================================
Elasticsearch LLM/Semantic Search Setup
==================================================================
[INFO] Using web-provided environment: identity-sau-main-dev
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
Environment: identity-sau-main-dev
Service    : identity
🔍 Checking Elasticsearch availability…
✅ Elasticsearch is accessible at https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200

=== Phase 1: Common steps under /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps ===
   (no numbered steps in: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps)
=== Phase 2: Service-scoped steps for 'identity' under /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/identity ===
📚 Detected features: login

── Feature: login
▶️  Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/identity/login/01-create-model-and-pipeline.sh
==================================================================
STEP 1: Create Model and Ingest Pipeline
==================================================================
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] ES URL: https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Endpoint ID (ES): identity-text-embedding-001
[INFO] Provider model: text-embedding-3-large
[INFO] Pipeline ID: identity-embed-pipeline-001
[INFO] Checking authentication identity…
{
"username":"elastic","roles":["superuser"],"full_name":null,"email":null,"metadata":{"_reserved":true},"enabled":true,"authentication_realm":{"name":"reserved","type":"reserved"},"lookup_realm":{"name":"reserved","type":"reserved"},"authentication_type":"realm"
}
[INFO] Checking Elasticsearch license…
[INFO] License type: unknown
[WARN] Inference API requires Enterprise/Platinum license (found: unknown)
[WARN] Skipping inference endpoint and pipeline creation
[OK]   Setup completed (inference features skipped due to license)
✅ 01-create-model-and-pipeline.sh completed

▶️  Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/identity/login/02-create-index.sh
==================================================================
STEP 2: Create Semantic Search Index (initial bootstrap)
==================================================================
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] [create] Ensuring clean slate for: identity_sau_main_dev_account_router-000001
[INFO] Index identity_sau_main_dev_account_router-000001 does not exist (status 404), proceeding.
[INFO] [create] Creating index identity_sau_main_dev_account_router-000001 with write alias identity_sau_main_dev_account_router
[OK]   Index + alias ready.
   Index (concrete): identity_sau_main_dev_account_router-000001
   Alias (stable)  : identity_sau_main_dev_account_router  (is_write_index=true)
   Default pipeline: identity-embed-pipeline-001
   Vector dims     : 3072 (KNN cosine)
✅ 02-create-index.sh completed

▶️  Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/identity/login/03-llm.sh
==================================================================
STEP 2: Create Semantic Search Index (ILM bootstrap)
==================================================================
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] [cluster] Ensure disk watermarks permit allocation
[OK]   Cluster watermarks set/confirmed.
[INFO] [create] Create/Update ILM policy: identity-account-router-ilm
[OK]   ILM policy ready.
[INFO] [create] Create/Update index template: identity_sau_main_dev_account_router_template
[OK]   Index template ready.
[INFO] [check] Concrete index: identity_sau_main_dev_account_router-000001
[OK]   Concrete index identity_sau_main_dev_account_router-000001 already exists (skip create).
[INFO] [verify] Wait for index to be at least YELLOW
[OK]   Cluster health OK for identity_sau_main_dev_account_router-000001.
[INFO] [verify] Alias points to a concrete write index
[OK]   Alias verification passed.
[INFO] [explain] ILM status
{
  "indices" : {
    "identity_sau_main_dev_account_router-000001" : {
      "index" : "identity_sau_main_dev_account_router-000001",
      "managed" : false
    }
  }
}

[OK]   ILM/alias bootstrap complete.
   Index (concrete): identity_sau_main_dev_account_router-000001
   Alias (stable)  : identity_sau_main_dev_account_router  (is_write_index=true)
   ILM policy      : identity-account-router-ilm
   Default pipeline: identity-embed-pipeline-001
✅ 03-llm.sh completed

▶️  Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/identity/login/04-index-sample-data.sh
==================================================================
STEP 3: Index Sample Data
==================================================================
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[WARN] Pipeline 'identity-embed-pipeline-001' not found (HTTP 404); proceeding without it.
[INFO] [bulk] Index seed documents → identity_sau_main_dev_account_router
[WARN] Bulk completed with item-level errors. Showing first 50 lines:
{"errors":true,"took":0,"ingest_took":0,"items":[{"index":{"_index":"identity_sau_main_dev_account_router","_id":"auto-generated","status":400,"error":{"type":"illegal_argument_exception","reason":"pipeline with id [identity-embed-pipeline-001] does not exist"}}},{"index":{"_index":"identity_sau_main_dev_account_router","_id":"auto-generated","status":400,"error":{"type":"illegal_argument_exception","reason":"pipeline with id [identity-embed-pipeline-001] does not exist"}}},{"index":{"_index":"identity_sau_main_dev_account_router","_id":"auto-generated","status":400,"error":{"type":"illegal_argument_exception","reason":"pipeline with id [identity-embed-pipeline-001] does not exist"}}}]}[summary] items=3 errors=3
[INFO] [verify] Search a sample term: 'password'
  {
    "took" : 148,
    "timed_out" : false,
    "_shards" : {
      "total" : 1,
      "successful" : 1,
      "skipped" : 0,
      "failed" : 0
    },
    "hits" : {
      "total" : {
        "value" : 0,
        "relation" : "eq"
      },
      "max_score" : null,
      "hits" : [ ]
    }
  }
[OK]   Sample data indexing step completed.
✅ 04-index-sample-data.sh completed

▶️  Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/identity/login/05-create-cdc-index.sh
==================================================================
STEP 5: Create CDC Account Router Index (for dashboard visibility)
==================================================================
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Creating CDC index: identity_sau_main_dev_account_router
[OK]   Index identity_sau_main_dev_account_router already exists
✅ 05-create-cdc-index.sh completed

=== Phase 3: Optional search smoke tests ===
   (semantic search test script not found: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/search-semantic.sh)
   (hybrid search test script not found: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/hybrid-search.sh)
==================================================================
🎉 LLM/Semantic Search setup completed successfully!
==================================================================

Available commands:
  • Test semantic search:
    bash steps/search-semantic.sh en "password policy"
    bash steps/search-semantic.sh ar "كلمة المرور"

  • Test hybrid search:
    bash steps/hybrid-search.sh en "user authentication"
    bash steps/hybrid-search.sh ar "مصادقة المستخدم"

Alias   : identity_sau_main_dev_account_router
Index   : identity_sau_main_dev_account_router-000001
ILM     : identity-account-router-ilm
Model   : identity-text-embedding-001
Pipeline: identity-embed-pipeline-001
==================================================================

✓ Step 3 completed successfully!

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 4: Executing Monitoring Setup
Folder: 10-monitoring-setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using web-provided environment: identity-sau-main-dev
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Elasticsearch Monitoring Integration for identity-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[OK]   ✓ Observability cell is ready

[INFO] 2️⃣ Discovering Elasticsearch configuration...
[OK]   ✓ Found Elasticsearch at 10.100.1.186:9200

[INFO] 3️⃣ Setting up elasticsearch_exporter integration...
[INFO] Using elasticsearch_exporter port: 9114
[INFO] SSL certificates configured for elasticsearch_exporter:
[INFO]   CA cert: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt
[INFO]   Client cert: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.crt
[INFO]   Client key: /etc/elasticsearch/identity-sau-main-dev/node-01/certs/clients/es-client/es-client.key
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[INFO] Setting up elasticsearch_exporter for identity-sau-main-dev
[INFO] Elasticsearch exporter will bind to: 10.100.1.186:9114
[2026-01-03 07:37:39 UTC] USER=www-data EUID=0 PID=2965702 ACTION=passthru ARGS=mv /tmp/elasticsearch_exporter-identity-sau-main-dev.service /etc/systemd/system/elasticsearch_exporter-identity-sau-main-dev.service
[2026-01-03 07:37:39 UTC] USER=www-data EUID=0 PID=2965711 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-03 07:37:40 UTC] USER=www-data EUID=0 PID=2965756 ACTION=passthru ARGS=systemctl enable elasticsearch_exporter-identity-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/elasticsearch_exporter-identity-sau-main-dev.service -> /etc/systemd/system/elasticsearch_exporter-identity-sau-main-dev.service.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  IP Conflict Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: identity-sau-main-dev
IP Address:  10.100.1.186
Port:        9114
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔍 Checking IP conflict for identity-sau-main-dev on 10.100.1.186:9114...
✅ IP 10.100.1.186:9114 is available - no conflicts detected

🔍 Checking for orphaned processes that might conflict...
✅ No orphaned processes detected

✅ All checks passed - safe to proceed with identity-sau-main-dev setup
[2026-01-03 07:37:41 UTC] USER=www-data EUID=0 PID=2965841 ACTION=passthru ARGS=systemctl restart elasticsearch_exporter-identity-sau-main-dev.service
[OK]   elasticsearch_exporter configured on 10.100.1.186:9114
[INFO] Register this endpoint in metrics-identity-sau-main-dev.fastorder.com scrape config
[OK]   ✓ elasticsearch_exporter integration complete

[INFO] 3.5️⃣ Configuring Prometheus to scrape Elasticsearch metrics...
[2026-01-03 07:37:44 UTC] USER=www-data EUID=0 PID=2965900 ACTION=passthru ARGS=grep -q job_name: 'elasticsearch' /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[INFO] Adding Elasticsearch scrape target to Prometheus configuration...
[2026-01-03 07:37:44 UTC] USER=www-data EUID=0 PID=2965923 ACTION=fsop ARGS=cp /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml.backup-1767425864
[INFO] Created backup: /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml.backup-1767425864
[2026-01-03 07:37:44 UTC] USER=www-data EUID=0 PID=2965944 ACTION=passthru ARGS=sed -i /# Prometheus self-monitoring/r /tmp/prometheus_es_add.yml /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[2026-01-03 07:37:44 UTC] USER=www-data EUID=0 PID=2965965 ACTION=passthru ARGS=grep -q job_name: 'elasticsearch' /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[INFO] ✓ Elasticsearch job successfully inserted into config
[INFO] Validating Prometheus configuration with promtool...
Checking /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml is valid prometheus config file syntax

Checking /etc/prometheus/obs-identity-sau-main-dev/rules/basic_alerts.yml
  SUCCESS: 4 rules found

[OK]   ✓ Prometheus configuration validation PASSED
[OK]   ✓ Prometheus configuration updated successfully
[2026-01-03 07:37:44 UTC] USER=www-data EUID=0 PID=2965994 ACTION=passthru ARGS=systemctl is-active --quiet prometheus-obs-identity-sau-main-dev.service
[INFO] Reloading Prometheus configuration...
[2026-01-03 07:37:45 UTC] USER=www-data EUID=0 PID=2966015 ACTION=passthru ARGS=systemctl restart prometheus-obs-identity-sau-main-dev.service
[2026-01-03 07:37:48 UTC] USER=www-data EUID=0 PID=2966059 ACTION=passthru ARGS=systemctl is-active --quiet prometheus-obs-identity-sau-main-dev.service
[OK]   ✓ Prometheus reloaded successfully
[2026-01-03 07:37:48 UTC] USER=www-data EUID=0 PID=2966080 ACTION=fsop ARGS=rm -f /tmp/prometheus_es_add.yml

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Elasticsearch Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Service: elasticsearch_exporter-identity-sau-main-dev.service
[INFO] Metrics: http://localhost:9114/metrics
[INFO] Prometheus: https://metrics-identity-sau-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-identity-sau-main-dev.fastorder.com
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 4️⃣ Registering Elasticsearch nodes to monitoring database...
[INFO]    Constructed FQDN: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com
[INFO] Registering: identity-sau-main-dev-node-01
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Elasticsearch
[INFO]   Identifier:        identity-sau-main-dev-node-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                10.100.1.186
[INFO]   Port:              9200
[INFO]   FQDN:              search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 0c4bbf26-5554-4b0a-91b2-b9563a7e8fbf
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ Registered: identity-sau-main-dev-node-01
[OK]   ✓ Elasticsearch node registration completed successfully

[INFO] 5️⃣ Verifying monitoring integration...

[INFO] Checking elasticsearch_exporter service...
[OK]   ✓ elasticsearch_exporter-identity-sau-main-dev.service is ACTIVE
[INFO] Checking Prometheus service...
[OK]   ✓ prometheus-obs-identity-sau-main-dev.service is ACTIVE
[INFO] Validating Prometheus configuration...
[OK]   ✓ Prometheus configuration is VALID
[INFO] Checking Prometheus targets (waiting 35s for first scrape cycle)...
[2026-01-03 07:38:24 UTC] USER=www-data EUID=0 PID=2966504 ACTION=passthru ARGS=grep -q tls_server_config /etc/prometheus/obs-identity-sau-main-dev/web-config.yml
[OK]   ✓ Prometheus has Elasticsearch target configured
[OK]   ✓ Elasticsearch target is UP and being scraped

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ All monitoring integration steps completed
[INFO] ✅ All verifications PASSED
[INFO] ✅ Elasticsearch registered to dashboard database
[INFO] ✅ Prometheus scraping Elasticsearch metrics
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


✓ Step 4 completed successfully!

════════════════════════════════════════════════════════════════
🎉 All deployment tasks completed successfully!

✓ ✅ Search infrastructure (elasticsearch) setup completed successfully

[INFO] Using eventbus engine from EVENTBUS_ENGINE environment variable: kafka
[INFO] Cleaning up any existing locks...

Starting eventbus engine: kafka
═══════════════════════════════════════════════

[INFO] Loaded from topology.json: identity-sau-main-dev
[2026-01-03 07:38:25] Loaded environment: identity-sau-main-dev
[2026-01-03 07:38:25] Service: identity, Zone: sau, Branch: main, Env: dev
[2026-01-03 07:38:25] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-03 07:38:25] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-03 07:38:25] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
[INFO] Starting Kafka setup process...
[INFO] Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps
[INFO] Environment: identity-sau-main-dev

[INFO] Found 9 step(s) to execute

[INFO] 📦 Step 1/9: install debezium connector...
═══════════════════════════════════════════════════════════════════
Fetching latest versions from Maven Central...
Installing Debezium PostgreSQL Connector
  Debezium version: 3.4.0.Final
  pgjdbc version:   42.7.8
═══════════════════════════════════════════════════════════════════
[OK] Debezium 3.4.0.Final with pgjdbc 42.7.8 already installed
[OK] ✅ Step 1 completed: 00-install-debezium-connector.sh

[INFO] 📦 Step 2/9: kafka setup...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials for secrets vault...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
🧹 Checking for orphaned Kafka processes on ports 9092, 9093, 8083...
  ⚠️  Found process on port 9092 (PIDs: [2026-01-03 07:38:26 UTC] USER=www-data EUID=0 PID=2966620 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 2>/dev/null || true
1602481
2067199), killing...
[2026-01-03 07:38:26 UTC] USER=www-data EUID=0 PID=2966630 ACTION=passthru ARGS=bash -c kill -9 [2026-01-03 07:38:26 UTC] USER=www-data EUID=0 PID=2966620 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 2>/dev/null || true
1602481
2067199 2>/dev/null || true
/usr/bin/bash: line 2: 1602481: command not found
  ⚠️  Found process on port 9093 (PIDs: [2026-01-03 07:38:27 UTC] USER=www-data EUID=0 PID=2966642 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 2>/dev/null || true
1595983
1602481
2954501), killing...
[2026-01-03 07:38:28 UTC] USER=www-data EUID=0 PID=2966652 ACTION=passthru ARGS=bash -c kill -9 [2026-01-03 07:38:27 UTC] USER=www-data EUID=0 PID=2966642 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 2>/dev/null || true
1595983
1602481
2954501 2>/dev/null || true
/usr/bin/bash: line 2: 1595983: command not found
/usr/bin/bash: line 3: 1602481: command not found
  ⚠️  Found process on port 8083 (PIDs: [2026-01-03 07:38:29 UTC] USER=www-data EUID=0 PID=2966665 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 2>/dev/null || true
2067199), killing...
[2026-01-03 07:38:29 UTC] USER=www-data EUID=0 PID=2966675 ACTION=passthru ARGS=bash -c kill -9 [2026-01-03 07:38:29 UTC] USER=www-data EUID=0 PID=2966665 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 2>/dev/null || true
2067199 2>/dev/null || true
✅ Port cleanup completed
Ensuring KAFKA application environment for coordinator...
[INFO] Creating KAFKA application environment...
[INFO] 🎯 Custom Environment Creation (Example Wrapper)
[INFO] 📁 Orchestrator Library: /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] 💾 State Directory: /opt/fastorder/bash/scripts/env_app_setup/state

[INFO] 🚀 Calling centralized orchestrator: fo-env create-app
[INFO] 📋 Arguments: --service identity --zone sau --branch main --env dev --domain eventbus-identity-sau-main-dev-kafka-connect --app kafka-connect

[INFO] Creating application-specific environment configuration
[INFO] Environment ID: identity-sau-main-dev
[INFO] Application: kafka-connect
[INFO] Base environment identity-sau-main-dev already exists
[INFO] Allocated kafka-connect IP: 10.100.1.212
[INFO] Generated domain: eventbus-identity-sau-main-dev-kafka-connect.fastorder.com
[INFO] Configuring network interface for kafka-connect IP: 10.100.1.212
[INFO] IP 10.100.1.212 is already configured
[INFO] Updating topology with application-specific configuration...
[ OK ] Topology updated with application-specific configuration
[INFO] Binding kafka-connect IP to domain: 10.100.1.212 -> eventbus-identity-sau-main-dev-kafka-connect.fastorder.com
[ OK ] Successfully bound eventbus-identity-sau-main-dev-kafka-connect.fastorder.com to 10.100.1.212
[ OK ] Domain correctly mapped
[ OK ] Application environment created successfully!
[INFO] 
[INFO] Application Details:
[INFO]   Environment ID: identity-sau-main-dev
[INFO]   Application: kafka-connect
[INFO]   IP: 10.100.1.212
[INFO]   Domain: eventbus-identity-sau-main-dev-kafka-connect.fastorder.com
[INFO] 
[INFO] To use this application:
[INFO]   source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO]   init_environment kafka-connect
[INFO]   echo $VM_IP  # Returns: 10.100.1.212

[ OK ] 🎉 Environment creation completed successfully!

[INFO] 📋 What happened:
[INFO]   ✅ Called centralized orchestrator at /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO]   ✅ All topology.json management handled centrally
[INFO]   ✅ Application-specific IP and domain configured
[INFO]   ✅ Network interface configured and made persistent
[INFO]   ✅ Domain binding added to /etc/hosts (if not skipped)

[INFO] 🔧 To use the centralized orchestrator directly:
[INFO]   # Add orchestrator to PATH
[INFO]   export PATH="/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/bin:$PATH"
[INFO]   # Then call directly
[INFO]   fo-env create-app --service auth --zone uae --env dev --app redis

[INFO] 📚 For more orchestrator commands:
[INFO]   fo-env --help
Created KAFKA environment: eventbus-identity-sau-main-dev-kafka-connect.fastorder.com (10.100.1.212)
Ensuring KAFKA_BROKER_IP application environment for coordinator...
[INFO] Creating KAFKA application environment...
[INFO] 🎯 Custom Environment Creation (Example Wrapper)
[INFO] 📁 Orchestrator Library: /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] 💾 State Directory: /opt/fastorder/bash/scripts/env_app_setup/state

[INFO] 🚀 Calling centralized orchestrator: fo-env create-app
[INFO] 📋 Arguments: --service identity --zone sau --branch main --env dev --domain eventbus-identity-sau-main-dev-kafka-broker-01 --app kafka-broker

[INFO] Creating application-specific environment configuration
[INFO] Environment ID: identity-sau-main-dev
[INFO] Application: kafka-broker
[INFO] Base environment identity-sau-main-dev already exists
[INFO] Allocated kafka-broker IP: 10.100.1.213
[INFO] Generated domain: eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com
[INFO] Configuring network interface for kafka-broker IP: 10.100.1.213
[INFO] IP 10.100.1.213 is already configured
[INFO] Updating topology with application-specific configuration...
[ OK ] Topology updated with application-specific configuration
[INFO] Binding kafka-broker IP to domain: 10.100.1.213 -> eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com
[ OK ] Successfully bound eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com to 10.100.1.213
[ OK ] Domain correctly mapped
[ OK ] Application environment created successfully!
[INFO] 
[INFO] Application Details:
[INFO]   Environment ID: identity-sau-main-dev
[INFO]   Application: kafka-broker
[INFO]   IP: 10.100.1.213
[INFO]   Domain: eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com
[INFO] 
[INFO] To use this application:
[INFO]   source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO]   init_environment kafka-broker
[INFO]   echo $VM_IP  # Returns: 10.100.1.213

[ OK ] 🎉 Environment creation completed successfully!

[INFO] 📋 What happened:
[INFO]   ✅ Called centralized orchestrator at /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO]   ✅ All topology.json management handled centrally
[INFO]   ✅ Application-specific IP and domain configured
[INFO]   ✅ Network interface configured and made persistent
[INFO]   ✅ Domain binding added to /etc/hosts (if not skipped)

[INFO] 🔧 To use the centralized orchestrator directly:
[INFO]   # Add orchestrator to PATH
[INFO]   export PATH="/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/bin:$PATH"
[INFO]   # Then call directly
[INFO]   fo-env create-app --service auth --zone uae --env dev --app redis

[INFO] 📚 For more orchestrator commands:
[INFO]   fo-env --help
Created KAFKA_BROKER_DOMAIN environment: eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com (10.100.1.213)
[INFO] Kafka Broker IP: 10.100.1.213
[INFO] Kafka Connect IP: 10.100.1.212
[INFO] Registered /etc/hosts: eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com -> 10.100.1.213
[INFO] Registered /etc/hosts: eventbus-identity-sau-main-dev-kafka-connect.fastorder.com -> 10.100.1.212
🔐 Initializing keystore passwords...
[INFO] 🔍 Checking secrets backend (provider: aws)...
✅ Retrieved passwords from remote backend
[INFO] ✅ Using existing passwords from backend
✅ Keystore passwords initialized
   - Keystore password: E4FDSwWT... (32 chars)
   - Truststore password: yOb0eqkA... (32 chars)
[INFO] 🔐 Vaulting kafka passwords to remote backend...
✅ Passwords vaulted to remote backend
✅ Kafka keystore passwords saved to AWS Secrets Manager
[INFO] Generating for: identity-sau-main-dev (host=eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com ip=10.100.1.213)
[2026-01-03 07:38:40 UTC] USER=www-data EUID=0 PID=2967574 ACTION=fsop ARGS=rm -rf /opt/kafka/secrets/identity-sau-main-dev/coordinator /var/lib/kafka/identity-sau-main-dev/coordinator
[2026-01-03 07:38:40 UTC] USER=www-data EUID=0 PID=2967583 ACTION=fsop ARGS=mkdir -p /opt/kafka/secrets/identity-sau-main-dev/coordinator /opt/kafka/config/identity-sau-main-dev/coordinator /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem /var/lib/kafka/identity-sau-main-dev_coordinator-data
[2026-01-03 07:38:40 UTC] USER=www-data EUID=0 PID=2967592 ACTION=fsop ARGS=chown -R kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator
[2026-01-03 07:38:40 UTC] USER=www-data EUID=0 PID=2967601 ACTION=fsop ARGS=chown -R kafka:kafka /opt/kafka/config/identity-sau-main-dev/coordinator /var/lib/kafka/identity-sau-main-dev_coordinator-data
[2026-01-03 07:38:40 UTC] USER=www-data EUID=0 PID=2967610 ACTION=fsop ARGS=chmod 770 /opt/kafka/config/identity-sau-main-dev/coordinator /var/lib/kafka/identity-sau-main-dev_coordinator-data
[2026-01-03 07:38:40 UTC] USER=www-data EUID=0 PID=2967619 ACTION=fsop ARGS=chmod 750 /opt/kafka/secrets/identity-sau-main-dev/coordinator
[2026-01-03 07:38:40 UTC] USER=www-data EUID=0 PID=2967628 ACTION=fsop ARGS=chmod 750 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
[2026-01-03 07:38:40 UTC] USER=www-data EUID=0 PID=2967638 ACTION=fsop ARGS=chmod 700 /tmp/fo-tls.6l2aP6
[2026-01-03 07:38:40 UTC] USER=www-data EUID=0 PID=2967647 ACTION=fsop ARGS=chmod 755 /tmp/fo-tls.6l2aP6
[2026-01-03 07:38:40 UTC] USER=www-data EUID=0 PID=2967656 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/fo-tls.6l2aP6/ra_root.crt
[2026-01-03 07:38:40 UTC] USER=www-data EUID=0 PID=2967665 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/fo-tls.6l2aP6/ra_root.key
[2026-01-03 07:38:40 UTC] USER=www-data EUID=0 PID=2967674 ACTION=fsop ARGS=chmod 644 /tmp/fo-tls.6l2aP6/ra_root.crt
[2026-01-03 07:38:40 UTC] USER=www-data EUID=0 PID=2967683 ACTION=fsop ARGS=chmod 644 /tmp/fo-tls.6l2aP6/ra_root.key
Certificate was added to keystore
[2026-01-03 07:38:41 UTC] USER=www-data EUID=0 PID=2967717 ACTION=fsop ARGS=mv /tmp/fo-tls.6l2aP6/truststore.jks /opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks
[2026-01-03 07:38:41 UTC] USER=www-data EUID=0 PID=2967726 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks
[2026-01-03 07:38:41 UTC] USER=www-data EUID=0 PID=2967735 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks
Generating 4,096 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 825 days
	for: CN=eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com, OU=Kafka Broker, O=FastOrder, C=AE

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /tmp/fo-tls.6l2aP6/kafka.server.keystore.jks -destkeystore /tmp/fo-tls.6l2aP6/kafka.server.keystore.jks -deststoretype pkcs12".

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /tmp/fo-tls.6l2aP6/kafka.server.keystore.jks -destkeystore /tmp/fo-tls.6l2aP6/kafka.server.keystore.jks -deststoretype pkcs12".
Certificate request self-signature ok
subject=C = AE, O = FastOrder, OU = Kafka Broker, CN = eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com
Certificate was added to keystore

Warning:
Certificate reply was installed in keystore

Warning:
[2026-01-03 07:38:44 UTC] USER=www-data EUID=0 PID=2967832 ACTION=fsop ARGS=mv /tmp/fo-tls.6l2aP6/kafka.server.keystore.jks /opt/kafka/secrets/identity-sau-main-dev/coordinator/kafka.server.keystore.jks
[2026-01-03 07:38:44 UTC] USER=www-data EUID=0 PID=2967841 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/identity-sau-main-dev/coordinator/kafka.server.keystore.jks
[2026-01-03 07:38:44 UTC] USER=www-data EUID=0 PID=2967850 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/kafka.server.keystore.jks
Generating 4,096 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 825 days
	for: CN=eventbus-identity-sau-main-dev-kafka-connect.fastorder.com, OU=Kafka Connect REST, O=FastOrder, C=AE
Certificate request self-signature ok
subject=C = AE, O = FastOrder, OU = Kafka Connect REST, CN = eventbus-identity-sau-main-dev-kafka-connect.fastorder.com
Certificate was added to keystore
Certificate reply was installed in keystore
[2026-01-03 07:38:49 UTC] USER=www-data EUID=0 PID=2967953 ACTION=fsop ARGS=mv /tmp/fo-tls.6l2aP6/connect-rest.keystore.p12 /opt/kafka/secrets/identity-sau-main-dev/coordinator/connect-rest.keystore.p12
[2026-01-03 07:38:49 UTC] USER=www-data EUID=0 PID=2967962 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/identity-sau-main-dev/coordinator/connect-rest.keystore.p12
[2026-01-03 07:38:49 UTC] USER=www-data EUID=0 PID=2967971 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/connect-rest.keystore.p12
Certificate request self-signature ok
subject=CN = kafka-client-identity-sau-main-dev, OU = Kafka Client, O = FastOrder, C = AE
[2026-01-03 07:38:49 UTC] USER=www-data EUID=0 PID=2967983 ACTION=fsop ARGS=cp /tmp/fo-tls.6l2aP6/ra_root.crt /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem
[2026-01-03 07:38:49 UTC] USER=www-data EUID=0 PID=2967992 ACTION=fsop ARGS=cp /tmp/fo-tls.6l2aP6/client-key.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:38:49 UTC] USER=www-data EUID=0 PID=2968001 ACTION=fsop ARGS=cp /tmp/fo-tls.6l2aP6/client-cert.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
[2026-01-03 07:38:49 UTC] USER=www-data EUID=0 PID=2968010 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
[2026-01-03 07:38:49 UTC] USER=www-data EUID=0 PID=2968019 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
[2026-01-03 07:38:49 UTC] USER=www-data EUID=0 PID=2968028 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:38:50 UTC] USER=www-data EUID=0 PID=2968038 ACTION=fsop ARGS=mv /tmp/fo-tls.6l2aP6/kafka.client.keystore.p12 /opt/kafka/secrets/identity-sau-main-dev/coordinator/kafka.client.keystore.p12
[2026-01-03 07:38:50 UTC] USER=www-data EUID=0 PID=2968047 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/identity-sau-main-dev/coordinator/kafka.client.keystore.p12
[2026-01-03 07:38:50 UTC] USER=www-data EUID=0 PID=2968056 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/kafka.client.keystore.p12
🔐 Ensuring kafka user has access to PostgreSQL certificates...
✅ kafka is already in postgres group
🧹 Cleaning up conflicting services and processes on Kafka ports on 10.100.1.213...
🔪 Killing processes on 10.100.1.213:8083: [2026-01-03 07:38:50 UTC] USER=www-data EUID=0 PID=2968094 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.213:8083 -t 2>/dev/null || true
[2026-01-03 07:38:50 UTC] USER=www-data EUID=0 PID=2968107 ACTION=passthru ARGS=bash -c kill -9 [2026-01-03 07:38:50 UTC] USER=www-data EUID=0 PID=2968094 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.213:8083 -t 2>/dev/null || true
🔪 Killing processes on 10.100.1.213:9092: [2026-01-03 07:38:50 UTC] USER=www-data EUID=0 PID=2968118 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.213:9092 -t 2>/dev/null || true
[2026-01-03 07:38:50 UTC] USER=www-data EUID=0 PID=2968131 ACTION=passthru ARGS=bash -c kill -9 [2026-01-03 07:38:50 UTC] USER=www-data EUID=0 PID=2968118 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.213:9092 -t 2>/dev/null || true
🔪 Killing processes on 10.100.1.213:9093: [2026-01-03 07:38:50 UTC] USER=www-data EUID=0 PID=2968142 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.213:9093 -t 2>/dev/null || true
[2026-01-03 07:38:51 UTC] USER=www-data EUID=0 PID=2968162 ACTION=passthru ARGS=bash -c kill -9 [2026-01-03 07:38:50 UTC] USER=www-data EUID=0 PID=2968142 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.213:9093 -t 2>/dev/null || true
✅ Port cleanup completed
🔧 Checking for Kafka Connect internal topics with incorrect cleanup policy...
📋 Kafka broker is running, checking topic cleanup policies...
✅ Topic cleanup policy fix completed
🔧 Creating environment-specific systemd units...
🔧 Writing client properties to /etc/kafka/client-identity-sau-main-dev-coordinator.properties ...
[2026-01-03 07:38:51 UTC] USER=www-data EUID=0 PID=2968219 ACTION=fsop ARGS=chown root:kafka /etc/kafka/client-identity-sau-main-dev-coordinator.properties
[2026-01-03 07:38:51 UTC] USER=www-data EUID=0 PID=2968228 ACTION=fsop ARGS=chmod 0640 /etc/kafka/client-identity-sau-main-dev-coordinator.properties
[2026-01-03 07:38:51 UTC] USER=www-data EUID=0 PID=2968237 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-03 07:38:58 UTC] USER=www-data EUID=0 PID=2968296 ACTION=passthru ARGS=systemctl mask kafka-server
Failed to print table: Broken pipe
🔒 Adjusting group ownership and permissions ...
[2026-01-03 07:39:02 UTC] USER=www-data EUID=0 PID=2968582 ACTION=fsop ARGS=chown :kafka /opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks /opt/kafka/secrets/identity-sau-main-dev/coordinator/kafka.server.keystore.jks
[2026-01-03 07:39:02 UTC] USER=www-data EUID=0 PID=2968604 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks /opt/kafka/secrets/identity-sau-main-dev/coordinator/kafka.server.keystore.jks
[2026-01-03 07:39:03 UTC] USER=www-data EUID=0 PID=2968615 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/kafka.client.keystore.p12
[2026-01-03 07:39:03 UTC] USER=www-data EUID=0 PID=2968624 ACTION=fsop ARGS=chown root:kafka /etc/kafka/client-identity-sau-main-dev-coordinator.properties
[2026-01-03 07:39:03 UTC] USER=www-data EUID=0 PID=2968633 ACTION=fsop ARGS=chmod 0640 /etc/kafka/client-identity-sau-main-dev-coordinator.properties

✅ Kafka configuration complete for identity-sau-main-dev_coordinator
  Broker ID         : 5
  Broker keystore   : /opt/kafka/secrets/identity-sau-main-dev/coordinator/kafka.server.keystore.jks
  REST keystore     : /opt/kafka/secrets/identity-sau-main-dev/coordinator/connect-rest.keystore.p12
  Truststore        : /opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks
  Client PKCS12     : /opt/kafka/secrets/identity-sau-main-dev/coordinator/kafka.client.keystore.p12
  Data directory    : /var/lib/kafka/identity-sau-main-dev_coordinator-data
  Server config     : /opt/kafka/config/identity-sau-main-dev/coordinator/server.properties
  Connect config    : /opt/kafka/config/identity-sau-main-dev/coordinator/connect-distributed.properties
  CLI client config : /etc/kafka/client-identity-sau-main-dev-coordinator.properties

🎯 Next step: Run 03-restart-kafka-related-services.sh to start services

[OK] ✅ Step 2 completed: 01-kafka-setup.sh

[INFO] 📦 Step 3/9: metadata...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🎯 Kafka metadata mode: kraft

╔════════════════════════════════════════════════════════════════════╗
║            Kafka Metadata Layer Setup                             ║
╚════════════════════════════════════════════════════════════════════╝

  Environment    : identity-sau-main-dev
  Service        : identity
  Zone           : sau
  Branch         : main
  Environment    : dev
  VM IP          : 142.93.238.16
  Metadata Mode  : kraft

  📋 KRaft Mode (Modern)
  ────────────────────────────────────────────────────────────────
  ✅ No ZooKeeper dependency
  ✅ Faster metadata operations
  ✅ Simplified architecture
  ✅ Recommended for new deployments
  ⚠️  Requires Kafka 3.3+ in production


════════════════════════════════════════════════════════════════════

[INFO] 🚀 Executing KRaft setup script...
[INFO] Script: /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps/metadata/kraft.sh

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-03 07:39:04 UTC] USER=www-data EUID=0 PID=2968813 ACTION=fsop ARGS=mkdir -p /var/lib/kafka/identity-sau-main-dev_coordinator-meta /opt/kafka/config/identity-sau-main-dev/coordinator /var/lib/kafka/identity-sau-main-dev_coordinator-data
[2026-01-03 07:39:04 UTC] USER=www-data EUID=0 PID=2968822 ACTION=fsop ARGS=chown -R kafka:kafka /var/lib/kafka/identity-sau-main-dev_coordinator-meta /opt/kafka/config/identity-sau-main-dev/coordinator /var/lib/kafka/identity-sau-main-dev_coordinator-data
[2026-01-03 07:39:04 UTC] USER=www-data EUID=0 PID=2968831 ACTION=fsop ARGS=chmod 770 /var/lib/kafka/identity-sau-main-dev_coordinator-meta /opt/kafka/config/identity-sau-main-dev/coordinator /var/lib/kafka/identity-sau-main-dev_coordinator-data
[INFO] Adding eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com to /etc/hosts -> 10.100.1.213
[INFO] Adding eventbus-identity-sau-main-dev-kafka-connect.fastorder.com to /etc/hosts -> 10.100.1.212
[INFO] Setting up KRaft for: identity-sau-main-dev (host=eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com ip=10.100.1.213)
[2026-01-03 07:39:04 UTC] USER=www-data EUID=0 PID=2968855 ACTION=fsop ARGS=mkdir -p /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev
[INFO] Generated cluster.id=RlR_k9wRTDCXox5sByRqYQ

🔧 Configuring Kafka for KRaft mode...
[2026-01-03 07:39:08 UTC] USER=www-data EUID=0 PID=2969315 ACTION=fsop ARGS=test -f /opt/kafka/config/identity-sau-main-dev/coordinator/server.properties
[2026-01-03 07:39:08 UTC] USER=www-data EUID=0 PID=2969324 ACTION=fsop ARGS=test -r /opt/kafka/config/identity-sau-main-dev/coordinator/server.properties
[2026-01-03 07:39:08 UTC] USER=www-data EUID=0 PID=2969333 ACTION=fsop ARGS=sed -i /^zookeeper\.connect=/d /opt/kafka/config/identity-sau-main-dev/coordinator/server.properties
[2026-01-03 07:39:08 UTC] USER=www-data EUID=0 PID=2969342 ACTION=passthru ARGS=bash -c grep -q '^process.roles=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties'
[2026-01-03 07:39:08 UTC] USER=www-data EUID=0 PID=2969361 ACTION=passthru ARGS=bash -c grep -q '^node.id=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties'
[2026-01-03 07:39:08 UTC] USER=www-data EUID=0 PID=2969380 ACTION=passthru ARGS=bash -c grep -q '^broker.id=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties'
[2026-01-03 07:39:08 UTC] USER=www-data EUID=0 PID=2969389 ACTION=fsop ARGS=sed -i s|^broker.id=.*|broker.id=1| /opt/kafka/config/identity-sau-main-dev/coordinator/server.properties
[2026-01-03 07:39:08 UTC] USER=www-data EUID=0 PID=2969398 ACTION=passthru ARGS=bash -c grep -q '^controller.listener.names=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties'
[2026-01-03 07:39:08 UTC] USER=www-data EUID=0 PID=2969417 ACTION=passthru ARGS=bash -c grep -q '^controller.quorum.voters=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties'
[2026-01-03 07:39:08 UTC] USER=www-data EUID=0 PID=2969436 ACTION=passthru ARGS=bash -c grep -q '^metadata.log.dir=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties'
[2026-01-03 07:39:08 UTC] USER=www-data EUID=0 PID=2969455 ACTION=passthru ARGS=bash -c grep -q '^log.dirs=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties'
[2026-01-03 07:39:08 UTC] USER=www-data EUID=0 PID=2969464 ACTION=fsop ARGS=sed -i s|^log.dirs=.*|log.dirs=/var/lib/kafka/identity-sau-main-dev_coordinator-data| /opt/kafka/config/identity-sau-main-dev/coordinator/server.properties
[2026-01-03 07:39:08 UTC] USER=www-data EUID=0 PID=2969473 ACTION=passthru ARGS=bash -c grep -q '^listeners=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties'
[2026-01-03 07:39:08 UTC] USER=www-data EUID=0 PID=2969482 ACTION=passthru ARGS=bash -c grep -q 'CONTROLLER://' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties'
[2026-01-03 07:39:08 UTC] USER=www-data EUID=0 PID=2969491 ACTION=fsop ARGS=sed -i s|^listeners=.*|listeners=SSL://10.100.1.213:9092,CONTROLLER://10.100.1.213:9093| /opt/kafka/config/identity-sau-main-dev/coordinator/server.properties
[2026-01-03 07:39:08 UTC] USER=www-data EUID=0 PID=2969501 ACTION=passthru ARGS=bash -c grep -q '^advertised.listeners=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties'
[2026-01-03 07:39:09 UTC] USER=www-data EUID=0 PID=2969510 ACTION=fsop ARGS=sed -i s|^advertised.listeners=.*|advertised.listeners=SSL://eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com:9092| /opt/kafka/config/identity-sau-main-dev/coordinator/server.properties
[2026-01-03 07:39:09 UTC] USER=www-data EUID=0 PID=2969519 ACTION=passthru ARGS=bash -c grep -q '^listener.security.protocol.map=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties'
[2026-01-03 07:39:09 UTC] USER=www-data EUID=0 PID=2969528 ACTION=fsop ARGS=sed -i s|^listener.security.protocol.map=.*|listener.security.protocol.map=SSL:SSL,CONTROLLER:PLAINTEXT| /opt/kafka/config/identity-sau-main-dev/coordinator/server.properties
[2026-01-03 07:39:09 UTC] USER=www-data EUID=0 PID=2969537 ACTION=passthru ARGS=bash -c grep -q '^inter.broker.listener.name=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties'
[2026-01-03 07:39:09 UTC] USER=www-data EUID=0 PID=2969546 ACTION=fsop ARGS=sed -i s|^inter.broker.listener.name=.*|inter.broker.listener.name=SSL| /opt/kafka/config/identity-sau-main-dev/coordinator/server.properties
[2026-01-03 07:39:09 UTC] USER=www-data EUID=0 PID=2969555 ACTION=passthru ARGS=bash -c grep -q '^offsets.topic.replication.factor=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties'
[2026-01-03 07:39:09 UTC] USER=www-data EUID=0 PID=2969564 ACTION=fsop ARGS=sed -i s|^offsets.topic.replication.factor=.*|offsets.topic.replication.factor=1| /opt/kafka/config/identity-sau-main-dev/coordinator/server.properties
[2026-01-03 07:39:09 UTC] USER=www-data EUID=0 PID=2969573 ACTION=passthru ARGS=bash -c grep -q '^transaction.state.log.replication.factor=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties'
[2026-01-03 07:39:09 UTC] USER=www-data EUID=0 PID=2969582 ACTION=fsop ARGS=sed -i s|^transaction.state.log.replication.factor=.*|transaction.state.log.replication.factor=1| /opt/kafka/config/identity-sau-main-dev/coordinator/server.properties
[2026-01-03 07:39:09 UTC] USER=www-data EUID=0 PID=2969591 ACTION=passthru ARGS=bash -c grep -q '^transaction.state.log.min.isr=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties'
[2026-01-03 07:39:09 UTC] USER=www-data EUID=0 PID=2969600 ACTION=fsop ARGS=sed -i s|^transaction.state.log.min.isr=.*|transaction.state.log.min.isr=1| /opt/kafka/config/identity-sau-main-dev/coordinator/server.properties
[2026-01-03 07:39:09 UTC] USER=www-data EUID=0 PID=2969609 ACTION=passthru ARGS=bash -c grep -q '^min.insync.replicas=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties'
  ✅ KRaft configuration applied to server.properties
[2026-01-03 07:39:09 UTC] USER=www-data EUID=0 PID=2969628 ACTION=fsop ARGS=test -f /var/lib/kafka/identity-sau-main-dev_coordinator-meta/meta.properties
[2026-01-03 07:39:09 UTC] USER=www-data EUID=0 PID=2969637 ACTION=fsop ARGS=test -f /var/lib/kafka/identity-sau-main-dev_coordinator-data/meta.properties
[INFO] Already formatted: both /var/lib/kafka/identity-sau-main-dev_coordinator-meta and /var/lib/kafka/identity-sau-main-dev_coordinator-data have meta.properties
🔧 Creating/refreshing KRaft systemd unit...
[2026-01-03 07:39:09 UTC] USER=www-data EUID=0 PID=2969655 ACTION=fsop ARGS=sed -i s|\\$MAINPID|$MAINPID|g /etc/systemd/system/confluent-kraft-identity-sau-main-dev_coordinator.service
[2026-01-03 07:39:09 UTC] USER=www-data EUID=0 PID=2969664 ACTION=passthru ARGS=systemctl daemon-reload
  ✅ Ensured confluent-kraft-identity-sau-main-dev_coordinator.service
🛑 Stopping legacy ZooKeeper-mode services and current KRaft instance...
  🛑 Stopping current: confluent-kraft-identity-sau-main-dev_coordinator.service
[2026-01-03 07:39:10 UTC] USER=www-data EUID=0 PID=2969710 ACTION=passthru ARGS=systemctl stop confluent-kraft-identity-sau-main-dev_coordinator.service
  🧹 Cleaning up rogue Kafka processes...
  🧹 Killing any processes holding Kafka ports 9092, 9093...
  🔪 Killing processes on port 9092: 2067199
[2026-01-03 07:39:14 UTC] USER=www-data EUID=0 PID=2969760 ACTION=passthru ARGS=bash -c kill -9 2067199
  🔪 Killing processes on port 9093: 1595983
2954501
[2026-01-03 07:39:17 UTC] USER=www-data EUID=0 PID=2969798 ACTION=passthru ARGS=bash -c kill -9 1595983
[2026-01-03 07:39:17 UTC] USER=www-data EUID=0 PID=2969809 ACTION=passthru ARGS=bash -c kill -9 2954501
  ✅ Legacy services stopped and rogue processes cleaned
🔓 Removing stale lock files...
[2026-01-03 07:39:22 UTC] USER=www-data EUID=0 PID=2969851 ACTION=fsop ARGS=test -f /var/lib/kafka/identity-sau-main-dev_coordinator-meta/.lock
[2026-01-03 07:39:22 UTC] USER=www-data EUID=0 PID=2969861 ACTION=fsop ARGS=test -f /var/lib/kafka/identity-sau-main-dev_coordinator-data/.lock
  ✅ Lock file check complete
🚀 Starting confluent-kraft-identity-sau-main-dev_coordinator.service ...
[2026-01-03 07:39:22 UTC] USER=www-data EUID=0 PID=2969870 ACTION=passthru ARGS=systemctl enable confluent-kraft-identity-sau-main-dev_coordinator.service
[2026-01-03 07:39:23 UTC] USER=www-data EUID=0 PID=2969921 ACTION=passthru ARGS=systemctl restart confluent-kraft-identity-sau-main-dev_coordinator.service
🔧 Patching shared Connect unit to follow KRaft broker...
[2026-01-03 07:39:26 UTC] USER=www-data EUID=0 PID=2970360 ACTION=fsop ARGS=sed -i -e s|${FULL_ENV}|identity-sau-main-dev|g -e s|${IDENTIFIER}|coordinator|g -e s|${CONFIG_DIR}|/opt/kafka/config/identity-sau-main-dev/coordinator|g /etc/systemd/system/confluent-connect-identity-sau-main-dev_coordinator.service
[2026-01-03 07:39:26 UTC] USER=www-data EUID=0 PID=2970369 ACTION=fsop ARGS=sed -i s|\\$MAINPID|$MAINPID|g /etc/systemd/system/confluent-connect-identity-sau-main-dev_coordinator.service
[2026-01-03 07:39:26 UTC] USER=www-data EUID=0 PID=2970379 ACTION=fsop ARGS=sed -i s|^After=.*|After=network-online.target confluent-kraft-identity-sau-main-dev_coordinator.service| /etc/systemd/system/confluent-connect-identity-sau-main-dev_coordinator.service
[2026-01-03 07:39:26 UTC] USER=www-data EUID=0 PID=2970390 ACTION=fsop ARGS=sed -i s|^Wants=.*|Wants=confluent-kraft-identity-sau-main-dev_coordinator.service| /etc/systemd/system/confluent-connect-identity-sau-main-dev_coordinator.service
[2026-01-03 07:39:26 UTC] USER=www-data EUID=0 PID=2970402 ACTION=fsop ARGS=sed -i s|^ExecStart=.*|ExecStart=/opt/kafka/bin/connect-distributed.sh /opt/kafka/config/identity-sau-main-dev/coordinator/connect-distributed.properties| /etc/systemd/system/confluent-connect-identity-sau-main-dev_coordinator.service
[2026-01-03 07:39:26 UTC] USER=www-data EUID=0 PID=2970414 ACTION=passthru ARGS=systemctl daemon-reload
  ✅ Connect unit patched
[2026-01-03 07:39:27 UTC] USER=www-data EUID=0 PID=2970467 ACTION=fsop ARGS=test -f /opt/kafka/config/identity-sau-main-dev/coordinator/connect-distributed.properties
[2026-01-03 07:39:27 UTC] USER=www-data EUID=0 PID=2970476 ACTION=fsop ARGS=ln -sf /opt/kafka/config/identity-sau-main-dev/coordinator/connect-distributed.properties /opt/kafka/config/connect-distributed.properties
⏳ Waiting for broker coordinator on SSL://eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com:9092 ...
⏳ Waiting for KRaft broker... (attempt 1, 0s/600s)
   Debug: Last error was: [2026-01-03 07:39:27 UTC] USER=www-data EUID=0 PID=2970488 ACTION=passthru ARGS=bash -c timeout 5 sudo -u kafka /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server 'eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com:9092' --command-config '/etc/kafka/client-identity-sau-main-dev-coordinator.properties' describe --status
[2026-01-03 07:39:30,611] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com/10.100.1.213:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2026-01-03 07:39:30,723] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com/10.100.1.213:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2026-01-03 07:39:30,928] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com/10.100.1.213:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2026-01-03 07:39:31,131] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com/10.100.1.213:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2026-01-03 07:39:31,636] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com/10.100.1.213:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2026-01-03 07:39:32,643] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com/10.100.1.213:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
✅ coordinator responded after 13s (attempt 3)
---- server.properties (key lines) ----
[2026-01-03 07:40:03 UTC] USER=www-data EUID=0 PID=2974213 ACTION=passthru ARGS=bash -c grep -E '^(listeners|advertised\.listeners|process\.roles|controller\.quorum\.voters|controller\.listener\.names|inter\.broker\.listener\.name|log\.dirs|metadata\.log\.dir)=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties'
listeners=SSL://10.100.1.213:9092,CONTROLLER://10.100.1.213:9093
advertised.listeners=SSL://eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com:9092
inter.broker.listener.name=SSL
log.dirs=/var/lib/kafka/identity-sau-main-dev_coordinator-data
process.roles=broker,controller
controller.listener.names=CONTROLLER
controller.quorum.voters=1@10.100.1.213:9093
metadata.log.dir=/var/lib/kafka/identity-sau-main-dev_coordinator-meta
---------------------------------------

✅ KRaft setup complete for identity-sau-main-dev_coordinator
  server.properties : /opt/kafka/config/identity-sau-main-dev/coordinator/server.properties
  data dir          : /var/lib/kafka/identity-sau-main-dev_coordinator-data
  meta dir          : /var/lib/kafka/identity-sau-main-dev_coordinator-meta
  systemd unit      : confluent-kraft-identity-sau-main-dev_coordinator.service

🔧 Kafka Configuration Modified:
  ✓ process.roles, node.id, controller.quorum.voters, controller.listener.names
  ✓ listeners (SSL + CONTROLLER) and advertised.listeners (FQDN fallback to IP)
  ✓ listener.security.protocol.map, inter.broker.listener.name
  ✓ log.dirs -> /var/lib/kafka/identity-sau-main-dev_coordinator-data, metadata.log.dir -> /var/lib/kafka/identity-sau-main-dev_coordinator-meta
  ✓ removed zookeeper.connect (if present)
  ✓ created/refreshed dedicated KRaft systemd unit
  ✓ patched shared Connect unit to follow KRaft broker
  ✓ symlinked /opt/kafka/config/identity-sau-main-dev/coordinator/connect-distributed.properties -> /opt/kafka/config/connect-distributed.properties (compat)

🔎 Check quorum:
  /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com:9092 --command-config /etc/kafka/client-identity-sau-main-dev-coordinator.properties describe --status

📋 Next steps:
  1) Review KRaft config:   sudo grep -E 'process.roles|node.id|controller|listeners|advertised.listeners|log.dirs|metadata.log.dir' /opt/kafka/config/identity-sau-main-dev/coordinator/server.properties
  2) Verify topics:         /opt/kafka/bin/kafka-topics.sh --bootstrap-server eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com:9092 --command-config /etc/kafka/client-identity-sau-main-dev-coordinator.properties --list

✅ KRaft metadata layer setup completed successfully

Next steps:
  1. Verify KRaft quorum status
  2. Create Kafka topics
  3. Configure Kafka Connect
[2026-01-03 07:40:03 UTC] USER=www-data EUID=0 PID=2974223 ACTION=fsop ARGS=mkdir -p /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev
[INFO] Saved metadata mode to: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/kafka_metadata_mode

════════════════════════════════════════════════════════════════════

✅ Kafka Metadata Layer Setup Complete

  Mode           : kraft
  Environment    : identity-sau-main-dev
  State saved    : /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/kafka_metadata_mode

  KRaft cluster.id: RlR_k9wRTDCXox5sByRqYQ

  Verify quorum:
    kafka-metadata-quorum.sh --bootstrap-server ... describe

════════════════════════════════════════════════════════════════════

[OK] ✅ Step 3 completed: 02-metadata.sh

[INFO] 📦 Step 4/9: restart kafka related services...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-03 07:40:04 UTC] USER=www-data EUID=0 PID=2974269 ACTION=fsop ARGS=test -f /opt/kafka/config/identity-sau-main-dev/coordinator/server.properties
[2026-01-03 07:40:04 UTC] USER=www-data EUID=0 PID=2974278 ACTION=passthru ARGS=bash -c grep -E '^[[:space:]]*process\.roles=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties' | grep -Eq '(broker|controller)'
[INFO] 📋 Detected mode from server.properties: kraft
[2026-01-03 07:40:05 UTC] USER=www-data EUID=0 PID=2974356 ACTION=passthru ARGS=systemctl stop confluent-connect-identity-sau-main-dev_coordinator.service
[2026-01-03 07:40:12 UTC] USER=www-data EUID=0 PID=2974444 ACTION=passthru ARGS=systemctl stop confluent-kafka-zk-identity-sau-main-dev_coordinator.service
[2026-01-03 07:40:13 UTC] USER=www-data EUID=0 PID=2974500 ACTION=passthru ARGS=systemctl stop confluent-zookeeper-identity-sau-main-dev_coordinator.service
Failed to stop confluent-zookeeper-identity-sau-main-dev_coordinator.service: Unit confluent-zookeeper-identity-sau-main-dev_coordinator.service not loaded.
[INFO] 🧹 Removing stale Kafka lock files...
[2026-01-03 07:40:16 UTC] USER=www-data EUID=0 PID=2974510 ACTION=fsop ARGS=rm -f /var/lib/kafka/identity-sau-main-dev_coordinator-meta/.lock
[2026-01-03 07:40:16 UTC] USER=www-data EUID=0 PID=2974521 ACTION=fsop ARGS=rm -f /var/lib/kafka/identity-sau-main-dev_coordinator-data/.lock
[INFO] 🧹 Cleaning up orphaned processes on Kafka ports...
[2026-01-03 07:40:16 UTC] USER=www-data EUID=0 PID=2974530 ACTION=passthru ARGS=bash -c 
for port in 9092 9093 8083 2181; do
  pids=$(lsof -ti tcp:$port 2>/dev/null || true)
  if [[ -n "$pids" ]]; then
    echo "   Killing orphaned processes on port $port: $pids"
    kill -9 $pids 2>/dev/null || true
    sleep 1
  fi
done

   Killing orphaned processes on port 9092: 2969928
   Killing orphaned processes on port 9093: 2970462
2970465

🚀 Restarting Kafka components…
[INFO] 🚀 starting confluent-kraft-identity-sau-main-dev_coordinator.service…
[2026-01-03 07:40:21 UTC] USER=www-data EUID=0 PID=2974648 ACTION=passthru ARGS=systemctl restart confluent-kraft-identity-sau-main-dev_coordinator.service
[INFO] 🚀 starting confluent-connect-identity-sau-main-dev_coordinator.service…
[2026-01-03 07:40:22 UTC] USER=www-data EUID=0 PID=2975162 ACTION=passthru ARGS=systemctl restart confluent-connect-identity-sau-main-dev_coordinator.service

[INFO] ⏳ Waiting for Kafka broker readiness (FQDN: eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com, IP: 10.100.1.213) ...
[OK] ✅ Broker ready (attempt 1)
[OK] ✅ Port 9092 listening (Kafka Broker)
[OK] ✅ Port 8083 listening (Kafka Connect REST)
[INFO] ⏳ Waiting for Connect REST at https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083 …
[OK] ✅ Connect REST is up (attempt 1)

📋 Reconciling Connect internal topics…
  [ok] connect-configs exists
  [ok] connect-offsets exists
  [ok] connect-status exists

═══════════════════════════════════════════════════════════════════
                           KAFKA SUMMARY
═══════════════════════════════════════════════════════════════════
Env: identity-sau-main-dev   Identifier: coordinator   Mode: kraft
Broker Unit : confluent-kraft-identity-sau-main-dev_coordinator.service  (status: active)
Connect Unit: confluent-connect-identity-sau-main-dev_coordinator.service (status: active)
Bootstrap   : eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com:9092
Connect URL : https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
═══════════════════════════════════════════════════════════════════
[OK] ✅ All required services are up.
[OK] ✅ Step 4 completed: 03-restart-kafka-related-services.sh

[INFO] 📦 Step 5/9: checking services...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-03 07:40:53 UTC] USER=www-data EUID=0 PID=2977482 ACTION=passthru ARGS=bash -c grep -E '^[[:space:]]*process\.roles=' '/opt/kafka/config/identity-sau-main-dev/coordinator/server.properties' | grep -Eq '(broker|controller)'
[INFO] Detected mode from server.properties: kraft

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 1: Service status
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] confluent-kraft-identity-sau-main-dev_coordinator.service status: active
[WARN] confluent-kafka-zk-identity-sau-main-dev_coordinator.service present but should be stopped in KRaft
[WARN] confluent-zookeeper-identity-sau-main-dev_coordinator.service present but not required in KRaft
[OK] confluent-connect-identity-sau-main-dev_coordinator.service status: active

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 2: Port checks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Port 9092 listening (Kafka Broker)
[OK] ✅ Port 8083 listening (Kafka Connect REST)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 3: Broker readiness
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] Broker API responding (attempt 1)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 4: Kafka Connect REST
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] Connect REST responding (attempt 1)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Kafka Services Summary
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment : identity-sau-main-dev
Identifier  : coordinator
Mode        : kraft
Broker Unit : confluent-kraft-identity-sau-main-dev_coordinator.service  (status: active)
Connect Unit: confluent-connect-identity-sau-main-dev_coordinator.service (status: active)
Broker FQDN : eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com:9092
Broker IP   : eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com:9092
Connect URL : https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ All required services are reachable.
[OK] ✅ Step 5 completed: 04-checking-services.sh

[INFO] 📦 Step 6/9: create audit topic...
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Creating Kafka Audit Topics
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Environment: identity-sau-main-dev
[INFO] Replication Factor: 1

[INFO] Waiting for Kafka to be ready...
[ERROR] Kafka not ready after 60s. Skipping audit topic creation.
[OK] ✅ Step 6 completed: 05-create-audit-topic.sh

[INFO] 📦 Step 7/9: setup backups...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Kafka Backup Configuration
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Environment: identity-sau-main-dev

🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[INFO] 1️⃣ Creating S3 bucket for Kafka backups...
make_bucket failed: s3://fastorder-kafka-backups-identity-sau-main-dev An error occurred (AccessDenied) when calling the CreateBucket operation: User: arn:aws:iam::464621692046:user/fo-dev is not authorized to perform: s3:CreateBucket on resource: "arn:aws:s3:::fastorder-kafka-backups-identity-sau-main-dev" because no identity-based policy allows the s3:CreateBucket action

An error occurred (NoSuchBucket) when calling the PutBucketVersioning operation: The specified bucket does not exist

Parameter validation failed:
Unknown parameter in LifecycleConfiguration.Rules[0]: "Id", must be one of: Expiration, ID, Prefix, Filter, Status, Transitions, NoncurrentVersionTransitions, NoncurrentVersionExpiration, AbortIncompleteMultipartUpload
[OK] ✅ S3 bucket created: fastorder-kafka-backups-identity-sau-main-dev

[INFO] 2️⃣ Creating local backup directory...
[2026-01-03 07:43:01 UTC] USER=www-data EUID=0 PID=2980836 ACTION=fsop ARGS=mkdir -p /var/backups/kafka/identity-sau-main-dev
[2026-01-03 07:43:01 UTC] USER=www-data EUID=0 PID=2980848 ACTION=fsop ARGS=mkdir -p /var/backups/kafka/identity-sau-main-dev/topics
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2980857 ACTION=fsop ARGS=mkdir -p /var/backups/kafka/identity-sau-main-dev/metadata
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2980867 ACTION=fsop ARGS=mkdir -p /var/log/kafka/backups
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2980879 ACTION=fsop ARGS=chown -R kafka:kafka /var/backups/kafka/identity-sau-main-dev
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2980888 ACTION=fsop ARGS=chown -R kafka:kafka /var/log/kafka/backups
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2980897 ACTION=fsop ARGS=chmod 750 /var/backups/kafka/identity-sau-main-dev
[OK] ✅ Local backup directory created

[INFO] 3️⃣ Creating topic backup script...
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2980915 ACTION=fsop ARGS=sed -i s|__ENV_ID__|identity-sau-main-dev|g /usr/local/bin/kafka-backup-identity-sau-main-dev.sh
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2980924 ACTION=fsop ARGS=sed -i s|__KAFKA_BROKER__|eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com:9092|g /usr/local/bin/kafka-backup-identity-sau-main-dev.sh
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2980934 ACTION=fsop ARGS=sed -i s|__BACKUP_DIR__|/var/backups/kafka/identity-sau-main-dev|g /usr/local/bin/kafka-backup-identity-sau-main-dev.sh
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2980946 ACTION=fsop ARGS=sed -i s|__S3_BUCKET__|fastorder-kafka-backups-identity-sau-main-dev|g /usr/local/bin/kafka-backup-identity-sau-main-dev.sh
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2980955 ACTION=fsop ARGS=sed -i s|__S3_REGION__|me-central-1|g /usr/local/bin/kafka-backup-identity-sau-main-dev.sh
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2980964 ACTION=fsop ARGS=chmod 750 /usr/local/bin/kafka-backup-identity-sau-main-dev.sh
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2980974 ACTION=fsop ARGS=chown root:kafka /usr/local/bin/kafka-backup-identity-sau-main-dev.sh
[OK] ✅ Backup script created: /usr/local/bin/kafka-backup-identity-sau-main-dev.sh

[INFO] 4️⃣ Setting up cron jobs for automated backups...
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2980992 ACTION=fsop ARGS=chmod 644 /etc/cron.d/kafka-backups-identity-sau-main-dev
[OK] ✅ Cron job configured: Daily backups at 2:00 AM

[INFO] 5️⃣ Creating restore documentation...
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2981012 ACTION=fsop ARGS=sed -i s|__S3_BUCKET__|fastorder-kafka-backups-identity-sau-main-dev|g /var/backups/kafka/identity-sau-main-dev/RESTORE_INSTRUCTIONS.md
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2981021 ACTION=fsop ARGS=sed -i s|__S3_REGION__|me-central-1|g /var/backups/kafka/identity-sau-main-dev/RESTORE_INSTRUCTIONS.md
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2981030 ACTION=fsop ARGS=sed -i s|__KAFKA_BROKER__|eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com|g /var/backups/kafka/identity-sau-main-dev/RESTORE_INSTRUCTIONS.md
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2981039 ACTION=fsop ARGS=chmod 644 /var/backups/kafka/identity-sau-main-dev/RESTORE_INSTRUCTIONS.md
[2026-01-03 07:43:02 UTC] USER=www-data EUID=0 PID=2981048 ACTION=fsop ARGS=chown kafka:kafka /var/backups/kafka/identity-sau-main-dev/RESTORE_INSTRUCTIONS.md
[OK] ✅ Restore documentation created: /var/backups/kafka/identity-sau-main-dev/RESTORE_INSTRUCTIONS.md

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Kafka Backup Configured
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] S3 Bucket: fastorder-kafka-backups-identity-sau-main-dev
[INFO] Region: me-central-1
[INFO] Local backup dir: /var/backups/kafka/identity-sau-main-dev
[INFO] Schedule: Daily at 2:00 AM
[INFO] Script: /usr/local/bin/kafka-backup-identity-sau-main-dev.sh
[INFO] Restore docs: /var/backups/kafka/identity-sau-main-dev/RESTORE_INSTRUCTIONS.md

[WARN] ⚠️  Note: This backs up Kafka metadata only (topics, configs, offsets)
[WARN]    For full message data backup, configure Kafka Connect S3 Sink

[OK] ✅ Step 7 completed: 06-setup-backups.sh

[INFO] 📦 Step 8/9: monitoring setup...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Kafka Monitoring Integration for identity-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[OK]   ✓ Observability cell is ready

[INFO] 2️⃣ Setting up Kafka JMX exporter integration...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[INFO] Setting up Kafka JMX exporter for identity-sau-main-dev
[INFO] JMX Prometheus Java Agent already exists at /opt/kafka/libs/jmx_prometheus_javaagent.jar
[2026-01-03 07:43:03 UTC] USER=www-data EUID=0 PID=2981088 ACTION=passthru ARGS=mv /tmp/jmx_exporter.yml /opt/kafka/config/jmx_exporter.yml
[2026-01-03 07:43:03 UTC] USER=www-data EUID=0 PID=2981097 ACTION=passthru ARGS=chmod 644 /opt/kafka/config/jmx_exporter.yml
[OK]   JMX exporter configuration created at /opt/kafka/config/jmx_exporter.yml
[OK]   JMX exporter configuration created
[INFO] Configuring Kafka systemd services to use JMX exporter...
[2026-01-03 07:43:03 UTC] USER=www-data EUID=0 PID=2981121 ACTION=fsop ARGS=test -f /etc/systemd/system/[2026-01-03
[INFO] All Kafka services already configured with JMX exporter
[OK]   Kafka JMX exporter integration complete
[INFO] Metrics endpoint: http://142.93.238.16:9308/metrics
[INFO] Prometheus will automatically scrape: https://metrics-identity-sau-main-dev.fastorder.com:9090
[INFO] View dashboards at: https://dashboards-identity-sau-main-dev.fastorder.com
[OK]   ✓ Kafka JMX exporter integration complete
[INFO] Configuring KAFKA_OPTS environment variable for kafka user...
[2026-01-03 07:43:03 UTC] USER=www-data EUID=0 PID=2981142 ACTION=passthru ARGS=grep -q KAFKA_OPTS.*javaagent.*jmx_prometheus_javaagent /home/kafka/.bashrc
[OK]   ✓ KAFKA_OPTS already configured
[INFO] 2.5️⃣ Enabling JMX exporter in Kafka systemd service...
[2026-01-03 07:43:03 UTC] USER=www-data EUID=0 PID=2981164 ACTION=passthru ARGS=grep -q javaagent.*jmx_prometheus_javaagent /etc/systemd/system/confluent-kraft-identity-sau-main-dev_coordinator.service
[INFO] Updating confluent-kraft-identity-sau-main-dev_coordinator.service to enable JMX exporter...
[2026-01-03 07:43:03 UTC] USER=www-data EUID=0 PID=2981185 ACTION=passthru ARGS=sed -i s|^Environment=KAFKA_OPTS=.*|Environment=KAFKA_OPTS=-javaagent:/opt/kafka/libs/jmx_prometheus_javaagent.jar=9308:/opt/kafka/config/jmx_exporter.yml| /etc/systemd/system/confluent-kraft-identity-sau-main-dev_coordinator.service
[OK]   ✓ Updated confluent-kraft-identity-sau-main-dev_coordinator.service
[INFO] Reloading systemd daemon and restarting Kafka services...
[2026-01-03 07:43:03 UTC] USER=www-data EUID=0 PID=2981208 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-03 07:43:04 UTC] USER=www-data EUID=0 PID=2981279 ACTION=passthru ARGS=systemctl is-active --quiet confluent-kraft-identity-sau-main-dev_coordinator
[INFO] Restarting confluent-kraft-identity-sau-main-dev_coordinator...
[2026-01-03 07:43:04 UTC] USER=www-data EUID=0 PID=2981300 ACTION=passthru ARGS=systemctl restart confluent-kraft-identity-sau-main-dev_coordinator
[2026-01-03 07:43:09 UTC] USER=www-data EUID=0 PID=2981813 ACTION=passthru ARGS=systemctl is-active --quiet confluent-kraft-identity-sau-main-dev_coordinator
[OK]   ✓ confluent-kraft-identity-sau-main-dev_coordinator restarted successfully
[OK]   ✓ JMX exporter enabled in Kafka systemd services
[INFO] 2.6️⃣ Configuring Prometheus to scrape Kafka metrics...
[2026-01-03 07:43:09 UTC] USER=www-data EUID=0 PID=2981839 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[INFO] Adding Kafka scrape target to Prometheus configuration...
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[2026-01-03 07:43:09 UTC] USER=www-data EUID=0 PID=2981872 ACTION=passthru ARGS=sed -i /# Prometheus self-monitoring/r /tmp/prometheus_kafka_add.yml /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[ERROR] Invalid Prometheus configuration - rolling back
[2026-01-03 07:43:09 UTC] USER=www-data EUID=0 PID=2981905 ACTION=passthru ARGS=sed -i /job_name: 'kafka'/,+6d /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[2026-01-03 07:43:09 UTC] USER=www-data EUID=0 PID=2981926 ACTION=fsop ARGS=rm -f /tmp/prometheus_kafka_add.yml

[INFO] 3️⃣ Registering Kafka nodes to monitoring database...
[INFO] Registering Kafka Broker to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Kafka Broker
[INFO]   Identifier:        identity-sau-main-dev-broker-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                142.93.238.16
[INFO]   Port:              9092
[INFO]   FQDN:              eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 1a310579-24b9-4091-8626-7335f80305c3
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ Kafka broker registered
[INFO] Registering Kafka Connect to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Kafka Connect
[INFO]   Identifier:        identity-sau-main-dev-connect-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8083
[INFO]   FQDN:              eventbus-identity-sau-main-dev-kafka-connect.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 71719f62-65ea-4a2b-a0ed-4a8d3f80403b
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ Kafka Connect registered
[INFO] Schema Registry not running, skipping registration

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Kafka Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Metrics: http://localhost:9308/metrics
[INFO] Prometheus: https://metrics-identity-sau-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-identity-sau-main-dev.fastorder.com
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Step 8 completed: 10-monitoring-setup.sh

[INFO] 📦 Step 9/9: update www data certs...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
📋 Post-Kafka Setup: Updating www-data Kafka certificates...
   Environment: identity-sau-main-dev
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev

✓ Kafka certificates found
✓ www-data user exists

[2026-01-03 07:43:11 UTC] USER=www-data EUID=0 PID=2982050 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:43:11 UTC] USER=www-data EUID=0 PID=2982063 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:43:11 UTC] USER=www-data EUID=0 PID=2982077 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-03 07:43:11 UTC] USER=www-data EUID=0 PID=2982088 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-03 07:43:12 UTC] USER=www-data EUID=0 PID=2982105 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-03 07:43:12 UTC] USER=www-data EUID=0 PID=2982118 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:43:12 UTC] USER=www-data EUID=0 PID=2982129 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:43:12 UTC] USER=www-data EUID=0 PID=2982140 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem

✅ Kafka certificate symlinks created for www-data
   PHP Kafka consumers can now use:
   - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✓ Post-Kafka setup complete
[OK] ✅ Step 9 completed: 99-update-www-data-certs.sh


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Kafka setup completed successfully!
[OK] Executed all 9 steps
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Environment: identity-sau-main-dev
[INFO] Service: identity
[INFO] Zone: sau
[INFO] Branch: main
[INFO] Env: dev
[INFO] Registering Kafka nodes via API...
[OK] ✔ Kafka node registration completed
[INFO] Setting up Kafka observability integration...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK] Observability cell endpoints registered for identity-sau-main-dev
[INFO] Observability cell verified for identity-sau-main-dev
[INFO] Monitoring will be configured after Kafka deployment (step 10-monitoring-setup.sh)
[INFO] Cleaning up temporary files...
[INFO] Starting cleanup of temporary files...
[INFO] Cleaning up SSL temp files for identity-sau-main-dev...
[INFO] Cleaning up old provisioning logs...
[SUCCESS] Removed 61 old log files
[INFO] Cleaning up old configuration backups...
[OK] ✔ Cleanup completed

✓ ✅ Event bus infrastructure (kafka) setup completed successfully

[INFO] Using database engine from DB_ENGINE environment variable: postgresql
[INFO] Cleaning up any existing locks...

Starting database engine: postgresql
═══════════════════════════════════════════════

[INFO] Loaded from topology.json: identity-sau-main-dev
[2026-01-03 07:43:14] Loaded environment: identity-sau-main-dev
[2026-01-03 07:43:14] Service: identity, Zone: sau, Branch: main, Env: dev
[2026-01-03 07:43:14] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-03 07:43:14] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-03 07:43:14] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[INFO] Observability cell verified for identity-sau-main-dev
[INFO] Monitoring will be configured after PostgreSQL deployment (step 10-monitoring-setup.sh)
[INFO] Citus mode ENABLED
[INFO] → Coordinator + 1 worker(s) + 1 standby node(s) per worker
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up coordinator (Citus control plane)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-03 07:43:15 UTC] USER=unknown EUID=33 PID=2982315 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-03 07:43:15 UTC] USER=unknown EUID=33 PID=2982323 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-03 07:43:15 UTC] USER=unknown EUID=33 PID=2982330 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-03 07:43:15 UTC] USER=unknown EUID=33 PID=2982337 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-03 07:43:15 UTC] USER=unknown EUID=33 PID=2982344 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-03 07:43:15 UTC] USER=unknown EUID=33 PID=2982351 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895 JOB_UUID=b166d639-0d14-4485-904a-cf625a2ce6d8

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] PostgreSQL IP: 10.100.1.203
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

[INFO] Adding /etc/hosts entries for coordinator...
[INFO]   1. db-identity-sau-main-dev-postgresql.fastorder.com → 10.100.1.203 (primary/short)
[INFO]   2. db-identity-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.203 (compatibility)

[INFO]   ➕ Adding db-identity-sau-main-dev-postgresql.fastorder.com → 10.100.1.203
✅     ✅ Added: db-identity-sau-main-dev-postgresql.fastorder.com → 10.100.1.203
[INFO]   ➕ Adding db-identity-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.203
✅     ✅ Added: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.203

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.203    db-identity-sau-main-dev-postgresql.fastorder.com
  10.100.1.203    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-03 07:43:19 UTC] USER=www-data EUID=0 PID=2982538 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator and /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:43:19 UTC] USER=www-data EUID=0 PID=2982547 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-03 07:43:19 UTC] USER=www-data EUID=0 PID=2982558 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-2982492
[2026-01-03 07:43:19 UTC] USER=www-data EUID=0 PID=2982567 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-2982492/ra_root.crt
[2026-01-03 07:43:20 UTC] USER=www-data EUID=0 PID=2982576 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-2982492/ra_root.key
[2026-01-03 07:43:20 UTC] USER=www-data EUID=0 PID=2982585 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-2982492/ra_root.crt
[2026-01-03 07:43:20 UTC] USER=www-data EUID=0 PID=2982594 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-2982492/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-03 07:43:21 UTC] USER=www-data EUID=0 PID=2982637 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2982492/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-03 07:43:21 UTC] USER=www-data EUID=0 PID=2982657 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-03 07:43:21 UTC] USER=www-data EUID=0 PID=2982666 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2982492/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:21 UTC] USER=www-data EUID=0 PID=2982675 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:21 UTC] USER=www-data EUID=0 PID=2982684 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:21 UTC] USER=www-data EUID=0 PID=2982694 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-03 07:43:21 UTC] USER=www-data EUID=0 PID=2982707 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-03 07:43:21 UTC] USER=www-data EUID=0 PID=2982719 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-03 07:43:21 UTC] USER=www-data EUID=0 PID=2982728 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-03 07:43:21 UTC] USER=www-data EUID=0 PID=2982738 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-03 07:43:21 UTC] USER=www-data EUID=0 PID=2982747 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:43:22 UTC] USER=www-data EUID=0 PID=2982756 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        coordinator
Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:43:22 UTC] USER=www-data EUID=0 PID=2982813 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-01-03 07:43:22 UTC] USER=www-data EUID=0 PID=2982823 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-03 07:43:22 UTC] USER=www-data EUID=0 PID=2982832 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-01-03 07:43:22 UTC] USER=www-data EUID=0 PID=2982841 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-03 07:43:22 UTC] USER=www-data EUID=0 PID=2982850 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2982864 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2982873 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2982882 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2982896 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2982905 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2982914 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2982924 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2982933 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2982942 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2982951 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2982960 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2982969 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2982978 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2982989 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2983001 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2983011 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2983037 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:23 UTC] USER=www-data EUID=0 PID=2983048 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983057 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983066 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983075 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983084 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983093 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983106 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983115 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983124 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983144 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983157 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983166 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983175 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983184 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983193 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983206 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983215 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983224 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983233 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983243 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983254 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983265 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983275 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:24 UTC] USER=www-data EUID=0 PID=2983293 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983302 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983311 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983320 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983329 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983338 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983347 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983356 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983367 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983377 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983387 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983396 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983405 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983414 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983423 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983432 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983441 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983450 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983459 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983468 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983477 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983487 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983497 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983506 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983515 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983524 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983533 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983542 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983551 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983560 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:43:25 UTC] USER=www-data EUID=0 PID=2983569 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:43:26 UTC] USER=www-data EUID=0 PID=2983618 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-01-03 07:43:26 UTC] USER=www-data EUID=0 PID=2983628 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-03 07:43:26 UTC] USER=www-data EUID=0 PID=2983642 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-01-03 07:43:26 UTC] USER=www-data EUID=0 PID=2983651 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-03 07:43:26 UTC] USER=www-data EUID=0 PID=2983661 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983676 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983685 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983694 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983703 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983713 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983722 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983731 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983742 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983751 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983760 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983769 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983778 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983787 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983796 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983805 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983814 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983823 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983832 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983859 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983868 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983877 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983886 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983895 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983904 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983913 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-03 07:43:27 UTC] USER=www-data EUID=0 PID=2983922 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2983931 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2983940 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2983949 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2983959 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2983969 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2983983 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2983992 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2984001 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2984010 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2984020 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2984029 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2984038 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2984047 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2984056 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2984065 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2984076 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2984099 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2984108 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2984119 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2984128 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2984137 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2984147 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2984156 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:28 UTC] USER=www-data EUID=0 PID=2984165 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984174 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984183 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984193 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984203 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984212 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984221 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984241 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984250 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984259 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984268 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984277 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984286 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984295 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres_der.key
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984305 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984315 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984324 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984333 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984342 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984351 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984360 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984369 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984378 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:43:29 UTC] USER=www-data EUID=0 PID=2984387 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Creating Postgresql Ident:db-coordinator-postgresql application environment...
[INFO] 🎯 Custom Environment Creation (Example Wrapper)
[INFO] 📁 Orchestrator Library: /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] 💾 State Directory: /opt/fastorder/bash/scripts/env_app_setup/state

[INFO] 🚀 Calling centralized orchestrator: fo-env create-app
[INFO] 📋 Arguments: --service identity --zone sau --branch main --env dev --domain db-identity-sau-main-dev-postgresql-coordinator --app db-coordinator-postgresql

[INFO] Creating application-specific environment configuration
[INFO] Environment ID: identity-sau-main-dev
[INFO] Application: db-coordinator-postgresql
[INFO] Base environment identity-sau-main-dev already exists
[INFO] Allocated db-coordinator-postgresql IP: 10.100.1.214
[INFO] Generated domain: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[INFO] Configuring network interface for db-coordinator-postgresql IP: 10.100.1.214
[INFO] IP 10.100.1.214 is already configured
[INFO] Updating topology with application-specific configuration...
[ OK ] Topology updated with application-specific configuration
[INFO] Binding db-coordinator-postgresql IP to domain: 10.100.1.214 -> db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[WARN] Domain 'db-identity-sau-main-dev-postgresql-coordinator.fastorder.com' already exists in /etc/hosts
[INFO] Removing old entries for domain: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-03 07:43:32 UTC] USER=www-data EUID=0 PID=2984881 ACTION=fsop ARGS=sed -i /\sdb-identity-sau-main-dev-postgresql-coordinator.fastorder.com\(\s\|$\)/d /etc/hosts
[ OK ] Successfully bound db-identity-sau-main-dev-postgresql-coordinator.fastorder.com to 10.100.1.214
[ OK ] Domain correctly mapped
[ OK ] Application environment created successfully!
[INFO] 
[INFO] Application Details:
[INFO]   Environment ID: identity-sau-main-dev
[INFO]   Application: db-coordinator-postgresql
[INFO]   IP: 10.100.1.214
[INFO]   Domain: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[INFO] 
[INFO] To use this application:
[INFO]   source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO]   init_environment db-coordinator-postgresql
[INFO]   echo $VM_IP  # Returns: 10.100.1.214

[ OK ] 🎉 Environment creation completed successfully!

[INFO] 📋 What happened:
[INFO]   ✅ Called centralized orchestrator at /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO]   ✅ All topology.json management handled centrally
[INFO]   ✅ Application-specific IP and domain configured
[INFO]   ✅ Network interface configured and made persistent
[INFO]   ✅ Domain binding added to /etc/hosts (if not skipped)

[INFO] 🔧 To use the centralized orchestrator directly:
[INFO]   # Add orchestrator to PATH
[INFO]   export PATH="/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/bin:$PATH"
[INFO]   # Then call directly
[INFO]   fo-env create-app --service auth --zone uae --env dev --app redis

[INFO] 📚 For more orchestrator commands:
[INFO]   fo-env --help
[ OK ] Created db-coordinator-postgresql environment: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com (10.100.1.214)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.214
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] Data dir:   /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[INFO] Port:       5432
[INFO] Hostname:   db-identity-sau-main-dev-postgresql-coordinator
[2026-01-03 07:43:33 UTC] USER=www-data EUID=0 PID=2984935 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:43:33 UTC] USER=www-data EUID=0 PID=2984978 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:43:33 UTC] USER=www-data EUID=0 PID=2984999 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        coordinator
  Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
  Coordinator variants:
    - db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
    - db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-03 07:43:34 UTC] USER=www-data EUID=0 PID=2985040 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator and /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:43:34 UTC] USER=www-data EUID=0 PID=2985049 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-03 07:43:34 UTC] USER=www-data EUID=0 PID=2985059 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-2985006
[2026-01-03 07:43:34 UTC] USER=www-data EUID=0 PID=2985068 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-2985006/ra_root.crt
[2026-01-03 07:43:34 UTC] USER=www-data EUID=0 PID=2985078 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-2985006/ra_root.key
[2026-01-03 07:43:34 UTC] USER=www-data EUID=0 PID=2985091 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-2985006/ra_root.crt
[2026-01-03 07:43:34 UTC] USER=www-data EUID=0 PID=2985101 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-2985006/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-03 07:43:35 UTC] USER=www-data EUID=0 PID=2985153 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2985006/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-03 07:43:35 UTC] USER=www-data EUID=0 PID=2985163 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2985006/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-03 07:43:35 UTC] USER=www-data EUID=0 PID=2985173 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-03 07:43:35 UTC] USER=www-data EUID=0 PID=2985183 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2985006/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:35 UTC] USER=www-data EUID=0 PID=2985192 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:35 UTC] USER=www-data EUID=0 PID=2985204 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:43:35 UTC] USER=www-data EUID=0 PID=2985213 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-03 07:43:35 UTC] USER=www-data EUID=0 PID=2985225 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-03 07:43:35 UTC] USER=www-data EUID=0 PID=2985235 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-03 07:43:35 UTC] USER=www-data EUID=0 PID=2985244 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-03 07:43:35 UTC] USER=www-data EUID=0 PID=2985253 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-03 07:43:35 UTC] USER=www-data EUID=0 PID=2985262 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:43:36 UTC] USER=www-data EUID=0 PID=2985271 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-identity-sau-main-dev-postgresql.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        coordinator
Primary CN:  db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-coordinator.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-03 07:43:36 UTC] USER=www-data EUID=0 PID=2985300 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-03 07:43:36 UTC] USER=www-data EUID=0 PID=2985309 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.key
[2026-01-03 07:43:36 UTC] USER=www-data EUID=0 PID=2985318 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-03 07:43:36 UTC] USER=www-data EUID=0 PID=2985341 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[2026-01-03 07:43:36 UTC] USER=www-data EUID=0 PID=2985362 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-03 07:43:36 UTC] USER=www-data EUID=0 PID=2985395 ACTION=fsop ARGS=rm -rf /var/run/postgresql-identity-sau-main-dev-coordinator
[OK]   No conflicting Postgres left on port 5432
[OK]   Using postgres password from vault provider
[2026-01-03 07:43:38 UTC] USER=www-data EUID=0 PID=2985466 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.xce5rG
[2026-01-03 07:43:39 UTC] USER=www-data EUID=0 PID=2985487 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.xce5rG
[2026-01-03 07:43:39 UTC] USER=www-data EUID=0 PID=2985513 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-03 07:43:39 UTC] USER=www-data EUID=0 PID=2985559 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/identity-sau-main-dev
[INFO] Initializing cluster in /var/lib/postgresql/17/identity-sau-main-dev/coordinator (SCRAM; pwfile)
[WARN] Removing existing data directory: /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-03 07:43:39 UTC] USER=www-data EUID=0 PID=2985580 ACTION=fsop ARGS=rm -rf /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-03 07:43:39 UTC] USER=www-data EUID=0 PID=2985610 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-03 07:43:40 UTC] USER=www-data EUID=0 PID=2985631 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-03 07:43:40 UTC] USER=www-data EUID=0 PID=2985652 ACTION=fsop ARGS=chmod 700 /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[2026-01-03 07:43:40 UTC] USER=www-data EUID=0 PID=2985675 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-03 07:43:40 UTC] USER=www-data EUID=0 PID=2985696 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-03 07:43:40 UTC] USER=www-data EUID=0 PID=2985717 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-identity-sau-main-dev-coordinator
[2026-01-03 07:43:40 UTC] USER=www-data EUID=0 PID=2985726 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /var/lib/postgresql/17/identity-sau-main-dev/coordinator --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.xce5rG
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/17/identity-sau-main-dev/coordinator ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /var/lib/postgresql/17/identity-sau-main-dev/coordinator -l logfile start

[OK]   initdb complete
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-03 07:43:43 UTC] USER=www-data EUID=0 PID=2985834 ACTION=fsop ARGS=cp /tmp/tmp.NsgKy1jrMP /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
[2026-01-03 07:43:43 UTC] USER=www-data EUID=0 PID=2985857 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
[2026-01-03 07:43:43 UTC] USER=www-data EUID=0 PID=2985882 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@identity-sau-main-dev-coordinator.service
[2026-01-03 07:43:43 UTC] USER=www-data EUID=0 PID=2985909 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.Bo1qTH /etc/systemd/system/postgresql@identity-sau-main-dev-coordinator.service
[2026-01-03 07:43:44 UTC] USER=www-data EUID=0 PID=2985930 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@identity-sau-main-dev-coordinator.service
[OK]   systemd unit written
[2026-01-03 07:43:44 UTC] USER=www-data EUID=0 PID=2985952 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-03 07:43:44 UTC] USER=www-data EUID=0 PID=2985974 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-03 07:43:44 UTC] USER=www-data EUID=0 PID=2985996 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-03 07:43:46 UTC] USER=www-data EUID=0 PID=2986132 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-03 07:43:46 UTC] USER=www-data EUID=0 PID=2986191 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Provisioning application database and Debezium role (if not exists)...
[INFO] Checking if database fastorder_identity_sau_main_dev_db exists...
[INFO] DB check result: exit_code=0, output='[2026-01-03 07:43:48 UTC] USER=www-data EUID=0 PID=2986348 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_identity_sau_main_dev_db''
[INFO] Creating database fastorder_identity_sau_main_dev_db...
[2026-01-03 07:43:48 UTC] USER=www-data EUID=0 PID=2986377 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_identity_sau_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[OK]   Database fastorder_identity_sau_main_dev_db created
[INFO] Checking if role debezium_user exists...
[INFO] Role check result: exit_code=0, output='[2026-01-03 07:43:48 UTC] USER=www-data EUID=0 PID=2986407 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[INFO] Creating role debezium_user...
[2026-01-03 07:43:48 UTC] USER=www-data EUID=0 PID=2986434 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'tQ5T+FD63aicrkQrdeMZhmOd';
CREATE ROLE
[OK]   Role debezium_user created
[2026-01-03 07:43:49 UTC] USER=www-data EUID=0 PID=2986459 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_identity_sau_main_dev_db" TO debezium_user;
GRANT
[OK]   Application DB (fastorder_identity_sau_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[INFO] Applying connection and memory optimizations...
[INFO] Current settings: max_connections=100, work_mem=4MB
[INFO] Target settings (coordinator): max_connections=150, work_mem=8MB
[2026-01-03 07:43:49 UTC] USER=www-data EUID=0 PID=2986539 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 150;
ALTER SYSTEM
[2026-01-03 07:43:49 UTC] USER=www-data EUID=0 PID=2986563 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-03 07:43:49 UTC] USER=www-data EUID=0 PID=2986586 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   Settings applied to postgresql.auto.conf
[2026-01-03 07:43:50 UTC] USER=www-data EUID=0 PID=2986601 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/standby.signal
[INFO] Service recently started (4s ago) - restarting to apply max_connections...
[INFO] Stopping service...
[2026-01-03 07:43:50 UTC] USER=www-data EUID=0 PID=2986628 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[INFO] Waiting for port 5432 to be released...
[OK]   Port 5432 released
[INFO] Starting service...
[2026-01-03 07:43:53 UTC] USER=www-data EUID=0 PID=2986697 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[2026-01-03 07:43:59 UTC] USER=www-data EUID=0 PID=2986816 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[OK]   ✅ Optimization complete: max_connections=150, work_mem=8MB
[INFO] Setting postgres password via centralized script... for coordinator
[INFO] Temporarily disabling synchronous_commit on coordinator for password setting...
[OK]   Disabled synchronous_commit (was: on)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Initial setup: Using password from initdb
✓ PostgreSQL password already set during initdb
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/coordinator
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
[INFO] Restoring synchronous_commit on coordinator...
[OK]   Restored synchronous_commit to: on
[OK]   Password set and persisted
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] PostgreSQL IP: 10.100.1.214
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

[INFO] Adding /etc/hosts entries for coordinator...
[INFO]   1. db-identity-sau-main-dev-postgresql.fastorder.com → 10.100.1.214 (primary/short)
[INFO]   2. db-identity-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.214 (compatibility)

[INFO]   🔄 Updating db-identity-sau-main-dev-postgresql.fastorder.com → 10.100.1.214
✅     ✅ Updated: db-identity-sau-main-dev-postgresql.fastorder.com → 10.100.1.214
[INFO]   ✅ db-identity-sau-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.214    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
  10.100.1.214    db-identity-sau-main-dev-postgresql.fastorder.com


[OK]   PostgreSQL 'identity-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.key \
        host=db-identity-sau-main-dev-postgresql-coordinator port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.214
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-coordinator
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: ce097707-5ce5-40c8-a941-01512555cab8
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 📦 03 role...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-03 07:44:15 UTC] USER=www-data EUID=0 PID=2987685 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    debezium_user
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   debezium_user
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:44:39 UTC] USER=www-data EUID=0 PID=2988038 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-debezium_user
[2026-01-03 07:44:39 UTC] USER=www-data EUID=0 PID=2988047 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-01-03 07:44:39 UTC] USER=www-data EUID=0 PID=2988056 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-debezium_user/ra_root.key
[2026-01-03 07:44:39 UTC] USER=www-data EUID=0 PID=2988065 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-01-03 07:44:39 UTC] USER=www-data EUID=0 PID=2988074 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:44:40 UTC] USER=www-data EUID=0 PID=2988094 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:44:40 UTC] USER=www-data EUID=0 PID=2988103 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:44:40 UTC] USER=www-data EUID=0 PID=2988113 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-03 07:44:40 UTC] USER=www-data EUID=0 PID=2988123 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-03 07:44:40 UTC] USER=www-data EUID=0 PID=2988134 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:44:40 UTC] USER=www-data EUID=0 PID=2988145 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:44:40 UTC] USER=www-data EUID=0 PID=2988154 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-03 07:44:40 UTC] USER=www-data EUID=0 PID=2988163 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-03 07:44:40 UTC] USER=www-data EUID=0 PID=2988175 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-03 07:44:40 UTC] USER=www-data EUID=0 PID=2988184 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:44:40 UTC] USER=www-data EUID=0 PID=2988193 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:44:40 UTC] USER=www-data EUID=0 PID=2988202 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-03 07:44:40 UTC] USER=www-data EUID=0 PID=2988211 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-03 07:44:40 UTC] USER=www-data EUID=0 PID=2988220 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-03 07:44:40 UTC] USER=www-data EUID=0 PID=2988229 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:44:40 UTC] USER=www-data EUID=0 PID=2988238 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:40 UTC] USER=www-data EUID=0 PID=2988267 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988279 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988291 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988318 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988327 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988336 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988348 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988357 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988366 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988379 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988391 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988402 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988411 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988420 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988429 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988438 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988447 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988465 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988474 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988483 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-03 07:44:41 UTC] USER=www-data EUID=0 PID=2988493 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988503 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988514 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988523 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988532 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988541 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988550 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988559 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988568 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988577 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988586 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988595 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988605 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988615 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988624 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988633 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988642 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988651 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988660 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988669 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988678 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988687 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988696 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988705 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988715 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988725 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988738 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988749 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-03 07:44:42 UTC] USER=www-data EUID=0 PID=2988758 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-03 07:44:43 UTC] USER=www-data EUID=0 PID=2988767 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-03 07:44:43 UTC] USER=www-data EUID=0 PID=2988794 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:44:43 UTC] USER=www-data EUID=0 PID=2988803 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: debezium_user
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U debezium_user -d postgres

✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: fastorder_admin_gd
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: coordinator/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-identity-sau-main-dev-postgresql.fastorder.com
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    fastorder_admin_gd
Identifier:  coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        coordinator
  User (CN):   fastorder_admin_gd
  Hostname:    db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:44:58 UTC] USER=www-data EUID=0 PID=2989299 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-fastorder_admin_gd
[2026-01-03 07:44:58 UTC] USER=www-data EUID=0 PID=2989308 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-03 07:44:58 UTC] USER=www-data EUID=0 PID=2989317 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
[2026-01-03 07:44:58 UTC] USER=www-data EUID=0 PID=2989326 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-03 07:44:58 UTC] USER=www-data EUID=0 PID=2989337 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989358 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989367 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989376 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989385 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989394 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989403 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989415 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989424 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989433 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989442 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989451 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989460 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989469 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989478 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989487 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989499 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989540 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989585 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989733 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989774 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989786 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989796 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989806 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989815 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-03 07:44:59 UTC] USER=www-data EUID=0 PID=2989824 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-03 07:45:00 UTC] USER=www-data EUID=0 PID=2989837 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:45:00 UTC] USER=www-data EUID=0 PID=2989848 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:45:00 UTC] USER=www-data EUID=0 PID=2989857 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-03 07:45:00 UTC] USER=www-data EUID=0 PID=2989876 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:45:00 UTC] USER=www-data EUID=0 PID=2989886 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:45:00 UTC] USER=www-data EUID=0 PID=2989895 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:45:00 UTC] USER=www-data EUID=0 PID=2989904 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:45:00 UTC] USER=www-data EUID=0 PID=2989914 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:45:00 UTC] USER=www-data EUID=0 PID=2989923 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:45:00 UTC] USER=www-data EUID=0 PID=2989932 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-03 07:45:00 UTC] USER=www-data EUID=0 PID=2989955 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:45:00 UTC] USER=www-data EUID=0 PID=2989966 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:45:00 UTC] USER=www-data EUID=0 PID=2989975 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-03 07:45:00 UTC] USER=www-data EUID=0 PID=2989984 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-03 07:45:00 UTC] USER=www-data EUID=0 PID=2989994 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:45:00 UTC] USER=www-data EUID=0 PID=2990004 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990031 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990041 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990050 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990059 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990068 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990077 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990088 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990097 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990107 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990117 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990127 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990138 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990147 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990156 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990198 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990237 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990290 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990355 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990420 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-03 07:45:01 UTC] USER=www-data EUID=0 PID=2990484 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-03 07:45:02 UTC] USER=www-data EUID=0 PID=2990571 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:45:02 UTC] USER=www-data EUID=0 PID=2990706 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:45:02 UTC] USER=www-data EUID=0 PID=2990903 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:45:02 UTC] USER=www-data EUID=0 PID=2991047 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-03 07:45:03 UTC] USER=www-data EUID=0 PID=2991100 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-03 07:45:03 UTC] USER=www-data EUID=0 PID=2991215 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-03 07:45:03 UTC] USER=www-data EUID=0 PID=2991252 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:45:03 UTC] USER=www-data EUID=0 PID=2991299 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:45:03 UTC] USER=www-data EUID=0 PID=2991338 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:45:03 UTC] USER=www-data EUID=0 PID=2991408 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: fastorder_admin_gd
Node: coordinator
FQDN: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-coordinator.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-identity-sau-main-dev-coordinator:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_identity_sau_main_dev_db already exists, skipping creation
[2026-01-03 07:45:05 UTC] USER=www-data EUID=0 PID=2991755 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-03 07:45:06 UTC] USER=www-data EUID=0 PID=2991914 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

=== Pre-flight Checks ===
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
✓ AWS Secrets Manager accessible

=== Retrieving Credentials from AWS ===
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
/opt/fastorder/bash/infra_core/cache.sh: line 145: /var/cache/secrets/fastorder_db_identity_sau_main_dev_postgresql_coordinator_fastorder_admin_gd.cache.tmp.2992719: Permission denied
✅ Retrieved from secrets manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials loaded for coordinator/fastorder_admin_gd: fastorder_admin_gd@db-identity-sau-main-dev-postgresql.fastorder.com:5432/fastorder_identity_sau_main_dev_db
✓ Credentials retrieved: fastorder_admin_gd@db-identity-sau-main-dev-postgresql.fastorder.com:5432/fastorder_identity_sau_main_dev_db
╔════════════════════════════════════════════╗
║  PostgreSQL Test Suite (AWS Secrets MGR)  ║
╚════════════════════════════════════════════╝

=== PostgreSQL Authentication Test ===
✗ PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-identity-sau-main-dev-postgresql.fastorder.com" (10.100.1.214), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/identity/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-03 07:45:16 UTC] USER=www-data EUID=0 PID=2993312 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : coordinator
  PG HOST     : db-identity-sau-main-dev-postgresql.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_identity_sau_main_dev_db
  SCHEMA      : auth
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator
  DNS → 10.100.1.214
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: coordinator

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-coordinator
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User debezium_user does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: debezium_user
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/coordinator/debezium_user
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials coordinator

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: coordinator/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-01-03 07:45:31 UTC] USER=www-data EUID=0 PID=2993757 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
[2026-01-03 07:45:31 UTC] USER=www-data EUID=0 PID=2993783 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-coordinator -p 5432 -d fastorder_identity_sau_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=coordinator IDENTIFIER_PARENT=coordinator
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║                    IAM Database Schema Initialization                       ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] 🟢 Starting IAM schema provisioning...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

[INFO] 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: core
  Core Identity Directory (tenants, realms, identities, devices, MFA)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [1/20]: core/01-tenant
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
🔧 Installing extensions...
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
   Creating reference table: core.tenant
 create_reference_table 
------------------------
 
(1 row)

✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[OK] Table core/01-tenant initialized

[INFO] 🔸 Table [2/20]: core/02-realm
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.realm$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_realm_updated" for relation "core.realm" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[OK] Table core/02-realm initialized

[INFO] 🔸 Table [3/20]: core/03-identity
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_identity_updated" for relation "core.identity" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[OK] Table core/03-identity initialized

[INFO] 🔸 Table [4/20]: core/04-device
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.device$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[OK] Table core/04-device initialized

[INFO] 🔸 Table [5/20]: core/05-identity_account
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity_account$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_identity_account_updated" for relation "core.identity_account" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[OK] Table core/05-identity_account initialized

[INFO] 🔸 Table [6/20]: core/06-identity_mfa
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.identity_mfa$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[OK] Table core/06-identity_mfa initialized

[INFO] 🔸 Table [7/20]: core/07-external_idp_link
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$core.external_idp_link$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[OK] Table core/07-external_idp_link initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: policy
  RBAC/ABAC Authorization (clients, roles, permissions, policies)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [8/20]: policy/01-client
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.client$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
NOTICE:  trigger "tr_client_updated" for relation "policy.client" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[OK] Table policy/01-client initialized

[INFO] 🔸 Table [9/20]: policy/02-resource
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.resource$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[OK] Table policy/02-resource initialized

[INFO] 🔸 Table [10/20]: policy/03-scope
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.scope$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_scope_updated" for relation "policy.scope" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[OK] Table policy/03-scope initialized

[INFO] 🔸 Table [11/20]: policy/04-permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.permission$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_permission_updated" for relation "policy.permission" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[OK] Table policy/04-permission initialized

[INFO] 🔸 Table [12/20]: policy/05-role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.role$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_role_updated" for relation "policy.role" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[OK] Table policy/05-role initialized

[INFO] 🔸 Table [13/20]: policy/06-role_permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.role_permission$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[OK] Table policy/06-role_permission initialized

[INFO] 🔸 Table [14/20]: policy/07-identity_role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.identity_role$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[OK] Table policy/07-identity_role initialized

[INFO] 🔸 Table [15/20]: policy/08-policy_rule
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.policy_rule$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
NOTICE:  trigger "tr_policy_rule_updated" for relation "policy.policy_rule" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[OK] Table policy/08-policy_rule initialized

[INFO] 🔸 Table [16/20]: policy/09-api_key
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  local tables that are added to metadata automatically by citus, but not chained with reference tables via foreign keys might be automatically converted back to postgres tables
HINT:  Executing citus_add_local_table_to_metadata($$policy.api_key$$) prevents this for the given relation, and all of the connected relations
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[OK] Table policy/09-api_key initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: audit
  Audit & Risk Logging (auth events, admin actions, risk decisions)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [17/20]: audit/01-auth_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[OK] Table audit/01-auth_event initialized

[INFO] 🔸 Table [18/20]: audit/02-admin_action
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[OK] Table audit/02-admin_action initialized

[INFO] 🔸 Table [19/20]: audit/03-risk_decision
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[OK] Table audit/03-risk_decision initialized

[INFO] 🔸 Table [20/20]: audit/04-consent_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
CREATE TABLE
DO
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[OK] Table audit/04-consent_event initialized


════════════════════════════════════════════════════════════════════════════
[OK] ✅ IAM Schema Initialization Complete!
[OK] All 20 tables initialized successfully

Schemas created:
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

Design highlights:
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

════════════════════════════════════════════════════════════════════════════

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=coordinator IDENTIFIER_PARENT=coordinator
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: identity/run.sh started for SERVICE=identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity
🔍 DEBUG_CHECKPOINT_A3: SERVICE_ROOT exists, discovering table folders
🔍 DEBUG_CHECKPOINT_A4: Found subfolder: auth
🔍 DEBUG_CHECKPOINT_A4b: Checking for nested schema layout in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth
🔍 DEBUG_CHECKPOINT_A4c: Found nested steps dir: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps (display: auth/login)
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: auth/login|/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
[INFO] 📚 Detected grouped table folders under identity/: auth/login

🔍 DEBUG_CHECKPOINT_A7: Current IDENTIFIER=coordinator
🔍 DEBUG_CHECKPOINT_A8_PROCEED: Processing tables on coordinator/main node
🔍 DEBUG_CHECKPOINT_A9: Processing table: auth/login at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
[INFO] 🔸 Table group: auth/login
🔍 DEBUG_CHECKPOINT_A10: About to run numbered steps for table: auth/login
🔍 DEBUG_CHECKPOINT_B1: run_all_numbered_steps_in_dir called for dir=/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps table=auth/login
🔍 DEBUG_CHECKPOINT_B2: Found 1 numbered steps: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B3: About to run step: 01-init-schema.sh
Ab substep 0 compelete start
[DEBUG] Tracking substep start: steps/01-install/steps/identity/auth/login/01-init-schema (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
Ab substep 0 compelete start
[INFO] 📦 01 init schema...
Ab substep 1 compelete start
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing auth.login_account table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating auth schema...
NOTICE:  schema "auth" already exists, skipping
CREATE SCHEMA
✅ Schema created
🔧 Creating account_status ENUM...
DO
✅ ENUM created
🔧 Creating auth.login_account table...
CREATE TABLE
✅ Table created (Citus-compatible with region_hint in all constraints)
🔧 Creating indexes...
CREATE INDEX
CREATE INDEX
✅ Indexes created
🔧 Creating Citus REFERENCE table for CDC compatibility...
 create_reference_table 
------------------------
 
(1 row)

✅ Table created as REFERENCE table (replicated to all nodes)
   CDC via Debezium will work correctly on coordinator
🎉 Schema initialization complete for fastorder_identity_sau_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

📊 Registering environment in monitoring database (obs schema)...
   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
   Resource IP: 142.93.238.16
⚠️  Could not connect to monitoring database, skipping registration
   You can manually register later using:
   /opt/fastorder/bash/scripts/env_app_setup/setup/04-postgresql/steps/register-authN-af-aaaa1-dev.sh

==========================================
✅ Schema initialization complete!
==========================================
Ab substep 1 compelete end
Ab substep 2 compelete start
Ab substep 2 compelete end

🔍 DEBUG_CHECKPOINT_B4: Completed step: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B5: All numbered steps completed for /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A11: Completed numbered steps for table: auth/login
compeleted here

🔍 DEBUG_CHECKPOINT_A12: All tables processed
End of 04-postgresql/steps/01-install/steps/identity/run.sh

✓ ✅ Coordinator setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up 1 worker(s) (Citus data nodes)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up worker: worker-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-03 07:47:09 UTC] USER=unknown EUID=33 PID=2997893 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-03 07:47:09 UTC] USER=unknown EUID=33 PID=2997900 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-03 07:47:09 UTC] USER=unknown EUID=33 PID=2997907 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-03 07:47:09 UTC] USER=unknown EUID=33 PID=2997914 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-03 07:47:09 UTC] USER=unknown EUID=33 PID=2997921 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-03 07:47:09 UTC] USER=unknown EUID=33 PID=2997928 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895 JOB_UUID=b166d639-0d14-4485-904a-cf625a2ce6d8

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[WARN] Could not find PostgreSQL IP for worker-01 in topology.json, allocating new VM IP...
[INFO] Allocated new VM IP: 10.100.1.215 for db-worker-01-postgresql
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.215
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01...
[INFO]   db-identity-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.215

[INFO]   ➕ Adding db-identity-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.215
✅     ✅ Added: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.215

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.215    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-03 07:47:13 UTC] USER=www-data EUID=0 PID=2998441 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:47:13 UTC] USER=www-data EUID=0 PID=2998450 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-03 07:47:13 UTC] USER=www-data EUID=0 PID=2998460 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-2998408
[2026-01-03 07:47:13 UTC] USER=www-data EUID=0 PID=2998469 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-2998408/ra_root.crt
[2026-01-03 07:47:13 UTC] USER=www-data EUID=0 PID=2998478 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-2998408/ra_root.key
[2026-01-03 07:47:14 UTC] USER=www-data EUID=0 PID=2998487 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-2998408/ra_root.crt
[2026-01-03 07:47:14 UTC] USER=www-data EUID=0 PID=2998496 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-2998408/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[2026-01-03 07:47:15 UTC] USER=www-data EUID=0 PID=2998531 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2998408/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-03 07:47:15 UTC] USER=www-data EUID=0 PID=2998540 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2998408/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-03 07:47:15 UTC] USER=www-data EUID=0 PID=2998549 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-03 07:47:15 UTC] USER=www-data EUID=0 PID=2998558 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-2998408/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:15 UTC] USER=www-data EUID=0 PID=2998567 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:15 UTC] USER=www-data EUID=0 PID=2998576 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:15 UTC] USER=www-data EUID=0 PID=2998585 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-03 07:47:15 UTC] USER=www-data EUID=0 PID=2998596 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-03 07:47:15 UTC] USER=www-data EUID=0 PID=2998605 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-03 07:47:15 UTC] USER=www-data EUID=0 PID=2998614 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-03 07:47:15 UTC] USER=www-data EUID=0 PID=2998623 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-03 07:47:15 UTC] USER=www-data EUID=0 PID=2998632 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:47:15 UTC] USER=www-data EUID=0 PID=2998641 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01
Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:47:16 UTC] USER=www-data EUID=0 PID=2998695 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-01-03 07:47:16 UTC] USER=www-data EUID=0 PID=2998704 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-03 07:47:16 UTC] USER=www-data EUID=0 PID=2998713 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-01-03 07:47:16 UTC] USER=www-data EUID=0 PID=2998722 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-03 07:47:16 UTC] USER=www-data EUID=0 PID=2998731 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:47:16 UTC] USER=www-data EUID=0 PID=2998745 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:47:16 UTC] USER=www-data EUID=0 PID=2998754 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:47:16 UTC] USER=www-data EUID=0 PID=2998763 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-03 07:47:16 UTC] USER=www-data EUID=0 PID=2998772 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998781 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998790 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998801 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998810 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998819 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998828 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998837 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998846 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998855 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998864 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998873 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998882 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998908 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998917 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998926 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998935 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998944 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998953 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998962 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998971 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998983 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2998992 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2999001 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2999011 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2999021 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2999030 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2999039 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2999048 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2999057 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:17 UTC] USER=www-data EUID=0 PID=2999066 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999075 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999084 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999093 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999102 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999111 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999121 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999131 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999140 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999149 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999158 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999167 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999176 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999185 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999194 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999203 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999212 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999221 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999231 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999241 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999250 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999259 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999268 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999278 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999289 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999299 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999308 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999317 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999326 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999335 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-03 07:47:18 UTC] USER=www-data EUID=0 PID=2999346 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:47:19 UTC] USER=www-data EUID=0 PID=2999356 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:47:19 UTC] USER=www-data EUID=0 PID=2999365 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:47:19 UTC] USER=www-data EUID=0 PID=2999374 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-03 07:47:19 UTC] USER=www-data EUID=0 PID=2999383 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-03 07:47:19 UTC] USER=www-data EUID=0 PID=2999392 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-03 07:47:19 UTC] USER=www-data EUID=0 PID=2999401 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:47:19 UTC] USER=www-data EUID=0 PID=2999410 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:47:19 UTC] USER=www-data EUID=0 PID=2999420 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:47:19 UTC] USER=www-data EUID=0 PID=2999430 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:47:19 UTC] USER=www-data EUID=0 PID=2999471 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-01-03 07:47:19 UTC] USER=www-data EUID=0 PID=2999480 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-03 07:47:19 UTC] USER=www-data EUID=0 PID=2999489 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-01-03 07:47:19 UTC] USER=www-data EUID=0 PID=2999498 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-03 07:47:19 UTC] USER=www-data EUID=0 PID=2999507 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:47:20 UTC] USER=www-data EUID=0 PID=2999521 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:47:20 UTC] USER=www-data EUID=0 PID=2999530 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:47:20 UTC] USER=www-data EUID=0 PID=2999539 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-03 07:47:20 UTC] USER=www-data EUID=0 PID=2999548 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-03 07:47:20 UTC] USER=www-data EUID=0 PID=2999557 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:20 UTC] USER=www-data EUID=0 PID=2999566 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:47:20 UTC] USER=www-data EUID=0 PID=2999575 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999584 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999593 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999602 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999611 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999620 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999629 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999638 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999647 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999656 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999665 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999674 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999700 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999709 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999718 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999727 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999736 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999745 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999754 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999763 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999772 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999781 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999790 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999800 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999810 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999819 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999828 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999837 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:21 UTC] USER=www-data EUID=0 PID=2999846 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=2999855 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=2999864 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=2999875 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=2999884 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=2999893 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=2999902 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=2999912 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=2999922 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=2999931 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=2999940 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=2999949 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=2999958 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=2999967 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=2999977 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=2999986 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=2999995 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=3000004 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=3000013 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=3000023 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=3000033 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=3000042 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=3000051 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=3000060 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=3000069 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=3000078 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=3000087 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=3000096 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=3000105 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:47:22 UTC] USER=www-data EUID=0 PID=3000114 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-03 07:47:23 UTC] USER=www-data EUID=0 PID=3000123 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres_der.key
[2026-01-03 07:47:23 UTC] USER=www-data EUID=0 PID=3000133 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:47:23 UTC] USER=www-data EUID=0 PID=3000143 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:47:23 UTC] USER=www-data EUID=0 PID=3000152 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:47:23 UTC] USER=www-data EUID=0 PID=3000161 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-03 07:47:23 UTC] USER=www-data EUID=0 PID=3000170 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-03 07:47:23 UTC] USER=www-data EUID=0 PID=3000179 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-03 07:47:23 UTC] USER=www-data EUID=0 PID=3000188 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:47:23 UTC] USER=www-data EUID=0 PID=3000197 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:47:23 UTC] USER=www-data EUID=0 PID=3000206 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:47:23 UTC] USER=www-data EUID=0 PID=3000215 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-postgresql environment: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com (10.100.1.215)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.215
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01
[INFO] Data dir:   /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[INFO] Port:       5432
[INFO] Hostname:   db-identity-sau-main-dev-postgresql-worker-01
[2026-01-03 07:47:25 UTC] USER=www-data EUID=0 PID=3000311 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:47:25 UTC] USER=www-data EUID=0 PID=3000332 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:47:25 UTC] USER=www-data EUID=0 PID=3000354 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:47:25 UTC] USER=www-data EUID=0 PID=3000375 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01
  Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-03 07:47:26 UTC] USER=www-data EUID=0 PID=3000415 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:47:26 UTC] USER=www-data EUID=0 PID=3000424 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-03 07:47:26 UTC] USER=www-data EUID=0 PID=3000434 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-3000382
[2026-01-03 07:47:26 UTC] USER=www-data EUID=0 PID=3000443 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-3000382/ra_root.crt
[2026-01-03 07:47:26 UTC] USER=www-data EUID=0 PID=3000452 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-3000382/ra_root.key
[2026-01-03 07:47:26 UTC] USER=www-data EUID=0 PID=3000461 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3000382/ra_root.crt
[2026-01-03 07:47:26 UTC] USER=www-data EUID=0 PID=3000470 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3000382/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[2026-01-03 07:47:31 UTC] USER=www-data EUID=0 PID=3000510 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3000382/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-03 07:47:31 UTC] USER=www-data EUID=0 PID=3000519 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3000382/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-03 07:47:32 UTC] USER=www-data EUID=0 PID=3000528 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-03 07:47:32 UTC] USER=www-data EUID=0 PID=3000537 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3000382/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:32 UTC] USER=www-data EUID=0 PID=3000548 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:32 UTC] USER=www-data EUID=0 PID=3000557 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:47:32 UTC] USER=www-data EUID=0 PID=3000566 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-03 07:47:32 UTC] USER=www-data EUID=0 PID=3000577 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-03 07:47:32 UTC] USER=www-data EUID=0 PID=3000586 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-03 07:47:32 UTC] USER=www-data EUID=0 PID=3000595 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-03 07:47:32 UTC] USER=www-data EUID=0 PID=3000604 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-03 07:47:32 UTC] USER=www-data EUID=0 PID=3000613 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:47:32 UTC] USER=www-data EUID=0 PID=3000622 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01
Primary CN:  db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01.service

3. Test SSL connection:
   psql "host=db-identity-sau-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-03 07:47:32 UTC] USER=www-data EUID=0 PID=3000651 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-03 07:47:32 UTC] USER=www-data EUID=0 PID=3000660 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.key
[2026-01-03 07:47:32 UTC] USER=www-data EUID=0 PID=3000669 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-03 07:47:32 UTC] USER=www-data EUID=0 PID=3000690 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-worker-01.service
[2026-01-03 07:47:32 UTC] USER=www-data EUID=0 PID=3000711 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-03 07:47:32 UTC] USER=www-data EUID=0 PID=3000742 ACTION=fsop ARGS=rm -rf /var/run/postgresql-identity-sau-main-dev-worker-01
[OK]   No conflicting Postgres left on port 5432
[OK]   Using postgres password from vault provider
[2026-01-03 07:47:34 UTC] USER=www-data EUID=0 PID=3000796 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.N5QtL0
[2026-01-03 07:47:35 UTC] USER=www-data EUID=0 PID=3000817 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.N5QtL0
[2026-01-03 07:47:35 UTC] USER=www-data EUID=0 PID=3000839 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-03 07:47:35 UTC] USER=www-data EUID=0 PID=3000861 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-03 07:47:35 UTC] USER=www-data EUID=0 PID=3000883 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/identity-sau-main-dev
[INFO] Initializing cluster in /var/lib/postgresql/17/identity-sau-main-dev/worker-01 (SCRAM; pwfile)
[WARN] Removing existing data directory: /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-03 07:47:35 UTC] USER=www-data EUID=0 PID=3000904 ACTION=fsop ARGS=rm -rf /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-03 07:47:35 UTC] USER=www-data EUID=0 PID=3000926 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-03 07:47:35 UTC] USER=www-data EUID=0 PID=3000947 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-03 07:47:35 UTC] USER=www-data EUID=0 PID=3000968 ACTION=fsop ARGS=chmod 700 /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[2026-01-03 07:47:35 UTC] USER=www-data EUID=0 PID=3000989 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-03 07:47:35 UTC] USER=www-data EUID=0 PID=3001010 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-03 07:47:36 UTC] USER=www-data EUID=0 PID=3001031 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-identity-sau-main-dev-worker-01
[2026-01-03 07:47:36 UTC] USER=www-data EUID=0 PID=3001040 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /var/lib/postgresql/17/identity-sau-main-dev/worker-01 --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.N5QtL0
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/17/identity-sau-main-dev/worker-01 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/17/bin/pg_ctl -D /var/lib/postgresql/17/identity-sau-main-dev/worker-01 -l logfile start

[OK]   initdb complete
[2026-01-03 07:47:37 UTC] USER=www-data EUID=0 PID=3001078 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.N5QtL0
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-03 07:47:37 UTC] USER=www-data EUID=0 PID=3001127 ACTION=fsop ARGS=cp /tmp/tmp.CRuQi2xfWP /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-03 07:47:37 UTC] USER=www-data EUID=0 PID=3001148 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-03 07:47:37 UTC] USER=www-data EUID=0 PID=3001169 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01.service
[2026-01-03 07:47:37 UTC] USER=www-data EUID=0 PID=3001194 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.4BPIj0 /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01.service
[2026-01-03 07:47:37 UTC] USER=www-data EUID=0 PID=3001215 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01.service
[OK]   systemd unit written
[2026-01-03 07:47:38 UTC] USER=www-data EUID=0 PID=3001236 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-03 07:47:38 UTC] USER=www-data EUID=0 PID=3001257 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-03 07:47:38 UTC] USER=www-data EUID=0 PID=3001278 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-03 07:47:39 UTC] USER=www-data EUID=0 PID=3001393 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-worker-01.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-03 07:47:40 UTC] USER=www-data EUID=0 PID=3001435 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-worker-01.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Provisioning application database and Debezium role (if not exists)...
[INFO] Checking if database fastorder_identity_sau_main_dev_db exists...
[INFO] DB check result: exit_code=0, output='[2026-01-03 07:47:41 UTC] USER=www-data EUID=0 PID=3001590 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_identity_sau_main_dev_db''
[INFO] Creating database fastorder_identity_sau_main_dev_db...
[2026-01-03 07:47:41 UTC] USER=www-data EUID=0 PID=3001613 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_identity_sau_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[OK]   Database fastorder_identity_sau_main_dev_db created
[INFO] Checking if role debezium_user exists...
[INFO] Role check result: exit_code=0, output='[2026-01-03 07:47:41 UTC] USER=www-data EUID=0 PID=3001637 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[INFO] Creating role debezium_user...
[2026-01-03 07:47:42 UTC] USER=www-data EUID=0 PID=3001666 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'Fg2z6GOjKEAVfBR38x0JiEKw';
CREATE ROLE
[OK]   Role debezium_user created
[2026-01-03 07:47:42 UTC] USER=www-data EUID=0 PID=3001689 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_identity_sau_main_dev_db" TO debezium_user;
GRANT
[OK]   Application DB (fastorder_identity_sau_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[INFO] Applying connection and memory optimizations...
[INFO] Current settings: max_connections=100, work_mem=4MB
[INFO] Target settings (worker): max_connections=100, work_mem=8MB
[2026-01-03 07:47:42 UTC] USER=www-data EUID=0 PID=3001766 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 100;
ALTER SYSTEM
[2026-01-03 07:47:43 UTC] USER=www-data EUID=0 PID=3001789 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-03 07:47:43 UTC] USER=www-data EUID=0 PID=3001812 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   Settings applied to postgresql.auto.conf
[2026-01-03 07:47:43 UTC] USER=www-data EUID=0 PID=3001827 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01/standby.signal
[INFO] Service recently started (3s ago) - restarting to apply max_connections...
[INFO] Stopping service...
[2026-01-03 07:47:43 UTC] USER=www-data EUID=0 PID=3001849 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-worker-01.service
[INFO] Waiting for port 5432 to be released...
[OK]   Port 5432 released
[INFO] Starting service...
[2026-01-03 07:47:46 UTC] USER=www-data EUID=0 PID=3001883 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-worker-01.service
[2026-01-03 07:47:52 UTC] USER=www-data EUID=0 PID=3001936 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-worker-01.service
[OK]   ✅ Optimization complete: max_connections=100, work_mem=8MB
[OK]   Synchronous replication already configured (synchronous_commit: on)
[INFO] Setting postgres password via centralized script... for worker-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Initial setup: Using password from initdb
✓ PostgreSQL password already set during initdb
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/worker-01
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
[OK]   Password set and persisted
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.215
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01...
[INFO]   db-identity-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.215

[INFO]   ✅ db-identity-sau-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.215    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com


[OK]   PostgreSQL 'identity-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.key \
        host=db-identity-sau-main-dev-postgresql-worker-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-worker-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.215
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-worker-01
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 2a8d7237-0c1b-4286-8ffc-cd46f4f7052e
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 📦 03 role...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-03 07:48:05 UTC] USER=www-data EUID=0 PID=3002534 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01/standby.signal
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    debezium_user
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   debezium_user
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:48:28 UTC] USER=www-data EUID=0 PID=3002725 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-debezium_user
[2026-01-03 07:48:28 UTC] USER=www-data EUID=0 PID=3002734 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-01-03 07:48:28 UTC] USER=www-data EUID=0 PID=3002743 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-debezium_user/ra_root.key
[2026-01-03 07:48:28 UTC] USER=www-data EUID=0 PID=3002752 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-01-03 07:48:28 UTC] USER=www-data EUID=0 PID=3002761 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002775 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002784 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002793 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002803 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002812 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002821 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002830 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002839 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002848 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002857 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002866 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002875 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002884 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002893 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002902 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002911 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002937 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002946 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002955 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002964 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002973 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:29 UTC] USER=www-data EUID=0 PID=3002982 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3002991 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003000 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003009 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003018 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003027 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003037 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003047 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003056 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003065 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003074 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003083 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003092 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003101 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003110 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003119 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003128 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003137 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003147 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003157 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003166 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003175 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003184 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003193 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003202 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003211 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003220 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003229 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003238 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003247 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-03 07:48:30 UTC] USER=www-data EUID=0 PID=3003257 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003267 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003276 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003285 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003294 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003303 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003312 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003321 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003330 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003339 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003348 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003357 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/debezium_user_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003367 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003377 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003386 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003395 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003404 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003413 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003422 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003431 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003440 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:48:31 UTC] USER=www-data EUID=0 PID=3003449 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: debezium_user
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/debezium_user.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U debezium_user -d postgres

🔐 Generating replicator client certificate for worker-01...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:48:32 UTC] USER=www-data EUID=0 PID=3003492 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-03 07:48:32 UTC] USER=www-data EUID=0 PID=3003501 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-03 07:48:32 UTC] USER=www-data EUID=0 PID=3003510 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-03 07:48:32 UTC] USER=www-data EUID=0 PID=3003519 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-03 07:48:32 UTC] USER=www-data EUID=0 PID=3003528 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003542 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003551 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003560 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003569 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003578 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003587 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003596 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003605 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003614 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003623 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003632 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003641 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003650 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003659 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003668 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003677 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003686 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003695 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003721 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003730 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003739 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003748 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003757 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003766 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003775 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003784 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003793 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003802 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:48:33 UTC] USER=www-data EUID=0 PID=3003811 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003821 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003831 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003840 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003849 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003858 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003867 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003876 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003885 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003894 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003903 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003912 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003921 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003931 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003941 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003950 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003959 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003968 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003977 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003986 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3003995 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3004005 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3004014 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3004023 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3004032 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3004042 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3004052 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3004061 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3004070 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3004079 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3004088 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:34 UTC] USER=www-data EUID=0 PID=3004097 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:48:35 UTC] USER=www-data EUID=0 PID=3004106 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-03 07:48:35 UTC] USER=www-data EUID=0 PID=3004115 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:35 UTC] USER=www-data EUID=0 PID=3004124 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:48:35 UTC] USER=www-data EUID=0 PID=3004133 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:48:35 UTC] USER=www-data EUID=0 PID=3004142 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:48:35 UTC] USER=www-data EUID=0 PID=3004152 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:48:35 UTC] USER=www-data EUID=0 PID=3004162 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:48:35 UTC] USER=www-data EUID=0 PID=3004171 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:48:35 UTC] USER=www-data EUID=0 PID=3004180 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-03 07:48:35 UTC] USER=www-data EUID=0 PID=3004189 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-03 07:48:35 UTC] USER=www-data EUID=0 PID=3004198 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-03 07:48:35 UTC] USER=www-data EUID=0 PID=3004207 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:48:35 UTC] USER=www-data EUID=0 PID=3004216 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:48:35 UTC] USER=www-data EUID=0 PID=3004225 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:48:35 UTC] USER=www-data EUID=0 PID=3004234 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

✅ Replicator certificate generated for worker-01
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: fastorder_admin_gd
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: worker-01/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    fastorder_admin_gd
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   fastorder_admin_gd
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:48:49 UTC] USER=www-data EUID=0 PID=3004588 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-fastorder_admin_gd
[2026-01-03 07:48:49 UTC] USER=www-data EUID=0 PID=3004597 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-01-03 07:48:49 UTC] USER=www-data EUID=0 PID=3004606 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
[2026-01-03 07:48:49 UTC] USER=www-data EUID=0 PID=3004616 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-01-03 07:48:49 UTC] USER=www-data EUID=0 PID=3004625 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004639 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004648 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004657 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004667 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004678 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004688 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004697 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004706 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004715 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004724 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004733 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004742 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004751 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004760 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004769 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004778 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004787 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004796 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004824 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004833 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004842 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004851 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:50 UTC] USER=www-data EUID=0 PID=3004860 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3004869 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3004878 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3004887 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3004896 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3004905 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3004914 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3004924 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3004934 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3004943 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3004952 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3004961 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3004970 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3004979 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3004988 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3004997 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3005006 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3005015 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3005024 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3005034 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3005044 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3005053 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3005062 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3005071 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3005080 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3005089 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3005098 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3005107 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3005116 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3005125 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-03 07:48:51 UTC] USER=www-data EUID=0 PID=3005134 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005144 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005154 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005163 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005174 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005183 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005193 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005202 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005211 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005220 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005229 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005238 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005247 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005257 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005267 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005276 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005285 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005294 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005303 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005312 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005321 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005330 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:48:52 UTC] USER=www-data EUID=0 PID=3005339 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: fastorder_admin_gd
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/fastorder_admin_gd.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U fastorder_admin_gd -d postgres

🧱 Connecting via Unix socket to create role and database...
   Socket: /var/run/postgresql-identity-sau-main-dev-worker-01:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️  Database fastorder_identity_sau_main_dev_db already exists, skipping creation
[2026-01-03 07:48:53 UTC] USER=www-data EUID=0 PID=3005397 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️  Using pg_hba.conf: /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-03 07:48:53 UTC] USER=www-data EUID=0 PID=3005431 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

=== Pre-flight Checks ===
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
✓ AWS Secrets Manager accessible

=== Retrieving Credentials from AWS ===
ℹ️  Retrieving PostgreSQL credentials for: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️  Fetching secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
/opt/fastorder/bash/infra_core/cache.sh: line 145: /var/cache/secrets/fastorder_db_identity_sau_main_dev_postgresql_worker-01_fastorder_admin_gd.cache.tmp.3005441: Permission denied
✅ Retrieved from secrets manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials loaded for worker-01/fastorder_admin_gd: fastorder_admin_gd@db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_identity_sau_main_dev_db
✓ Credentials retrieved: fastorder_admin_gd@db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_identity_sau_main_dev_db
╔════════════════════════════════════════════╗
║  PostgreSQL Test Suite (AWS Secrets MGR)  ║
╚════════════════════════════════════════════╝

=== PostgreSQL Authentication Test ===
✗ PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-identity-sau-main-dev-postgresql-worker-01.fastorder.com" (10.100.1.215), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/identity/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-03 07:49:01 UTC] USER=www-data EUID=0 PID=3005670 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01/standby.signal
── fast setup ─────────────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : debezium_user
  DB          : fastorder_identity_sau_main_dev_db
  SCHEMA      : auth
  AUTH MODE   : scram (scram=password over TLS | cert=mTLS)
  SUBNET ALLOW: 10.201.0.0/16
  CONNECT /32 : 142.93.238.16
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
  DNS → 10.100.1.215
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
⚠️  ~/.aws/credentials file not found
⚠️  Using environment-based AWS authentication

╔════════════════════════════════════════════════════════════╗
║   PostgreSQL Password Rotation via AWS Secrets Manager    ║
╚════════════════════════════════════════════════════════════╝

Environment Configuration:
  Service:    identity
  Zone:       sau
  Environment: dev
  Identifier: worker-01

AWS Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user

Connection Info:
  Socket Dir: /var/run/postgresql-identity-sau-main-dev-worker-01
  Port:       5432

Testing AWS Secrets Manager connectivity...
ℹ️  Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
    "UserId": "AIDAWYLM4MSHFSCGU7QUM",
    "Account": "464621692046",
    "Arn": "arn:aws:iam::464621692046:user/fo-dev"
}

Method 1 (PREFERRED): AWS Secrets Manager Rotation
────────────────────────────────────────────────────────────

This method uses AWS Secrets Manager's built-in rotation:
  ✓ Zero-downtime (dual-password window)
  ✓ Automatic rollback on failure
  ✓ CloudTrail audit log
  ✓ CloudWatch metrics
  ✓ No secret exposure in scripts

Non-interactive mode: Proceeding with password rotation automatically

Generating new secure password...
User debezium_user does not exist yet - skipping ALTER, will be created by calling script
✓ Password generated for new user: debezium_user
Storing password in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting PostgreSQL credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
ℹ️  Setting secret in AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
✅ Secret updated: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
✓ Password stored in AWS Secrets Manager

Verifying new credentials...
✓ New credentials retrieved from AWS Secrets Manager

Testing PostgreSQL connection with new credentials...
✓ PostgreSQL connection successful (socket authentication)

✓ ╔════════════════════════════════════════════════════════════╗
✓ ║              Password Rotation Complete!                   ║
✓ ╚════════════════════════════════════════════════════════════╝

Secret: fastorder/db/identity/sau/main/dev/postgresql/worker-01/debezium_user
Method: Direct Update (stored in AWS Secrets Manager)
Status: Completed

To retrieve credentials:
  # Using Bash library
  source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  get_pg_credentials worker-01

Audit trail: AWS CloudTrail (for Secrets Manager operations)

✓ Done!
🔍 Retrieving password from vault with identifier: worker-01/debezium_user
✓ Retrieved password from secrets vault
  password   : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️  Role debezium_user exists, updating
[2026-01-03 07:49:14 UTC] USER=www-data EUID=0 PID=3006192 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
[2026-01-03 07:49:15 UTC] USER=www-data EUID=0 PID=3006218 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d fastorder_identity_sau_main_dev_db --no-psqlrc
CREATE SCHEMA
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
✅ Role/DB/grants ensured.
⚠️  Could not find pg_hba.conf (skipping HBA edits): /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
🔐 Creating replicator role for worker-01...
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
  DNS → 10.100.1.215
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Creating role: replicator with password
SET
CREATE ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/identity/sau/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🎉 Done.
✅ Replicator role created for worker-01

[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=worker-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║                    IAM Database Schema Initialization                       ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] 🟢 Starting IAM schema provisioning...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

[INFO] 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: core
  Core Identity Directory (tenants, realms, identities, devices, MFA)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [1/20]: core/01-tenant
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "pgcrypto" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "citext" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
NOTICE:  schema "utils" already exists, skipping
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
NOTICE:  schema "core" already exists, skipping
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
NOTICE:  relation "tenant" already exists, skipping
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[OK] Table core/01-tenant initialized

[INFO] 🔸 Table [2/20]: core/02-realm
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  relation "realm" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_realm_keycloak_id" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_realm_tenant" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[OK] Table core/02-realm initialized

[INFO] 🔸 Table [3/20]: core/03-identity
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  relation "identity" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_unique_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_unique_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_type" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[OK] Table core/03-identity initialized

[INFO] 🔸 Table [4/20]: core/04-device
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  relation "device" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_device_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_fingerprint" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_trusted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_last_seen" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[OK] Table core/04-device initialized

[INFO] 🔸 Table [5/20]: core/05-identity_account
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  relation "identity_account" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_account_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_lockout" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_last_login" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[OK] Table core/05-identity_account initialized

[INFO] 🔸 Table [6/20]: core/06-identity_mfa
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  relation "identity_mfa" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_mfa_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_active" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[OK] Table core/06-identity_mfa initialized

[INFO] 🔸 Table [7/20]: core/07-external_idp_link
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  relation "external_idp_link" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_external_idp_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_provider" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_email" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[OK] Table core/07-external_idp_link initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: policy
  RBAC/ABAC Authorization (clients, roles, permissions, policies)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [8/20]: policy/01-client
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
NOTICE:  schema "policy" already exists, skipping
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  relation "client" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_client_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_key" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_status" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[OK] Table policy/01-client initialized

[INFO] 🔸 Table [9/20]: policy/02-resource
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  relation "resource" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_resource_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_external" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_owner" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[OK] Table policy/02-resource initialized

[INFO] 🔸 Table [10/20]: policy/03-scope
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  relation "scope" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_scope_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_scope_name" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[OK] Table policy/03-scope initialized

[INFO] 🔸 Table [11/20]: policy/04-permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  relation "permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_permission_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_resource" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[OK] Table policy/04-permission initialized

[INFO] 🔸 Table [12/20]: policy/05-role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  relation "role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_keycloak" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[OK] Table policy/05-role initialized

[INFO] 🔸 Table [13/20]: policy/06-role_permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  relation "role_permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_permission_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_permission_perm" already exists, skipping
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[OK] Table policy/06-role_permission initialized

[INFO] 🔸 Table [14/20]: policy/07-identity_role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  relation "identity_role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_role_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_active" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[OK] Table policy/07-identity_role initialized

[INFO] 🔸 Table [15/20]: policy/08-policy_rule
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  relation "policy_rule" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_policy_rule_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_enabled" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_priority" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[OK] Table policy/08-policy_rule initialized

[INFO] 🔸 Table [16/20]: policy/09-api_key
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  relation "api_key" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_api_key_prefix" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[OK] Table policy/09-api_key initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: audit
  Audit & Risk Logging (auth events, admin actions, risk decisions)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [17/20]: audit/01-auth_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
NOTICE:  schema "audit" already exists, skipping
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
NOTICE:  relation "auth_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_auth_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_result" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_ip" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_session" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_trace" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_risk" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[OK] Table audit/01-auth_event initialized

[INFO] 🔸 Table [18/20]: audit/02-admin_action
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
NOTICE:  relation "admin_action" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_admin_action_actor" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_target" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_trace" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[OK] Table audit/02-admin_action initialized

[INFO] 🔸 Table [19/20]: audit/03-risk_decision
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
NOTICE:  relation "risk_decision" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_risk_decision_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_level" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_decision" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_auth" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[OK] Table audit/03-risk_decision initialized

[INFO] 🔸 Table [20/20]: audit/04-consent_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
NOTICE:  relation "consent_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_consent_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_version" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_granted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  relation "audit.auth_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  relation "audit.auth_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  relation "audit.admin_action_2026_03" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  relation "audit.admin_action_2026_04" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  relation "audit.risk_decision_2026_03" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  relation "audit.risk_decision_2026_04" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  relation "audit.consent_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  relation "audit.consent_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[OK] Table audit/04-consent_event initialized


════════════════════════════════════════════════════════════════════════════
[OK] ✅ IAM Schema Initialization Complete!
[OK] All 20 tables initialized successfully

Schemas created:
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

Design highlights:
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

════════════════════════════════════════════════════════════════════════════

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=worker-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: identity/run.sh started for SERVICE=identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity
🔍 DEBUG_CHECKPOINT_A3: SERVICE_ROOT exists, discovering table folders
🔍 DEBUG_CHECKPOINT_A4: Found subfolder: auth
🔍 DEBUG_CHECKPOINT_A4b: Checking for nested schema layout in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth
🔍 DEBUG_CHECKPOINT_A4c: Found nested steps dir: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps (display: auth/login)
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: auth/login|/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
[INFO] 📚 Detected grouped table folders under identity/: auth/login

🔍 DEBUG_CHECKPOINT_A7: Current IDENTIFIER=coordinator
🔍 DEBUG_CHECKPOINT_A8_PROCEED: Processing tables on coordinator/main node
🔍 DEBUG_CHECKPOINT_A9: Processing table: auth/login at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
[INFO] 🔸 Table group: auth/login
🔍 DEBUG_CHECKPOINT_A10: About to run numbered steps for table: auth/login
🔍 DEBUG_CHECKPOINT_B1: run_all_numbered_steps_in_dir called for dir=/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps table=auth/login
🔍 DEBUG_CHECKPOINT_B2: Found 1 numbered steps: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B3: About to run step: 01-init-schema.sh
Ab substep 0 compelete start
[DEBUG] Tracking substep start: steps/01-install/steps/identity/auth/login/01-init-schema (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
Ab substep 0 compelete start
[INFO] 📦 01 init schema...
Ab substep 1 compelete start
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing auth.login_account table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating auth schema...
NOTICE:  schema "auth" already exists, skipping
CREATE SCHEMA
✅ Schema created
🔧 Creating account_status ENUM...
DO
✅ ENUM created
🔧 Creating auth.login_account table...
NOTICE:  relation "login_account" already exists, skipping
CREATE TABLE
✅ Table created (Citus-compatible with region_hint in all constraints)
🔧 Creating indexes...
NOTICE:  relation "idx_login_account_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_login_account_username" already exists, skipping
CREATE INDEX
✅ Indexes created
ℹ️  Table already registered with Citus
🎉 Schema initialization complete for fastorder_identity_sau_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

📊 Registering environment in monitoring database (obs schema)...
   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
   Resource IP: 142.93.238.16
⚠️  Could not connect to monitoring database, skipping registration
   You can manually register later using:
   /opt/fastorder/bash/scripts/env_app_setup/setup/04-postgresql/steps/register-authN-af-aaaa1-dev.sh

==========================================
✅ Schema initialization complete!
==========================================
Ab substep 1 compelete end
Ab substep 2 compelete start
Ab substep 2 compelete end

🔍 DEBUG_CHECKPOINT_B4: Completed step: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B5: All numbered steps completed for /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A11: Completed numbered steps for table: auth/login
compeleted here

🔍 DEBUG_CHECKPOINT_A12: All tables processed
End of 04-postgresql/steps/01-install/steps/identity/run.sh

✓ ✅ Worker worker-01 setup completed

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up standby replicas (1 per worker)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up standby: worker-01-standby-01 (replica of worker-01)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-03 07:50:56 UTC] USER=unknown EUID=33 PID=3010114 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-03 07:50:56 UTC] USER=unknown EUID=33 PID=3010121 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-03 07:50:57 UTC] USER=unknown EUID=33 PID=3010128 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-03 07:50:57 UTC] USER=unknown EUID=33 PID=3010135 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-03 07:50:57 UTC] USER=unknown EUID=33 PID=3010142 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-03 07:50:57 UTC] USER=unknown EUID=33 PID=3010149 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895 JOB_UUID=b166d639-0d14-4485-904a-cf625a2ce6d8

[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[WARN] Could not find PostgreSQL IP for worker-01-standby-01 in topology.json, allocating new VM IP...
[INFO] Allocated new VM IP: 10.100.1.216 for db-worker-01-standby-01-postgresql
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] PostgreSQL IP: 10.100.1.216
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01-standby-01...
[INFO]   db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.216

[INFO]   ➕ Adding db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.216
✅     ✅ Added: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.216

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.216    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com


[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  identity-sau-main-dev.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-03 07:51:02 UTC] USER=www-data EUID=0 PID=3010947 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:02 UTC] USER=www-data EUID=0 PID=3010956 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-03 07:51:02 UTC] USER=www-data EUID=0 PID=3010966 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-3010782
[2026-01-03 07:51:02 UTC] USER=www-data EUID=0 PID=3010975 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-3010782/ra_root.crt
[2026-01-03 07:51:02 UTC] USER=www-data EUID=0 PID=3010984 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-3010782/ra_root.key
[2026-01-03 07:51:02 UTC] USER=www-data EUID=0 PID=3010993 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3010782/ra_root.crt
[2026-01-03 07:51:02 UTC] USER=www-data EUID=0 PID=3011002 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3010782/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
[2026-01-03 07:51:04 UTC] USER=www-data EUID=0 PID=3011054 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3010782/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-03 07:51:04 UTC] USER=www-data EUID=0 PID=3011072 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-03 07:51:04 UTC] USER=www-data EUID=0 PID=3011081 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3010782/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:04 UTC] USER=www-data EUID=0 PID=3011090 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:04 UTC] USER=www-data EUID=0 PID=3011099 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:04 UTC] USER=www-data EUID=0 PID=3011109 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-03 07:51:04 UTC] USER=www-data EUID=0 PID=3011120 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-03 07:51:04 UTC] USER=www-data EUID=0 PID=3011131 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-03 07:51:04 UTC] USER=www-data EUID=0 PID=3011141 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-03 07:51:04 UTC] USER=www-data EUID=0 PID=3011153 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-03 07:51:04 UTC] USER=www-data EUID=0 PID=3011163 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:04 UTC] USER=www-data EUID=0 PID=3011173 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:identity-sau-main-dev.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01-standby-01
Primary CN:  identity-sau-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=identity-sau-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:51:05 UTC] USER=www-data EUID=0 PID=3011236 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-01-03 07:51:05 UTC] USER=www-data EUID=0 PID=3011245 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-03 07:51:05 UTC] USER=www-data EUID=0 PID=3011254 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
[2026-01-03 07:51:05 UTC] USER=www-data EUID=0 PID=3011263 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-03 07:51:05 UTC] USER=www-data EUID=0 PID=3011272 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:06 UTC] USER=www-data EUID=0 PID=3011299 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:06 UTC] USER=www-data EUID=0 PID=3011311 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:06 UTC] USER=www-data EUID=0 PID=3011321 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-03 07:51:06 UTC] USER=www-data EUID=0 PID=3011335 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-03 07:51:06 UTC] USER=www-data EUID=0 PID=3011344 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:06 UTC] USER=www-data EUID=0 PID=3011355 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-03 07:51:06 UTC] USER=www-data EUID=0 PID=3011368 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-03 07:51:06 UTC] USER=www-data EUID=0 PID=3011377 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-03 07:51:06 UTC] USER=www-data EUID=0 PID=3011397 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:06 UTC] USER=www-data EUID=0 PID=3011406 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:06 UTC] USER=www-data EUID=0 PID=3011415 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011424 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011433 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011442 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011451 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011481 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011490 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011499 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011510 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011519 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011528 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011537 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011546 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011555 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011564 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011573 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011583 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011594 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011603 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011612 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011621 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011630 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011639 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011648 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011657 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011666 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011675 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-03 07:51:07 UTC] USER=www-data EUID=0 PID=3011684 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011694 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011704 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011713 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011722 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011731 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011740 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011754 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011763 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011772 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011782 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011791 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011800 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011810 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011820 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011829 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011838 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011850 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011860 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011880 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011892 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011901 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-03 07:51:08 UTC] USER=www-data EUID=0 PID=3011910 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:51:09 UTC] USER=www-data EUID=0 PID=3011939 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:51:09 UTC] USER=www-data EUID=0 PID=3011948 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:51:09 UTC] USER=www-data EUID=0 PID=3011957 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-03 07:51:09 UTC] USER=www-data EUID=0 PID=3011966 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-03 07:51:09 UTC] USER=www-data EUID=0 PID=3011975 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-03 07:51:09 UTC] USER=www-data EUID=0 PID=3011987 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:51:09 UTC] USER=www-data EUID=0 PID=3011997 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:51:09 UTC] USER=www-data EUID=0 PID=3012008 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    postgres
Identifier:  worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  User (CN):   postgres
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:51:09 UTC] USER=www-data EUID=0 PID=3012069 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-01-03 07:51:09 UTC] USER=www-data EUID=0 PID=3012078 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-03 07:51:09 UTC] USER=www-data EUID=0 PID=3012087 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
[2026-01-03 07:51:10 UTC] USER=www-data EUID=0 PID=3012096 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt
[2026-01-03 07:51:10 UTC] USER=www-data EUID=0 PID=3012106 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:10 UTC] USER=www-data EUID=0 PID=3012122 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:10 UTC] USER=www-data EUID=0 PID=3012131 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:10 UTC] USER=www-data EUID=0 PID=3012140 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-03 07:51:10 UTC] USER=www-data EUID=0 PID=3012149 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-03 07:51:10 UTC] USER=www-data EUID=0 PID=3012158 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:10 UTC] USER=www-data EUID=0 PID=3012167 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-03 07:51:10 UTC] USER=www-data EUID=0 PID=3012176 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-03 07:51:10 UTC] USER=www-data EUID=0 PID=3012185 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-03 07:51:10 UTC] USER=www-data EUID=0 PID=3012194 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-03 07:51:10 UTC] USER=www-data EUID=0 PID=3012203 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-03 07:51:10 UTC] USER=www-data EUID=0 PID=3012212 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-03 07:51:10 UTC] USER=www-data EUID=0 PID=3012221 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012230 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012239 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012248 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012257 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012272 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012281 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012307 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012316 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012325 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012334 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012343 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012352 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012361 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012373 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012385 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012394 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012403 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012413 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012423 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012434 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012443 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012452 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:11 UTC] USER=www-data EUID=0 PID=3012461 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012470 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012479 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012488 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012497 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012506 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012516 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012526 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012538 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012547 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012558 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012567 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012576 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012585 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012594 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012603 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012612 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012621 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012630 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012640 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012651 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012660 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012669 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012678 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012687 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012696 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012705 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012714 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012723 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012732 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012741 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/postgres_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-03 07:51:12 UTC] USER=www-data EUID=0 PID=3012751 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:51:13 UTC] USER=www-data EUID=0 PID=3012761 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:51:13 UTC] USER=www-data EUID=0 PID=3012770 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:51:13 UTC] USER=www-data EUID=0 PID=3012779 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-03 07:51:13 UTC] USER=www-data EUID=0 PID=3012788 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-03 07:51:13 UTC] USER=www-data EUID=0 PID=3012797 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-03 07:51:13 UTC] USER=www-data EUID=0 PID=3012810 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:51:13 UTC] USER=www-data EUID=0 PID=3012820 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:51:13 UTC] USER=www-data EUID=0 PID=3012829 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:51:13 UTC] USER=www-data EUID=0 PID=3012838 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:51:13 UTC] USER=www-data EUID=0 PID=3012888 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-03 07:51:13 UTC] USER=www-data EUID=0 PID=3012897 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-03 07:51:14 UTC] USER=www-data EUID=0 PID=3012906 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-03 07:51:14 UTC] USER=www-data EUID=0 PID=3012915 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-03 07:51:14 UTC] USER=www-data EUID=0 PID=3012924 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:51:14 UTC] USER=www-data EUID=0 PID=3012940 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:51:14 UTC] USER=www-data EUID=0 PID=3012949 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:51:14 UTC] USER=www-data EUID=0 PID=3012958 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:51:14 UTC] USER=www-data EUID=0 PID=3012967 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-03 07:51:14 UTC] USER=www-data EUID=0 PID=3012976 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:51:14 UTC] USER=www-data EUID=0 PID=3012985 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:51:14 UTC] USER=www-data EUID=0 PID=3012994 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:51:14 UTC] USER=www-data EUID=0 PID=3013003 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:51:14 UTC] USER=www-data EUID=0 PID=3013012 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:51:14 UTC] USER=www-data EUID=0 PID=3013021 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:51:14 UTC] USER=www-data EUID=0 PID=3013030 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:51:14 UTC] USER=www-data EUID=0 PID=3013039 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:51:14 UTC] USER=www-data EUID=0 PID=3013048 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:51:14 UTC] USER=www-data EUID=0 PID=3013057 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013066 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013075 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013084 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013093 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013125 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013134 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013143 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013152 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013161 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013170 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013179 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013188 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013199 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013221 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013231 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013241 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013253 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:15 UTC] USER=www-data EUID=0 PID=3013262 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013280 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013289 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013298 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013307 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013316 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013325 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013334 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013344 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013355 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013366 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013375 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013384 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013393 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013402 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013411 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013420 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013429 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013438 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013447 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013457 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013467 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013476 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013485 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013494 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013503 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013512 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013521 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013530 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013539 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013548 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:51:16 UTC] USER=www-data EUID=0 PID=3013557 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:51:17 UTC] USER=www-data EUID=0 PID=3013567 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:51:17 UTC] USER=www-data EUID=0 PID=3013577 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:51:17 UTC] USER=www-data EUID=0 PID=3013586 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:51:17 UTC] USER=www-data EUID=0 PID=3013595 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-03 07:51:17 UTC] USER=www-data EUID=0 PID=3013610 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-03 07:51:17 UTC] USER=www-data EUID=0 PID=3013621 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-03 07:51:17 UTC] USER=www-data EUID=0 PID=3013630 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:51:17 UTC] USER=www-data EUID=0 PID=3013648 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:51:17 UTC] USER=www-data EUID=0 PID=3013657 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres


[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 📦 02 setup pg instance...
[DEADLOCK-PREVENTION] Deadlock prevention library loaded
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Using existing db-worker-01-standby-01-postgresql environment: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com (10.100.1.216)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.216
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] Data dir:   /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[INFO] Port:       5432
[INFO] Hostname:   db-identity-sau-main-dev-postgresql-worker-01-standby-01
[2026-01-03 07:51:19 UTC] USER=www-data EUID=0 PID=3013764 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:19 UTC] USER=www-data EUID=0 PID=3013789 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:19 UTC] USER=www-data EUID=0 PID=3013810 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:19 UTC] USER=www-data EUID=0 PID=3013832 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[WARN] Server certificate not found at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[INFO] Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau (Saudi Arabia)
  Branch:      main
  Env:         dev
  Node:        worker-01-standby-01
  Primary CN:  identity-sau-main-dev.fastorder.com
  Alt CN:      identity-sau-main-dev.fastorder.com
  VM IP:       142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-03 07:51:20 UTC] USER=www-data EUID=0 PID=3013902 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-3013839
[2026-01-03 07:51:20 UTC] USER=www-data EUID=0 PID=3013911 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-3013839/ra_root.crt
[2026-01-03 07:51:20 UTC] USER=www-data EUID=0 PID=3013920 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-3013839/ra_root.key
[2026-01-03 07:51:20 UTC] USER=www-data EUID=0 PID=3013929 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3013839/ra_root.crt
[2026-01-03 07:51:20 UTC] USER=www-data EUID=0 PID=3013938 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-3013839/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
[2026-01-03 07:51:23 UTC] USER=www-data EUID=0 PID=3014013 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3013839/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-03 07:51:23 UTC] USER=www-data EUID=0 PID=3014022 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3013839/server.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-03 07:51:23 UTC] USER=www-data EUID=0 PID=3014031 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-03 07:51:23 UTC] USER=www-data EUID=0 PID=3014040 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-3013839/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:24 UTC] USER=www-data EUID=0 PID=3014049 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:24 UTC] USER=www-data EUID=0 PID=3014063 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-03 07:51:24 UTC] USER=www-data EUID=0 PID=3014072 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
  Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-03 07:51:24 UTC] USER=www-data EUID=0 PID=3014084 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-03 07:51:24 UTC] USER=www-data EUID=0 PID=3014093 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-03 07:51:24 UTC] USER=www-data EUID=0 PID=3014102 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-03 07:51:24 UTC] USER=www-data EUID=0 PID=3014111 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-03 07:51:24 UTC] USER=www-data EUID=0 PID=3014120 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:24 UTC] USER=www-data EUID=0 PID=3014129 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01
🔍 Verifying certificate...

Certificate details:
        Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = identity-sau-main-dev.fastorder.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
--
            X509v3 Subject Alternative Name: 
                DNS:identity-sau-main-dev.fastorder.com, DNS:identity-sau-main-dev.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
            X509v3 Subject Key Identifier: 
⚠️  Certificate chain verification: FAILED (but certificate may still work)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment: identity-sau-main-dev
Node:        worker-01-standby-01
Primary CN:  identity-sau-main-dev.fastorder.com

Certificate files installed:
  📜 Server cert: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
  🔑 Server key:  /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
  🏛️  CA cert:     /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)

To use these certificates in PostgreSQL:
1. Update postgresql.conf:
   ssl = on
   ssl_cert_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt'
   ssl_key_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key'
   ssl_ca_file = '/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/root.crt'

2. Restart PostgreSQL:
   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-worker-01-standby-01.service

3. Test SSL connection:
   psql "host=identity-sau-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-03 07:51:24 UTC] USER=www-data EUID=0 PID=3014163 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-03 07:51:24 UTC] USER=www-data EUID=0 PID=3014173 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.key
[2026-01-03 07:51:24 UTC] USER=www-data EUID=0 PID=3014184 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[OK]   mTLS certificates OK (server cert + client certs verified) and keys secured
[INFO] Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-03 07:51:24 UTC] USER=www-data EUID=0 PID=3014205 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-worker-01-standby-01.service
[2026-01-03 07:51:24 UTC] USER=www-data EUID=0 PID=3014226 ACTION=passthru ARGS=systemctl stop postgresql
[WARN] Cleaning stale socket directory /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[2026-01-03 07:51:25 UTC] USER=www-data EUID=0 PID=3014257 ACTION=fsop ARGS=rm -rf /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[OK]   No conflicting Postgres left on port 5432
[OK]   Generated new postgres password for initdb
[2026-01-03 07:51:49 UTC] USER=www-data EUID=0 PID=3014610 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.ydEeGZ
[2026-01-03 07:51:49 UTC] USER=www-data EUID=0 PID=3014637 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.ydEeGZ
[2026-01-03 07:51:50 UTC] USER=www-data EUID=0 PID=3014659 ACTION=fsop ARGS=mkdir -p /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-03 07:51:50 UTC] USER=www-data EUID=0 PID=3014682 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev
[2026-01-03 07:51:50 UTC] USER=www-data EUID=0 PID=3014704 ACTION=fsop ARGS=chmod 755 /var/lib/postgresql/17/identity-sau-main-dev
[INFO] This is a standby. Using pg_basebackup from primary (worker-01)...
[INFO] Setting up replicator role and slot on primary (worker-01)...
ℹ️  Scanning primary for stuck queries from previous failed attempts...
ℹ️  Scanning for stuck queries (timeout: 30s)...
ℹ️  No stuck queries found
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
  NAME        : identity-sau-main-dev
  IDENTIFIER  : worker-01
  PG HOST     : db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
  ROLE        : replicator
  SLOT        : worker_01_standby_01
  SSL DIR     : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
  DNS → 10.100.1.215
  CA         : /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️  Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE:  Role replicator already exists, updating password and ensuring REPLICATION privilege
SET
ALTER ROLE
✅ Replicator role ensured with password authentication.
ℹ️  Password stored in: AWS Secrets Manager
   Secret name: fastorder/db/identity/sau/main/dev/postgresql/replicator

🔄 MIGRATION PATH: Password → Certificate Authentication
   Current:  SCRAM-SHA-256 password auth (production-ready)
   Future:   Certificate-based auth (requires CA automation)
   To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
               and configure standby to use SSL certificates instead of password
🔧 Ensuring replication slot: worker_01_standby_01…
🆕 Creating replication slot worker_01_standby_01
SET
 pg_create_physical_replication_slot 
-------------------------------------
 (worker_01_standby_01,)
(1 row)

✅ Replication slot worker_01_standby_01 created.
🎉 Done.
[OK]   Replicator role and slot created on primary
[INFO] Creating replicator client certificates for connecting to primary (worker-01)...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    replicator
Identifier:  worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        worker-01
  User (CN):   replicator
  Hostname:    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:51:54 UTC] USER=www-data EUID=0 PID=3014889 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-03 07:51:54 UTC] USER=www-data EUID=0 PID=3014898 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-03 07:51:54 UTC] USER=www-data EUID=0 PID=3014907 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-03 07:51:54 UTC] USER=www-data EUID=0 PID=3014916 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-03 07:51:54 UTC] USER=www-data EUID=0 PID=3014925 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:51:55 UTC] USER=www-data EUID=0 PID=3014945 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:51:55 UTC] USER=www-data EUID=0 PID=3014955 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:51:55 UTC] USER=www-data EUID=0 PID=3014969 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:51:55 UTC] USER=www-data EUID=0 PID=3014978 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-03 07:51:55 UTC] USER=www-data EUID=0 PID=3014987 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:51:55 UTC] USER=www-data EUID=0 PID=3014997 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:51:55 UTC] USER=www-data EUID=0 PID=3015007 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:51:55 UTC] USER=www-data EUID=0 PID=3015017 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:51:55 UTC] USER=www-data EUID=0 PID=3015026 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:51:55 UTC] USER=www-data EUID=0 PID=3015035 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:51:55 UTC] USER=www-data EUID=0 PID=3015044 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:51:55 UTC] USER=www-data EUID=0 PID=3015053 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:51:55 UTC] USER=www-data EUID=0 PID=3015062 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[2026-01-03 07:51:55 UTC] USER=www-data EUID=0 PID=3015071 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:51:55 UTC] USER=www-data EUID=0 PID=3015081 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:51:55 UTC] USER=www-data EUID=0 PID=3015090 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015099 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015108 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015136 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015145 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015154 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015163 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015172 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015181 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015191 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015200 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015209 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015218 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015227 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015237 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015247 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015256 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015265 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015274 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015283 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015292 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015301 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015310 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015319 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015328 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015337 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015347 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:56 UTC] USER=www-data EUID=0 PID=3015362 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015371 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015381 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015390 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015399 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015408 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015419 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015428 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015437 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015447 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015457 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015469 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015481 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015491 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015500 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015510 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015519 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015528 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015537 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015555 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015564 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015573 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator_der.key
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015583 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015593 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015602 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015611 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015621 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015630 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-03 07:51:57 UTC] USER=www-data EUID=0 PID=3015640 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:51:58 UTC] USER=www-data EUID=0 PID=3015649 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:51:58 UTC] USER=www-data EUID=0 PID=3015658 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:51:58 UTC] USER=www-data EUID=0 PID=3015667 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres

[OK]   Replicator certificate created for worker-01 in /home/postgres/
[INFO] Using replicator certificates from primary worker-01...
[2026-01-03 07:51:58 UTC] USER=www-data EUID=0 PID=3015695 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-03 07:51:58 UTC] USER=www-data EUID=0 PID=3015716 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.key
[2026-01-03 07:51:58 UTC] USER=www-data EUID=0 PID=3015737 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/replicator.crt
[OK]   Replicator certificates verified at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[OK]   root.crt verified at /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01
[INFO] Updating primary pg_hba.conf to allow replication...
[INFO]   Standby IP: 10.100.1.216/32 (standby's source IP)
[INFO]   Primary application IP: 10.100.1.215/32 (for local pg_basebackup)
[INFO]   Primary DNS IP: 10.100.1.215/32 (DNS resolution of db-identity-sau-main-dev-postgresql-worker-01.fastorder.com)
[2026-01-03 07:51:58 UTC] USER=www-data EUID=0 PID=3015767 ACTION=passthru ARGS=grep -qxF # BEGIN standby-replication (managed) /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-03 07:51:58 UTC] USER=www-data EUID=0 PID=3015811 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.216/32  scram-sha-256 
      $0==begin {inside=1}
      inside && $0==rule {found=1}
      $0==end {inside=0}
      END {exit found?0:1}
     /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-03 07:51:58 UTC] USER=www-data EUID=0 PID=3015841 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl  replication  replicator  10.100.1.216/32  scram-sha-256 /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-03 07:51:59 UTC] USER=www-data EUID=0 PID=3015862 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl  replication  replicator  10.100.1.215/32  scram-sha-256 
        $0==begin {inside=1}
        inside && $0==rule {found=1}
        $0==end {inside=0}
        END {exit found?0:1}
       /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[2026-01-03 07:51:59 UTC] USER=www-data EUID=0 PID=3015886 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl  replication  replicator  10.100.1.215/32  scram-sha-256 /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf
[INFO] Reloading primary PostgreSQL service...
[2026-01-03 07:51:59 UTC] USER=www-data EUID=0 PID=3015907 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
[OK]   Primary pg_hba.conf updated and service reloaded
[WARN] Removing existing data directory: /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[2026-01-03 07:51:59 UTC] USER=www-data EUID=0 PID=3015935 ACTION=fsop ARGS=rm -rf /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[INFO] Primary host: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO] Using replicator cert: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[INFO] Using replicator key: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key (PKCS#8 format)
[INFO] Using CA cert: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[INFO] Verifying postgres user can access certificates...
[ERR]  postgres user CANNOT read /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[INFO] File permissions:
lrwxrwxrwx 1 postgres ssl-cert 72 Jan  3 07:51 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt -> /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/root.crt
[INFO] Parent directory permissions:
drwx------ 2 postgres postgres 4096 Jan  3 07:51 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
drwx------ 5 postgres postgres 4096 Jan  3 07:51 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[WARN] Attempting to fix permissions (/usr/local/bin/fastorder-provisioning-wrapper.sh required)...
[INFO] Fixing /home/postgres/ directory...
[2026-01-03 07:52:00 UTC] USER=www-data EUID=0 PID=3016008 ACTION=fsop ARGS=chmod 755 /home/postgres/
[INFO] Fixing /home/postgres/ssl/.postgresql/...
[2026-01-03 07:52:00 UTC] USER=www-data EUID=0 PID=3016029 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/
[INFO] Fixing parent directory: /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:52:00 UTC] USER=www-data EUID=0 PID=3016053 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[INFO] Fixing certificate directory: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[2026-01-03 07:52:00 UTC] USER=www-data EUID=0 PID=3016074 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[INFO] Fixing CA certificate: /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[2026-01-03 07:52:00 UTC] USER=www-data EUID=0 PID=3016095 ACTION=fsop ARGS=chmod 644 /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[OK]   Permissions fixed
[OK]   postgres user can now read /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt after permission fix
[2026-01-03 07:52:00 UTC] USER=www-data EUID=0 PID=3016116 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[2026-01-03 07:52:00 UTC] USER=www-data EUID=0 PID=3016137 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[2026-01-03 07:52:00 UTC] USER=www-data EUID=0 PID=3016158 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[INFO] Checking primary database size before pg_basebackup...
[INFO] Total primary database size: 29 MB
[INFO] Estimated transfer time: ~0 minutes (at 10MB/s with compression)
[INFO] Retrieving replicator password from AWS Secrets Manager: fastorder/db/identity/sau/main/dev/postgresql/replicator
[OK]   Replicator password retrieved successfully
[INFO] Starting pg_basebackup...
[2026-01-03 07:52:03 UTC] USER=www-data EUID=0 PID=3016510 ACTION=passthru ARGS=sudo -u postgres env PGPASSWORD=qrzga0rZrBWHXjHNfE1t9bdwqo6QF84R PGSSLMODE=verify-full PGSSLCERT=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt PGSSLKEY=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key PGSSLROOTCERT=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt /usr/lib/postgresql/17/bin/pg_basebackup -h db-identity-sau-main-dev-postgresql-worker-01.fastorder.com -p 5432 -U replicator -D /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01 -Fp -Xs -P -R --checkpoint=fast --wal-method=stream --verbose
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/2000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_3016524"
23577/30540 kB (77%), 0/1 tablespace (...worker-01-standby-01/base/1/2840)
30550/30550 kB (100%), 0/1 tablespace (...-01-standby-01/global/pg_control)
30550/30550 kB (100%), 1/1 tablespace                                         
pg_basebackup: write-ahead log end point: 0/2000120
pg_basebackup: waiting for background process to finish streaming ...
pg_basebackup: syncing data to disk ...
pg_basebackup: renaming backup_manifest.tmp to backup_manifest
pg_basebackup: base backup completed
[OK]   pg_basebackup complete
[INFO] Fixing postgresql.auto.conf to use IP-based primary_conninfo (matching golden backup)...
[2026-01-03 07:52:05 UTC] USER=www-data EUID=0 PID=3016546 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-03 07:52:05 UTC] USER=www-data EUID=0 PID=3016568 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-03 07:52:05 UTC] USER=www-data EUID=0 PID=3016589 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-03 07:52:05 UTC] USER=www-data EUID=0 PID=3016598 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
[OK]   standby.signal verified and permissions set
[INFO] Fixing postgresql.conf with standby-specific settings...
[WARN] postgresql.conf not found at /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/postgresql.conf
[INFO] Verifying postgresql.auto.conf...
[WARN] postgresql.auto.conf not found - pg_basebackup may have failed
[INFO] Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[OK]   postgresql.conf updated successfully
[INFO] Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-03 07:52:05 UTC] USER=www-data EUID=0 PID=3016684 ACTION=fsop ARGS=cp /tmp/tmp.XcgOYkXJit /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/pg_hba.conf
[2026-01-03 07:52:06 UTC] USER=www-data EUID=0 PID=3016705 ACTION=fsop ARGS=chown postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/pg_hba.conf
[2026-01-03 07:52:06 UTC] USER=www-data EUID=0 PID=3016726 ACTION=fsop ARGS=chmod 600 /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/pg_hba.conf
[OK]   pg_hba.conf updated
[INFO] Creating systemd unit: /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01-standby-01.service
[2026-01-03 07:52:06 UTC] USER=www-data EUID=0 PID=3016751 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.rbhGGd /etc/systemd/system/postgresql@identity-sau-main-dev-worker-01-standby-01.service
[OK]   systemd unit written
[2026-01-03 07:52:06 UTC] USER=www-data EUID=0 PID=3016810 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-03 07:52:06 UTC] USER=www-data EUID=0 PID=3016832 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-03 07:52:06 UTC] USER=www-data EUID=0 PID=3016853 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] Starting PostgreSQL instance...
[2026-01-03 07:52:08 UTC] USER=www-data EUID=0 PID=3016979 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-worker-01-standby-01.service
[INFO] Waiting for ACTIVE (systemd)…
[2026-01-03 07:52:09 UTC] USER=www-data EUID=0 PID=3017031 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-worker-01-standby-01.service
[OK]   Service ACTIVE
[INFO] Waiting for port 5432 bind…
[OK]   Port bound
[INFO] Waiting pg_isready (socket)…
[OK]   Readiness via socket OK
[INFO] Waiting pg_isready (TCP db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com:5432)…
[OK]   Startup sequence complete
[INFO] Configuring synchronous replication on primary worker-01...
[INFO] Current synchronous_standby_names: ''
[INFO] Initializing synchronous_standby_names with first standby
[INFO] New synchronous_standby_names: 'ANY 1 (worker_01_standby_01)'
[2026-01-03 07:52:09 UTC] USER=www-data EUID=0 PID=3017098 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_commit = on;
ALTER SYSTEM
[2026-01-03 07:52:09 UTC] USER=www-data EUID=0 PID=3017121 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_standby_names = 'ANY 1 (worker_01_standby_01)';
ALTER SYSTEM
[2026-01-03 07:52:09 UTC] USER=www-data EUID=0 PID=3017150 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-identity-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

[OK]   ✅ Synchronous replication configured on primary
[OK]      Setting: ANY 1 (worker_01_standby_01)
[INFO] Validating core security GUCs (via local socket)…
[OK]   Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[INFO] Skipping database/role provisioning on standby node (read-only)
[INFO]   Database/roles will be replicated from primary: worker-01
[INFO] Applying connection and memory optimizations...
[INFO] Standby will use primary's max_connections: 100
[INFO] Current settings: max_connections=100, work_mem=8MB
[INFO] Target settings (standby): max_connections=100, work_mem=8MB
[OK]   Connection settings already optimized
[INFO] Skipping password setting - this is a standby (read-only)
[INFO] Use primary's postgres password to connect to this standby
[INFO] Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] PostgreSQL IP: 10.100.1.216
[INFO] Primary hostname: db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com

[INFO] Adding /etc/hosts entry for worker-01-standby-01...
[INFO]   db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.216

[INFO]   ✅ db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com already exists with correct IP

✅   ═══════════════════════════════════════════════════════════════
✅   ✅ Network & DNS configuration complete
✅   ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
  10.100.1.216    db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com


[OK]   PostgreSQL 'identity-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
  psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt \
        sslcert=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt \
        sslkey=/home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.key \
        host=db-identity-sau-main-dev-postgresql-worker-01-standby-01 port=5432 dbname=postgres user=postgres"
File  been compeleted perfectly: 02-setup-pg-instance
[INFO] Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-worker-01-standby-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.216
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-worker-01-standby-01
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 8eaa8059-bede-4f71-ae1d-d26590a898da
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   PostgreSQL node registered to observability API

[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 📦 03 role...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-03 07:52:16 UTC] USER=www-data EUID=0 PID=3017551 ACTION=fsop ARGS=test -f /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/standby.signal
⚠ This is a PostgreSQL STANDBY (read-only replica)
⚠ Skipping role creation - standby gets roles from primary via replication
⚠ Use the PRIMARY's credentials to connect to this standby


[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
ℹ️  Service-specific setup (identity) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)

🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps discovery
🔍 DEBUG_CHECKPOINT_02: Searching for service folders in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps
🔍 DEBUG_CHECKPOINT_03: Found directory: destroy
🔍 DEBUG_CHECKPOINT_03: Found directory: iam
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: iam
🔍 DEBUG_CHECKPOINT_03: Found directory: identity
🔍 DEBUG_CHECKPOINT_04: Found run.sh in: identity
🔍 DEBUG_CHECKPOINT_03: Found directory: lib
🔍 DEBUG_CHECKPOINT_03: Found directory: passwords
🔍 DEBUG_CHECKPOINT_03: Found directory: role
🔍 DEBUG_CHECKPOINT_03: Found directory: ssl
🔍 DEBUG_CHECKPOINT_05: Service folders found: iam identity
[INFO] 📚 Detected service folders: iam identity

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: iam at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/iam (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 🔸 Service: iam
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/iam/run.sh with IDENTIFIER=worker-01-standby-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running iam in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)

╔════════════════════════════════════════════════════════════════════════════╗
║                    IAM Database Schema Initialization                       ║
╚════════════════════════════════════════════════════════════════════════════╝

[INFO] 🟢 Starting IAM schema provisioning...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

[INFO] 📚 Discovered tables: core/01-tenant core/02-realm core/03-identity core/04-device core/05-identity_account core/06-identity_mfa core/07-external_idp_link policy/01-client policy/02-resource policy/03-scope policy/04-permission policy/05-role policy/06-role_permission policy/07-identity_role policy/08-policy_rule policy/09-api_key audit/01-auth_event audit/02-admin_action audit/03-risk_decision audit/04-consent_event


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: core
  Core Identity Directory (tenants, realms, identities, devices, MFA)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [1/20]: core/01-tenant
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.tenant Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "pgcrypto" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "citext" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Creating utils schema...
NOTICE:  schema "utils" already exists, skipping
CREATE SCHEMA
✅ Utils schema created
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating core schema...
NOTICE:  schema "core" already exists, skipping
CREATE SCHEMA
✅ Schema core created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating core.tenant table...
NOTICE:  relation "tenant" already exists, skipping
CREATE TABLE
COMMENT
COMMENT
COMMENT
✅ core.tenant created
🔧 Setting up Citus distribution for core.tenant...
✅ Citus distribution configured
🔧 Creating update trigger...
CREATE FUNCTION
ERROR:  triggers are not supported on reference tables
ERROR:  triggers are not supported on reference tables
✅ Update trigger created

✅ core.tenant initialization complete

[OK] Table core/01-tenant initialized

[INFO] 🔸 Table [2/20]: core/02-realm
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.realm Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.realm table...
NOTICE:  relation "realm" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_realm_keycloak_id" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_realm_tenant" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ core.realm created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.realm initialization complete

[OK] Table core/02-realm initialized

[INFO] 🔸 Table [3/20]: core/03-identity
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity table...
NOTICE:  relation "identity" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_unique_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_unique_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_type" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity initialization complete

[OK] Table core/03-identity initialized

[INFO] 🔸 Table [4/20]: core/04-device
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.device Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.device table...
NOTICE:  relation "device" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_device_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_fingerprint" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_trusted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_device_last_seen" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.device created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.device initialization complete

[OK] Table core/04-device initialized

[INFO] 🔸 Table [5/20]: core/05-identity_account
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_account Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_account table...
NOTICE:  relation "identity_account" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_account_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_lockout" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_account_last_login" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_account created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ core.identity_account initialization complete

[OK] Table core/05-identity_account initialized

[INFO] 🔸 Table [6/20]: core/06-identity_mfa
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.identity_mfa Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.identity_mfa table...
NOTICE:  relation "identity_mfa" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_mfa_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_mfa_active" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.identity_mfa created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.identity_mfa initialization complete

[OK] Table core/06-identity_mfa initialized

[INFO] 🔸 Table [7/20]: core/07-external_idp_link
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing core.external_idp_link Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating core.external_idp_link table...
NOTICE:  relation "external_idp_link" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_external_idp_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_provider" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_external_idp_email" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ core.external_idp_link created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ core.external_idp_link initialization complete

[OK] Table core/07-external_idp_link initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: policy
  RBAC/ABAC Authorization (clients, roles, permissions, policies)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [8/20]: policy/01-client
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.client Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy schema...
NOTICE:  schema "policy" already exists, skipping
CREATE SCHEMA
✅ Schema policy created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating policy.client table...
NOTICE:  relation "client" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_client_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_keycloak" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_key" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_client_status" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.client created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
CREATE FUNCTION
DROP TRIGGER
CREATE TRIGGER
✅ policy.client initialization complete

[OK] Table policy/01-client initialized

[INFO] 🔸 Table [9/20]: policy/02-resource
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.resource Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.resource table...
NOTICE:  relation "resource" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_resource_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_external" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_resource_owner" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.resource created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.resource initialization complete

[OK] Table policy/02-resource initialized

[INFO] 🔸 Table [10/20]: policy/03-scope
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.scope Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.scope table...
NOTICE:  relation "scope" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_scope_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_scope_name" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.scope created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.scope initialization complete

[OK] Table policy/03-scope initialized

[INFO] 🔸 Table [11/20]: policy/04-permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.permission table...
NOTICE:  relation "permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_permission_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_permission_resource" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.permission initialization complete

[OK] Table policy/04-permission initialized

[INFO] 🔸 Table [12/20]: policy/05-role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role table...
NOTICE:  relation "role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_name" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_keycloak" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.role initialization complete

[OK] Table policy/05-role initialized

[INFO] 🔸 Table [13/20]: policy/06-role_permission
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.role_permission Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.role_permission table...
NOTICE:  relation "role_permission" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_role_permission_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_role_permission_perm" already exists, skipping
CREATE INDEX
COMMENT
✅ policy.role_permission created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.role_permission initialization complete

[OK] Table policy/06-role_permission initialized

[INFO] 🔸 Table [14/20]: policy/07-identity_role
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.identity_role Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.identity_role table...
NOTICE:  relation "identity_role" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_identity_role_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_role" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_active" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_identity_role_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.identity_role created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.identity_role initialization complete

[OK] Table policy/07-identity_role initialized

[INFO] 🔸 Table [15/20]: policy/08-policy_rule
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.policy_rule Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.policy_rule table...
NOTICE:  relation "policy_rule" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_policy_rule_realm" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_enabled" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_policy_rule_priority" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ policy.policy_rule created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
DROP TRIGGER
CREATE TRIGGER
✅ policy.policy_rule initialization complete

[OK] Table policy/08-policy_rule initialized

[INFO] 🔸 Table [16/20]: policy/09-api_key
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing policy.api_key Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating policy.api_key table...
NOTICE:  relation "api_key" already exists, skipping
CREATE TABLE
NOTICE:  relation "idx_api_key_prefix" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_client" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_status" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_api_key_expires" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
COMMENT
✅ policy.api_key created
🔧 Setting up Citus distribution...
✅ Citus distribution configured
✅ policy.api_key initialization complete

[OK] Table policy/09-api_key initialized


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Schema: audit
  Audit & Risk Logging (auth events, admin actions, risk decisions)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔸 Table [17/20]: audit/01-auth_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.auth_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Database:    fastorder_identity_sau_main_dev_db
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit schema...
NOTICE:  schema "audit" already exists, skipping
CREATE SCHEMA
✅ Schema audit created
🔧 Creating ENUM types...
DO
✅ ENUM types created
🔧 Creating audit.auth_event table...
NOTICE:  relation "auth_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_auth_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_result" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_ip" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_session" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_trace" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_auth_event_risk" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.auth_event created (partitioned)
✅ audit.auth_event initialization complete

[OK] Table audit/01-auth_event initialized

[INFO] 🔸 Table [18/20]: audit/02-admin_action
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.admin_action Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.admin_action table...
NOTICE:  relation "admin_action" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_admin_action_actor" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_target" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_time" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_admin_action_trace" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.admin_action created (partitioned)
✅ audit.admin_action initialization complete

[OK] Table audit/02-admin_action initialized

[INFO] 🔸 Table [19/20]: audit/03-risk_decision
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.risk_decision Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.risk_decision table...
NOTICE:  relation "risk_decision" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_risk_decision_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_level" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_decision" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_auth" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_risk_decision_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.risk_decision created (partitioned)
✅ audit.risk_decision initialization complete

[OK] Table audit/03-risk_decision initialized

[INFO] 🔸 Table [20/20]: audit/04-consent_event
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing audit.consent_event Table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Creating audit.consent_event table...
NOTICE:  relation "consent_event" already exists, skipping
CREATE TABLE
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
DO
NOTICE:  relation "idx_consent_event_identity" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_type" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_version" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_granted" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_consent_event_time" already exists, skipping
CREATE INDEX
COMMENT
COMMENT
✅ audit.consent_event created (partitioned)
🔧 Creating partition management functions...
CREATE FUNCTION
NOTICE:  relation "audit.auth_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_01
NOTICE:  relation "audit.auth_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_02
NOTICE:  relation "audit.auth_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_03
NOTICE:  relation "audit.auth_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.auth_event_2026_04
NOTICE:  relation "audit.admin_action_2026_01" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_01
NOTICE:  relation "audit.admin_action_2026_02" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_02
NOTICE:  relation "audit.admin_action_2026_03" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_03
NOTICE:  relation "audit.admin_action_2026_04" already exists, skipping
NOTICE:  Created partition: audit.admin_action_2026_04
NOTICE:  relation "audit.risk_decision_2026_01" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_01
NOTICE:  relation "audit.risk_decision_2026_02" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_02
NOTICE:  relation "audit.risk_decision_2026_03" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_03
NOTICE:  relation "audit.risk_decision_2026_04" already exists, skipping
NOTICE:  Created partition: audit.risk_decision_2026_04
NOTICE:  relation "audit.consent_event_2026_01" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_01
NOTICE:  relation "audit.consent_event_2026_02" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_02
NOTICE:  relation "audit.consent_event_2026_03" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_03
NOTICE:  relation "audit.consent_event_2026_04" already exists, skipping
NOTICE:  Created partition: audit.consent_event_2026_04
 create_monthly_partitions 
---------------------------
 
(1 row)

CREATE VIEW
CREATE FUNCTION
COMMENT
COMMENT
✅ Partition management functions created
✅ audit.consent_event initialization complete

[OK] Table audit/04-consent_event initialized


════════════════════════════════════════════════════════════════════════════
[OK] ✅ IAM Schema Initialization Complete!
[OK] All 20 tables initialized successfully

Schemas created:
  • core   - Identity directory (tenant, realm, identity, devices, MFA)
  • policy - Authorization (clients, roles, permissions, policies, API keys)
  • audit  - Logging (auth events, admin actions, risk decisions, consent)

Design highlights:
  • Citus-ready with tenant_id distribution key
  • NIST 800-63 identity compliance
  • PCI DSS 4.0 audit logging
  • GDPR consent tracking
  • Keycloak integration via ID references

════════════════════════════════════════════════════════════════════════════

🔍 DEBUG_CHECKPOINT_06: Preparing to run service: identity at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh
[DEBUG] Tracking substep start: steps/01-install/steps/identity (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
[INFO] 🔸 Service: identity
🔍 DEBUG_CHECKPOINT_07: About to execute /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/run.sh with IDENTIFIER=worker-01-standby-01 IDENTIFIER_PARENT=worker-01
🔍 DEBUG_CHECKPOINT_08: Running identity in AUTO mode
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting PostgreSQL provisioning for identity in sau-dev...
[INFO] Environment: identity-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16

🔍 DEBUG_CHECKPOINT_A1: identity/run.sh started for SERVICE=identity
🔍 DEBUG_CHECKPOINT_A2: Checking SERVICE_ROOT: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity
🔍 DEBUG_CHECKPOINT_A3: SERVICE_ROOT exists, discovering table folders
🔍 DEBUG_CHECKPOINT_A4: Found subfolder: auth
🔍 DEBUG_CHECKPOINT_A4b: Checking for nested schema layout in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth
🔍 DEBUG_CHECKPOINT_A4c: Found nested steps dir: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps (display: auth/login)
🔍 DEBUG_CHECKPOINT_A5: Table step dirs discovered: auth/login|/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A6: Checking if we have table folders to process
[INFO] 📚 Detected grouped table folders under identity/: auth/login

🔍 DEBUG_CHECKPOINT_A7: Current IDENTIFIER=coordinator
🔍 DEBUG_CHECKPOINT_A8_PROCEED: Processing tables on coordinator/main node
🔍 DEBUG_CHECKPOINT_A9: Processing table: auth/login at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
[INFO] 🔸 Table group: auth/login
🔍 DEBUG_CHECKPOINT_A10: About to run numbered steps for table: auth/login
🔍 DEBUG_CHECKPOINT_B1: run_all_numbered_steps_in_dir called for dir=/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps table=auth/login
🔍 DEBUG_CHECKPOINT_B2: Found 1 numbered steps: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B3: About to run step: 01-init-schema.sh
Ab substep 0 compelete start
[DEBUG] Tracking substep start: steps/01-install/steps/identity/auth/login/01-init-schema (RUN_UUID=bd319392-f0b4-4ed1-a7ea-48f73d9a2895)
Ab substep 0 compelete start
[INFO] 📦 01 init schema...
Ab substep 1 compelete start
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Initializing auth.login_account table
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️  Checking database: fastorder_identity_sau_main_dev_db
ℹ️  Database fastorder_identity_sau_main_dev_db already exists
✅ Connected to database: fastorder_identity_sau_main_dev_db
ℹ️  Checking synchronous replication configuration...
   synchronous_standby_names: ''
   Connected standbys: 0
ℹ️  Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "dblink" already exists, skipping
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
NOTICE:  extension "citus" already exists, skipping
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating auth schema...
NOTICE:  schema "auth" already exists, skipping
CREATE SCHEMA
✅ Schema created
🔧 Creating account_status ENUM...
DO
✅ ENUM created
🔧 Creating auth.login_account table...
NOTICE:  relation "login_account" already exists, skipping
CREATE TABLE
✅ Table created (Citus-compatible with region_hint in all constraints)
🔧 Creating indexes...
NOTICE:  relation "idx_login_account_email" already exists, skipping
CREATE INDEX
NOTICE:  relation "idx_login_account_username" already exists, skipping
CREATE INDEX
✅ Indexes created
ℹ️  Table already registered with Citus
🎉 Schema initialization complete for fastorder_identity_sau_main_dev_db
ℹ️  Skipping LISTEN/NOTIFY trigger on coordinator
   CDC via Debezium is the primary change tracking mechanism

📊 Registering environment in monitoring database (obs schema)...
   Topology: /opt/fastorder/bash/scripts/env_app_setup/state/identity-sau-main-dev/topology.json
   Resource IP: 142.93.238.16
⚠️  Could not connect to monitoring database, skipping registration
   You can manually register later using:
   /opt/fastorder/bash/scripts/env_app_setup/setup/04-postgresql/steps/register-authN-af-aaaa1-dev.sh

==========================================
✅ Schema initialization complete!
==========================================
Ab substep 1 compelete end
Ab substep 2 compelete start
Ab substep 2 compelete end

🔍 DEBUG_CHECKPOINT_B4: Completed step: 01-init-schema.sh
🔍 DEBUG_CHECKPOINT_B5: All numbered steps completed for /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/identity/../identity/auth/login/steps
🔍 DEBUG_CHECKPOINT_A11: Completed numbered steps for table: auth/login
compeleted here

🔍 DEBUG_CHECKPOINT_A12: All tables processed
End of 04-postgresql/steps/01-install/steps/identity/run.sh

✓ ✅ Standby worker-01-standby-01 setup completed

✓ ✅ PostgreSQL installation completed
[INFO] Discovering additional setup steps...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 02-pg-bouncer.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up PgBouncer connection pooling...
[2026-01-03 07:53:55 UTC] USER=www-data EUID=0 PID=3021486 ACTION=fsop ARGS=rm -f /tmp/pgbouncer-ip.service /tmp/pgbouncer.service
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] Checking for existing PgBouncer application environment in topology …
[OK]   Using existing PgBouncer environment:
[INFO]   IP:     10.100.1.204
[INFO]   FQDN:   db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Domain: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO] Ensuring /etc/hosts entry for db-identity-sau-main-dev-postgresql-bouncer.fastorder.com …
[INFO] Adding db-identity-sau-main-dev-postgresql-bouncer.fastorder.com to /etc/hosts
[2026-01-03 07:53:56 UTC] USER=www-data EUID=0 PID=3021546 ACTION=fsop ARGS=sed -i /\sdb-identity-sau-main-dev-postgresql-bouncer.fastorder.com\(\s\|$\)/d /etc/hosts
[OK]   Added db-identity-sau-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.204 to /etc/hosts
[WARN] IP 10.100.1.204 is assigned to multiple interfaces:
    inet 10.100.1.103/32 scope global lo
       valid_lft forever preferred_lft forever
    inet 10.100.1.204/32 scope global lo:pgbouncer
--
    inet 10.100.1.214/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.100.1.204/32 scope global eth0:pgbouncer
[WARN] This may cause routing issues
[INFO] Final verification of /etc/hosts entry for db-identity-sau-main-dev-postgresql-bouncer.fastorder.com …
[OK]   /etc/hosts correctly maps db-identity-sau-main-dev-postgresql-bouncer.fastorder.com to 10.100.1.204
[OK]   PgBouncer IP 10.100.1.204 already correctly bound to lo:pgbouncer
[2026-01-03 07:53:56 UTC] USER=www-data EUID=0 PID=3021597 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-03 07:53:57 UTC] USER=www-data EUID=0 PID=3021693 ACTION=passthru ARGS=systemctl restart pgbouncer-ip@identity-sau-main-dev.service
[2026-01-03 07:53:57 UTC] USER=www-data EUID=0 PID=3021704 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer-ip@identity-sau-main-dev.service
[OK]   pgbouncer-ip@identity-sau-main-dev.service is active
[2026-01-03 07:53:57 UTC] USER=www-data EUID=0 PID=3021733 ACTION=fsop ARGS=mkdir -p /etc/pgbouncer/identity-sau-main-dev
[2026-01-03 07:53:57 UTC] USER=www-data EUID=0 PID=3021742 ACTION=fsop ARGS=mkdir -p /run/pgbouncer/identity-sau-main-dev
[2026-01-03 07:53:58 UTC] USER=www-data EUID=0 PID=3021751 ACTION=fsop ARGS=mkdir -p /var/log/pgbouncer/identity-sau-main-dev
[2026-01-03 07:53:58 UTC] USER=www-data EUID=0 PID=3021760 ACTION=fsop ARGS=chmod 750 /etc/pgbouncer/identity-sau-main-dev
[2026-01-03 07:53:58 UTC] USER=www-data EUID=0 PID=3021769 ACTION=fsop ARGS=chmod 750 /run/pgbouncer/identity-sau-main-dev
[2026-01-03 07:53:58 UTC] USER=www-data EUID=0 PID=3021778 ACTION=fsop ARGS=chmod 750 /var/log/pgbouncer/identity-sau-main-dev
[2026-01-03 07:53:58 UTC] USER=www-data EUID=0 PID=3021787 ACTION=fsop ARGS=chown root:postgres /etc/pgbouncer/identity-sau-main-dev
[2026-01-03 07:53:58 UTC] USER=www-data EUID=0 PID=3021796 ACTION=fsop ARGS=chown postgres:postgres /run/pgbouncer/identity-sau-main-dev
[2026-01-03 07:53:58 UTC] USER=www-data EUID=0 PID=3021805 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbouncer/identity-sau-main-dev
[INFO] Generating pgbouncer_admin client certificates...
[INFO] ⏳ This may take 30-60 seconds...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
Environment: identity-sau-main-dev
Username:    pgbouncer_admin
Identifier:  pgbouncer
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Service:     identity
  Zone:        sau
  Branch:      main
  Env:         dev
  Node:        pgbouncer
  User (CN):   pgbouncer_admin
  Hostname:    db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:53:58 UTC] USER=www-data EUID=0 PID=3021847 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-pgbouncer-pgbouncer_admin
[2026-01-03 07:53:58 UTC] USER=www-data EUID=0 PID=3021856 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-03 07:53:58 UTC] USER=www-data EUID=0 PID=3021867 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
[2026-01-03 07:53:58 UTC] USER=www-data EUID=0 PID=3021876 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3021885 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = pgbouncer_admin
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3021902 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3021911 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3021920 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3021929 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3021938 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3021947 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3021956 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key.pkcs1 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3021965 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_der.key /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3021974 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3021983 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3021992 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3022001 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3022010 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3022019 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3022034 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3022043 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-03 07:53:59 UTC] USER=www-data EUID=0 PID=3022052 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022087 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022096 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022105 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022115 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022125 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022138 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022147 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022157 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022166 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022176 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022186 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022196 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022206 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022215 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022224 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022233 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022242 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022252 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022262 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-03 07:54:00 UTC] USER=www-data EUID=0 PID=3022272 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-03 07:54:01 UTC] USER=www-data EUID=0 PID=3022291 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-03 07:54:01 UTC] USER=www-data EUID=0 PID=3022300 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-03 07:54:01 UTC] USER=www-data EUID=0 PID=3022310 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-03 07:54:01 UTC] USER=www-data EUID=0 PID=3022320 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-03 07:54:01 UTC] USER=www-data EUID=0 PID=3022329 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:54:01 UTC] USER=www-data EUID=0 PID=3022338 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:54:01 UTC] USER=www-data EUID=0 PID=3022347 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-03 07:54:01 UTC] USER=www-data EUID=0 PID=3022398 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-03 07:54:01 UTC] USER=www-data EUID=0 PID=3022448 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-03 07:54:01 UTC] USER=www-data EUID=0 PID=3022504 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-03 07:54:01 UTC] USER=www-data EUID=0 PID=3022558 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-03 07:54:01 UTC] USER=www-data EUID=0 PID=3022614 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-03 07:54:01 UTC] USER=www-data EUID=0 PID=3022641 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-03 07:54:01 UTC] USER=www-data EUID=0 PID=3022651 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-03 07:54:01 UTC] USER=www-data EUID=0 PID=3022673 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-03 07:54:01 UTC] USER=www-data EUID=0 PID=3022695 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:54:01 UTC] USER=www-data EUID=0 PID=3022704 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022713 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022722 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022731 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022740 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022749 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022758 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022767 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022777 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022793 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
🎉 All requested users processed.

📋 Creating Kafka SSL certificate symlinks for www-data...
   Source: /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem
   Destination: /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022803 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022812 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022821 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
   ✅ Symlinked ca.pem
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022830 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
   ✅ Symlinked client-cert.pem
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022841 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem
   ✅ Symlinked client-key.pem
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022851 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/identity-sau-main-dev
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022860 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022878 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/identity-sau-main-dev/coordinator/pem/client-cert.pem
   ✅ Kafka certificate symlinks ready for www-data
      PHP Kafka consumers can now use:
      - ssl.ca.location: /var/www/ssl/kafka/identity-sau-main-dev/ca.pem
      - ssl.certificate.location: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem
      - ssl.key.location: /var/www/ssl/kafka/identity-sau-main-dev/client-key.pem

✅ Client certificate generated successfully!

Environment: identity-sau-main-dev
User: pgbouncer_admin
Node: pgbouncer
FQDN: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com

Next steps for Kafka Connect (Debezium → Postgres):

- Point connector to PEM key files:
    database.sslcert:     /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
    database.sslkey:      /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key   # PKCS#8 PEM
    database.sslrootcert: /home/kafka/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt

- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
  and use the container path in connector config.

For local testing:
    export PGSSLCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt"
    export PGSSLKEY="/home/$USER/ssl/.postgresql/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key"
    export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/identity-sau-main-dev/pgbouncer/root.crt"
    export PGSSLMODE="verify-full"

    psql -h db-identity-sau-main-dev-postgresql-bouncer.fastorder.com -U pgbouncer_admin -d postgres

[OK]   mTLS client certificate present: /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[INFO] Creating symlinks to canonical certificates in /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend...
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022895 ACTION=fsop ARGS=mkdir -p /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022904 ACTION=fsop ARGS=mkdir -p /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022913 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022922 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022931 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt
[INFO] Creating coordinator CA symlink for PostgreSQL server verification...
[2026-01-03 07:54:02 UTC] USER=www-data EUID=0 PID=3022940 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] Verifying canonical certificate permissions...
[2026-01-03 07:54:03 UTC] USER=www-data EUID=0 PID=3022952 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-03 07:54:03 UTC] USER=www-data EUID=0 PID=3022961 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-03 07:54:03 UTC] USER=www-data EUID=0 PID=3022970 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt
[2026-01-03 07:54:03 UTC] USER=www-data EUID=0 PID=3022979 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key
[OK]   Backend certificate symlinks created in /etc/ssl
[OK]   Coordinator CA symlink created for server verification
[OK]   Certificates already in canonical location - no symlinks needed
[2026-01-03 07:54:03 UTC] USER=www-data EUID=0 PID=3022991 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.crt
[2026-01-03 07:54:03 UTC] USER=www-data EUID=0 PID=3023000 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.key
[2026-01-03 07:54:03 UTC] USER=www-data EUID=0 PID=3023009 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt
[2026-01-03 07:54:03 UTC] USER=www-data EUID=0 PID=3023018 ACTION=fsop ARGS=test -r /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] PgBouncer will use PostgreSQL coordinator CA: /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[OK]   PostgreSQL coordinator at db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432 is reachable
[INFO] Dumping SCRAM secrets from coordinator for PgBouncer auth_file …
[2026-01-03 07:54:03 UTC] USER=www-data EUID=0 PID=3023037 ACTION=fsop ARGS=cp /tmp/tmp.EdONAqKGJ2 /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-03 07:54:03 UTC] USER=www-data EUID=0 PID=3023046 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-03 07:54:03 UTC] USER=www-data EUID=0 PID=3023056 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[OK]   Auth file written: /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[INFO] Retrieved password from vault for pgbouncer_admin
[INFO] Ensuring PgBouncer admin role 'pgbouncer_admin' exists in Postgres (coordinator) …
[OK]   Role pgbouncer_admin created/updated successfully
[SECRETS] Setting credentials in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/pgbouncer_admin
✓ [SECRETS] Credentials updated in vault: fastorder/db/identity/sau/main/dev/postgresql/coordinator/pgbouncer_admin
[INFO] ✅ PgBouncer admin password stored in centralized secrets vault
[INFO] Re-fetching SCRAM secrets after role creation to ensure pgbouncer_admin is included …
[2026-01-03 07:54:12 UTC] USER=www-data EUID=0 PID=3023226 ACTION=fsop ARGS=cp /tmp/tmp.F0s0sUDL5v /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-03 07:54:12 UTC] USER=www-data EUID=0 PID=3023235 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[2026-01-03 07:54:12 UTC] USER=www-data EUID=0 PID=3023244 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[OK]   Auth file updated with pgbouncer_admin SCRAM hash
[INFO] Auth file contains [2026-01-03 07:54:12 UTC] USER=www-data EUID=0 PID=3023254 ACTION=passthru ARGS=bash -c wc -l < '/etc/pgbouncer/identity-sau-main-dev/userlist.txt'
4 user(s)
[OK]   Admin 'pgbouncer_admin' password generated and saved
[INFO] Configuring PostgreSQL to prevent Citus metadata sync hangs...
ALTER ROLE
[OK]   Disabled Citus metadata sync for pgbouncer_admin
[INFO] Verifying application database fastorder_identity_sau_main_dev_db exists...
[OK]   ✓ Database fastorder_identity_sau_main_dev_db exists
[INFO] Granting permissions to pgbouncer_admin on fastorder_identity_sau_main_dev_db...
GRANT
[OK]   ✓ Granted CONNECT on fastorder_identity_sau_main_dev_db to pgbouncer_admin
GRANT
[OK]   ✓ Granted USAGE on schema public to pgbouncer_admin
GRANT
[OK]   ✓ Granted SELECT on all tables to pgbouncer_admin
ALTER DATABASE
[OK]   Set synchronous_commit=local for fastorder_identity_sau_main_dev_db
[INFO] Ensuring pg_hba.conf entry for pgbouncer_admin …
[INFO] Adding pg_hba.conf entries for pgbouncer_admin with cert auth …
[OK]   pg_hba.conf updated and PostgreSQL configuration reloaded
[2026-01-03 07:54:13 UTC] USER=unknown EUID=33 PID=3023295 ACTION=-u ARGS=postgres bash
ERROR: Invalid or unauthorized action: -u
[WARN] pg_hba.conf entry may not have loaded correctly
[INFO] Writing /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini …
[2026-01-03 07:54:14 UTC] USER=www-data EUID=0 PID=3023332 ACTION=fsop ARGS=cp /tmp/tmp.Qk6UBmb1z1 /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
[2026-01-03 07:54:14 UTC] USER=www-data EUID=0 PID=3023341 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
[2026-01-03 07:54:14 UTC] USER=www-data EUID=0 PID=3023350 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
[2026-01-03 07:54:14 UTC] USER=www-data EUID=0 PID=3023359 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbouncer/identity-sau-main-dev /run/pgbouncer/identity-sau-main-dev /var/log/pgbouncer/identity-sau-main-dev
[2026-01-03 07:54:14 UTC] USER=www-data EUID=0 PID=3023368 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[OK]   pgbouncer.ini ready
[INFO] Verifying TLS settings in pgbouncer.ini:
[2026-01-03 07:54:14 UTC] USER=www-data EUID=0 PID=3023378 ACTION=fsop ARGS=grep -E (client_tls_sslmode|server_tls) /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
client_tls_sslmode = verify-full
server_tls_sslmode = verify-full
server_tls_ca_file = /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
server_tls_cert_file = /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
server_tls_key_file  = /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[INFO] Verifying PgBouncer server certificate files:
[2026-01-03 07:54:14 UTC] USER=www-data EUID=0 PID=3023387 ACTION=fsop ARGS=test -r /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[OK]   Server cert readable by postgres: /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-01-03 07:54:14 UTC] USER=www-data EUID=0 PID=3023396 ACTION=fsop ARGS=test -r /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[OK]   Server key readable by postgres: /etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[INFO] Verifying coordinator CA certificate:
[2026-01-03 07:54:14 UTC] USER=www-data EUID=0 PID=3023405 ACTION=fsop ARGS=test -r /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[OK]   Coordinator CA readable by postgres: /etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[INFO] Preflight: stopping any conflicting PgBouncer on 6432 …
[2026-01-03 07:54:14 UTC] USER=www-data EUID=0 PID=3023414 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer.service
[2026-01-03 07:54:14 UTC] USER=www-data EUID=0 PID=3023428 ACTION=passthru ARGS=systemctl stop pgbouncer@identity-sau-main-dev.service
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json?all=1": dial unix /var/run/docker.sock: connect: permission denied
[WARN] Killing existing pgbouncer processes: 2059866
2059910
[2026-01-03 07:54:15 UTC] USER=www-data EUID=0 PID=3023452 ACTION=passthru ARGS=bash -c kill -9 2059866
[2026-01-03 07:54:15 UTC] USER=www-data EUID=0 PID=3023463 ACTION=passthru ARGS=bash -c kill -9 2059910
[2026-01-03 07:54:17 UTC] USER=www-data EUID=0 PID=3023519 ACTION=passthru ARGS=systemctl daemon-reload
[OK]   systemd unit installed: pgbouncer@identity-sau-main-dev.service
[INFO] Running pre-flight IP conflict check for 10.100.1.204:6432 …
[WARN] IP conflict checker not found at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/check-ip-conflicts.sh
[WARN] Skipping pre-flight check - conflicts may occur
[INFO] Starting PgBouncer (identity-sau-main-dev) …
[2026-01-03 07:54:18 UTC] USER=www-data EUID=0 PID=3023624 ACTION=passthru ARGS=systemctl restart pgbouncer@identity-sau-main-dev.service
[2026-01-03 07:54:18 UTC] USER=www-data EUID=0 PID=3023635 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer@identity-sau-main-dev.service
[OK]   Service ACTIVE
[INFO] Verifying auth_file before probing …
[INFO] Auth file contains 4 user(s)
[WARN] Auth file does NOT contain pgbouncer_admin entry - authentication will fail
[INFO] Probing admin console via SSL (psql to database 'pgbouncer') …
[INFO] Retrieved password from vault for admin console probe
[WARN] SSL connection issue detected
[INFO] Attempting connection with sslmode=disable for testing...
[WARN] If this fails, check PgBouncer client_tls_sslmode setting
[WARN] Admin console probe failed (see error below)
psql: error: connection to server at "10.100.1.204", port 6432 failed: SSL error: certificate verify failed
[WARN] Troubleshooting:
[WARN]   1. Check auth_file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[WARN]   2. Test with: PGPASSWORD='kppzNMG6WDrJWGUYcBARr4ME' psql -h 10.100.1.204 -p 6432 -U pgbouncer_admin -d pgbouncer
[WARN]   3. Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@identity-sau-main-dev.service -n 50

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Running Comprehensive PgBouncer Verification Tests
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Password extracted: kppzNMG6WD... (using postgres user certificates)

[INFO] Test 1/7: Admin Console - SHOW POOLS
[WARN] ✗ SHOW POOLS: FAILED
[WARN] Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@identity-sau-main-dev.service -n 50

[INFO] Test 2/7: Admin Console - SHOW VERSION
[WARN] ✗ SHOW VERSION: FAILED

[INFO] Test 3/7: Admin Console - SHOW STATS
[WARN] ✗ SHOW STATS: FAILED

[INFO] Test 4/7: Admin Console - SHOW DATABASES
[WARN] ✗ SHOW DATABASES: FAILED

[INFO] Test 5/7: Admin Console - SHOW CONFIG
[WARN] ✗ SHOW CONFIG: FAILED
psql   "host=db-identity-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_identity_sau_main_dev_db user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME    connect_timeout=5 sslmode=verify-full    sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/root.crt    sslcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.crt    sslkey=/etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/pgbouncer_admin.key"   --no-psqlrc -Atc 'SELECT version();'

[INFO] Test 6/7: Application Database - SELECT version()
[WARN] ✗ Application database query: FAILED (timeout or connection issue)
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[INFO] Test 7/8: Application Database - Connection Details
[WARN] ✗ Connection details: FAILED (timeout or connection issue)
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh

[INFO] Test 8/8: End-to-End Application Routing - Pool Verification
[INFO]   Running actual queries through PgBouncer to verify routing and pooling...
[WARN] ✗ End-to-end routing verification: FAILED - All 3 queries failed
[WARN]    If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[WARN]    Otherwise check if database fastorder_identity_sau_main_dev_db exists and user pgbouncer_admin has permissions

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Verification Complete - Tests 1-5 PASSED (Admin console verified)
[WARN]   Tests 6-8 FAILED - Application database not accessible
[WARN]   This is expected if Citus is not set up yet
[WARN]   Run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[OK]   PgBouncer is up for identity-sau-main-dev

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Connection Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Password stored in: AWS Secrets Manager (fastorder/db/web/ksa/main/dev/postgresqlidentity/sau/main/dev/coordinator-pgbouncer_admin)
Current password: kppzNMG6WDrJWGUYcBARr4ME

1. Admin Console (using IP address to avoid DNS/SSL issues):
   psql "host=10.100.1.204 port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME sslmode=verify-full sslrootcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

2. Admin Console (using hostname):
   psql "host=db-identity-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME sslmode=verify-full sslrootcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

3. Application Database:
   psql "host=db-identity-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_identity_sau_main_dev_db sslkey=/etc/ssl/private/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=kppzNMG6WDrJWGUYcBARr4ME sslmode=verify-full sslrootcert=/etc/ssl/certs/identity-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"

4. Using .pgpass file:
   echo "db-identity-sau-main-dev-postgresql-bouncer.fastorder.com:6432:*:pgbouncer_admin:kppzNMG6WDrJWGUYcBARr4ME" >> ~/.pgpass
   chmod 600 ~/.pgpass
   psql -h db-identity-sau-main-dev-postgresql-bouncer.fastorder.com -p 6432 -U pgbouncer_admin -d fastorder_identity_sau_main_dev_db

5. Retrieve password from vault:
   source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
   PGPASSWORD="$(get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password')" \
     psql -h 10.100.1.204 -p 6432 -U pgbouncer_admin -d pgbouncer -c "SHOW POOLS;"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Architecture
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  • Default db 'fastorder_identity_sau_main_dev_db' → Citus coordinator (db-identity-sau-main-dev-postgresql-coordinator.fastorder.com)
  • Worker access: 'fastorder_identity_sau_main_dev_db_worker_1', 'fastorder_identity_sau_main_dev_db_worker_2', … (if exist)
  • Client TLS: require (password auth) / verify-full (mTLS with certs)
  • Server TLS: verify-full (PgBouncer validates PostgreSQL certs)
  • Auth: SCRAM-SHA-256 via /etc/pgbouncer/identity-sau-main-dev/userlist.txt
  • Pool mode: transaction (stateless connections)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Management
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Service Status:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer@identity-sau-main-dev.service
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@identity-sau-main-dev.service

Logs:
  command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@identity-sau-main-dev.service -f
  /usr/local/bin/fastorder-provisioning-wrapper.sh tail -f /var/log/pgbouncer/identity-sau-main-dev/pgbouncer.log

Reload Config:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@identity-sau-main-dev.service

Restart:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@identity-sau-main-dev.service

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Files
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Config:        /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
Auth file:     /etc/pgbouncer/identity-sau-main-dev/userlist.txt
Server cert:   /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.crt
Server key:    /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/server.key
CA cert:       /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer/ca.crt
PG CA:         /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
Logs:          /var/log/pgbouncer/identity-sau-main-dev/pgbouncer.log

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Troubleshooting
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


If "SASL authentication failed":
  1. Check auth file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/identity-sau-main-dev/userlist.txt
  2. Verify pgbouncer_admin is present with SCRAM hash
  3. Get password from vault:
     source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
     get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password'
  4. Reload PgBouncer: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@identity-sau-main-dev.service

If "no pg_hba.conf entry":
  1. Check pg_hba.conf on coordinator
  2. Add rule: hostssl all pgbouncer_admin 10.100.1.204/32 cert clientcert=verify-full
  3. Reload PostgreSQL

To add users to PgBouncer:
  1. Create user in PostgreSQL with password
  2. Re-run SCRAM dump:
     psql "host=db-identity-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 dbname=postgres user=postgres \
       sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/root.crt \
       sslcert=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.crt sslkey=/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key" \
       -Atc "SELECT '\"' || rolname || '\" \"' || rolpassword || '\"' \
             FROM pg_authid WHERE rolpassword LIKE 'SCRAM-SHA-256%' \
             AND rolcanlogin ORDER BY rolname;" | command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop tee /etc/pgbouncer/identity-sau-main-dev/userlist.txt
  3. Reload: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@identity-sau-main-dev.service

[INFO] Registering PgBouncer node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        identity-sau-main-dev-pgbouncer
[INFO]   Identifier Parent: postgresql
[INFO]   IP:                10.100.1.204
[INFO]   Port:              6432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 426480a5-2f64-4fc0-b2b5-710f9ccb059a
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   PgBouncer node registered to observability API
✓ ✅ PgBouncer setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 03-citus-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] CITUS DISTRIBUTED CLUSTER SETUP
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Phase 1: Installing Citus extension on workers...
[INFO] Phase 2: Setting up coordinator and registering workers...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 📦 PHASE 1: Installing Citus extension on 1 worker(s)...

[INFO] → Worker 1/1: Installing Citus on worker-01...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS CLUSTER SETUP
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔧 Setting up Citus Worker...
[INFO] Temporarily disabling synchronous replication for extension installation...
t
[INFO] Installing Citus extension on worker...
[OK]   Citus extension installed on worker
[INFO] Restoring synchronous replication settings...
t
[INFO] Worker Citus extension installed - registration will happen when coordinator setup runs

[OK]   Citus setup complete for worker-01
[INFO] ═══════════════════════════════════════════════════════════════════════════════
✓   ✅ Citus extension installed on worker-01

✓ ✅ Phase 1 Complete: All 1 workers have Citus extension installed

[INFO] 🔧 PHASE 2: Setting up Citus coordinator and registering workers...

[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS CLUSTER SETUP
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔧 Setting up Citus Coordinator...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] DIAGNOSTIC: Configuration Variables
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PG_WORKERS_NUM: 1
[INFO] ENV_ID: identity-sau-main-dev
[INFO] DOMAIN: fastorder.com
[INFO] PORT: 5432
[INFO] SOCKET_DIR: /var/run/postgresql-identity-sau-main-dev-coordinator
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Ensuring postgres client certificates exist for coordinator...
[OK]   Postgres client certificates already exist for coordinator
[INFO] Adding citus_cert_map to coordinator pg_ident.conf...
[OK]   pg_ident.conf updated for coordinator
[INFO] Installing Citus extension on coordinator...
[OK]   Citus extension installed on coordinator (postgres database)
[INFO] Installing Citus extension on application database: fastorder_identity_sau_main_dev_db...
[OK]   Citus extension installed on application database: fastorder_identity_sau_main_dev_db
[INFO] Configuring Citus SSL connection parameters...
[2026-01-03 07:54:34 UTC] USER=www-data EUID=0 PID=3024096 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
[OK]   ✅ Citus SSL connection parameters configured: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator
[WARN] Node not identified as coordinator, initializing...
[INFO] Checking coordinator configuration...
[INFO] Persisting citus.local_hostname to postgresql.conf...
[2026-01-03 07:54:36 UTC] USER=www-data EUID=0 PID=3024154 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /var/lib/postgresql/17/identity-sau-main-dev/coordinator/postgresql.conf
[2026-01-03 07:54:36 UTC] USER=www-data EUID=0 PID=3024175 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
[OK]   ✅ citus.local_hostname persisted to config and reloaded
[INFO] Configuring coordinator hostname in postgres database: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432

[OK]   ✅ Coordinator hostname set to db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432 in postgres database
[INFO] Checking coordinator configuration in application database: fastorder_identity_sau_main_dev_db...
[WARN] ⚠️  Coordinator registered as 'localhost' in application database, fixing...
[INFO] Configuring coordinator hostname in application database: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432
[OK]   ✅ Coordinator hostname set to db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432 in application database
[INFO] Validating coordinator configuration before worker registration...
[OK]   ✅ Coordinator hostname validated: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[OK]   ✅ citus_tables view is accessible
[INFO] Checking coordinator self-registration...
[OK]   ✅ Coordinator is already self-registered
[INFO] Configuring coordinator shard placement policy...
[OK]   ✅ Coordinator already configured in postgres database (shouldhaveshards = false)
[WARN] ⚠️  Coordinator has 17 shards in fastorder_identity_sau_main_dev_db - cannot set shouldhaveshards=false
[WARN]    You must rebalance shards to workers first, then run this setup again
[WARN]    Skipping shouldhaveshards configuration for application database
[INFO] Registering 1 worker(s) to Citus cluster...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PRE-FLIGHT: Checking worker availability...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking worker worker-01...
[INFO]   FQDN: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[OK]   ✅ Worker worker-01 is reachable via SSL
[OK]   All workers are reachable - proceeding with registration

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Adding Citus worker: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com:5432
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Adding citus_cert_map to worker-01 pg_ident.conf...
[OK]   pg_ident.conf updated for worker-01
[INFO] Configuring worker worker-01 HBA for coordinator (10.100.1.214) access...
[OK]   Worker worker-01 HBA configured for coordinator (10.100.1.214)
[INFO] Adding replication rules for 3 standby(s)...
[OK]   Replication rules already exist for worker-01
[INFO] Reloading worker worker-01 to apply HBA changes...
[2026-01-03 07:54:40 UTC] USER=www-data EUID=0 PID=3024333 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
[INFO] Configuring coordinator HBA for worker worker-01 (10.100.1.215) access...
[OK]   Coordinator HBA configured for worker worker-01 (10.100.1.215)
[INFO] Reloading coordinator to apply HBA changes...
[2026-01-03 07:54:40 UTC] USER=www-data EUID=0 PID=3024363 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-coordinator.service
[INFO] Ensuring postgres client certificates exist for worker-01...
[OK]   Postgres client certificates already exist for worker-01
[INFO] Configuring citus.node_conninfo on worker-01...
[2026-01-03 07:54:40 UTC] USER=www-data EUID=0 PID=3024385 ACTION=passthru ARGS=systemctl reload postgresql@identity-sau-main-dev-worker-01.service
[OK]   citus.node_conninfo configured on worker-01
[INFO] Temporarily relaxing sync-rep on worker worker-01...
t
[OK]   Worker worker-01 sync-rep relaxed (was: sync_commit=on)
[INFO] Ensuring Citus extension on worker databases...
CREATE EXTENSION
CREATE EXTENSION
[INFO] Running citus_add_node with 180s timeout...
NOTICE:  shards are still on the coordinator after adding the new node
HINT:  Use SELECT rebalance_table_shards(); to balance shards data between workers and coordinator or SELECT citus_drain_node('db-identity-sau-main-dev-postgresql-coordinator.fastorder.com',5432); to permanently move shards away from the coordinator.
2
[INFO] Restoring worker worker-01 sync-rep settings...
t
[OK]   Worker worker-01 sync-rep restored
[OK]   ✅ Worker db-identity-sau-main-dev-postgresql-worker-01.fastorder.com successfully added to Citus cluster
[INFO]    Node ID: 2
[INFO]    Registered in: postgres, fastorder_identity_sau_main_dev_db
[OK]   Worker worker-01 registration successful
[INFO] Configuring worker worker-01 shard placement policy...
[OK]   ✅ Worker worker-01 configured to hold shards in all databases


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] POST-REGISTRATION: Verifying cluster state...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Expected workers: 1
[INFO] Registered workers: 1
[OK]   ✅ All 1 workers successfully registered!

[INFO] Citus cluster configuration:
db-identity-sau-main-dev-postgresql-coordinator.fastorder.com  5432  0  t  primary  f
db-identity-sau-main-dev-postgresql-worker-01.fastorder.com    5432  1  t  primary  t

[INFO] Note: groupid=0 is the coordinator, groupid>0 are workers
[INFO]       shouldhaveshards: false=query router only, true=holds data shards

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] FINAL VALIDATION: Verifying configuration persistence...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:54:45 UTC] USER=www-data EUID=0 PID=3024576 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /var/lib/postgresql/17/identity-sau-main-dev/coordinator/postgresql.conf
[OK]   ✅ citus.local_hostname persisted in postgresql.conf
[OK]   ✅ All 1 worker(s) successfully registered and verified

[OK]   ✅ All validation checks passed
[OK]   Citus coordinator setup complete

[OK]   Citus setup complete for coordinator
[INFO] ═══════════════════════════════════════════════════════════════════════════════

✓ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ CITUS CLUSTER SETUP COMPLETED SUCCESSFULLY
✓    Coordinator: Ready and accepting connections
✓    Workers registered: 1
✓ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 05-backup-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up coordinator backup...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for identity-sau-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-01-03 07:54:47 UTC] USER=www-data EUID=0 PID=3024639 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-03 07:54:47 UTC] USER=www-data EUID=0 PID=3024648 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/identity-sau-main-dev
[2026-01-03 07:54:47 UTC] USER=www-data EUID=0 PID=3024657 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-01-03 07:54:47 UTC] USER=www-data EUID=0 PID=3024666 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-03 07:54:47 UTC] USER=www-data EUID=0 PID=3024675 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-03 07:54:47 UTC] USER=www-data EUID=0 PID=3024685 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-03 07:54:54 UTC] USER=www-data EUID=0 PID=3024717 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-03 07:54:54 UTC] USER=www-data EUID=0 PID=3024726 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-03 07:54:54 UTC] USER=www-data EUID=0 PID=3024735 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-01-03 07:54:54 UTC] USER=www-data EUID=0 PID=3024744 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-03 07:54:54 UTC] USER=www-data EUID=0 PID=3024753 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-identity-sau-main-dev
[2026-01-03 07:54:55 UTC] USER=www-data EUID=0 PID=3024774 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-03 07:54:55 UTC] USER=www-data EUID=0 PID=3024784 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-01-03 07:54:55 UTC] USER=www-data EUID=0 PID=3024793 ACTION=fsop ARGS=find /var/lib/postgresql/17/identity-sau-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-01-03 07:54:55 UTC] USER=www-data EUID=0 PID=3024802 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-03 07:54:55 UTC] USER=www-data EUID=0 PID=3024811 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-01-03 07:54:55 UTC] USER=www-data EUID=0 PID=3024820 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-01-03 07:54:55 UTC] USER=www-data EUID=0 PID=3024829 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-03 07:54:55 UTC] USER=www-data EUID=0 PID=3024838 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/PG_VERSION
[2026-01-03 07:54:55 UTC] USER=www-data EUID=0 PID=3024848 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza identity-sau-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-03 07:54:55 UTC] USER=www-data EUID=0 PID=3024903 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-03 07:54:58 UTC] USER=www-data EUID=0 PID=3024920 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[2026-01-03 07:55:02 UTC] USER=www-data EUID=0 PID=3025245 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-03 07:55:02 UTC] USER=www-data EUID=0 PID=3025285 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator --log-level-console=info check
2026-01-03 07:55:02.679 P00   INFO: check command begin 2.56.0: --exec-id=3025293-a9ca4abb --log-level-console=info --log-level-file=debug --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-03 07:55:02.741 P00   INFO: check repo1 configuration (primary)
2026-01-03 07:55:02.762 P00  ERROR: [028]: backup and archive info files exist but do not match the database
                                    HINT: is this the correct stanza?
                                    HINT: did an error occur during stanza-upgrade?
2026-01-03 07:55:02.762 P00   INFO: check command end: aborted with exception [028]
[WARN] ⚠️  Stanza verification failed - this may be normal if WAL archiving hasn't started yet
[WARN]    The backup system is configured and will work once WAL segments are generated

[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-03 07:55:02 UTC] USER=www-data EUID=0 PID=3025309 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-03 07:55:02 UTC] USER=www-data EUID=0 PID=3025319 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-03 07:55:02 UTC] USER=www-data EUID=0 PID=3025338 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[2026-01-03 07:55:03 UTC] USER=www-data EUID=0 PID=3025347 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-03 07:55:03 UTC] USER=www-data EUID=0 PID=3025375 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-identity-sau-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-01-03 07:55:03 UTC] USER=www-data EUID=0 PID=3025403 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-03 07:55:03 UTC] USER=www-data EUID=0 PID=3025414 ACTION=fsop ARGS=sed -i s|__ENV_ID__|identity-sau-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-03 07:55:03 UTC] USER=www-data EUID=0 PID=3025425 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/var/lib/postgresql/17/identity-sau-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-03 07:55:03 UTC] USER=www-data EUID=0 PID=3025434 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-03 07:55:03 UTC] USER=www-data EUID=0 PID=3025443 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-03 07:55:03.746 P00   INFO: start command begin 2.56.0: --exec-id=3025469-6dde8873 --log-level-console=info --log-level-file=debug --stanza=identity-sau-main-dev-coordinator
2026-01-03 07:55:03.746 P00   WARN: stop file does not exist for stanza identity-sau-main-dev-coordinator
2026-01-03 07:55:03.746 P00   INFO: start command end: completed successfully (6ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-03 07:55:03.817 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=3025481-180810ae --log-level-console=info --log-level-file=debug --no-online --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-03 07:55:03.818 P00   INFO: stanza-upgrade for stanza 'identity-sau-main-dev-coordinator' on repo1
2026-01-03 07:55:03.846 P00   INFO: stanza-upgrade command end: completed successfully (35ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-03 07:55:03 UTC] USER=www-data EUID=0 PID=3025487 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260103-075503.log
[2026-01-03 07:55:03 UTC] USER=www-data EUID=0 PID=3025496 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260103-075503.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-01-03 07:55:14 UTC] USER=www-data EUID=0 PID=3025676 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-3024610.log /var/log/pgbackrest/initial-backup-20260103-075503.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260103-075503.log
   2026-01-03 07:55:14.227 P00   INFO: repo1: remove expired backup 20260102-110513F
   2026-01-03 07:55:14.308 P00   INFO: repo1: 17-26 remove archive, start = 000000010000000000000004, stop = 000000010000000000000006
   2026-01-03 07:55:14.309 P00   INFO: repo1: 17-27 no archive to remove
   2026-01-03 07:55:14.310 P00   INFO: repo1: 17-28 remove archive, start = 000000010000000000000003, stop = 000000010000000000000003
   2026-01-03 07:55:14.310 P00   INFO: expire command end: completed successfully (104ms)

[INFO] Current backups:
stanza: identity-sau-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000007/000000010000000000000012

        full backup: 20260102-110544F
            timestamp start/stop: 2026-01-02 11:05:44+00 / 2026-01-02 11:05:48+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.3MB, database backup size: 37.3MB
            repo1: backup set size: 5.8MB, backup size: 5.8MB

    db (prior)
        wal archive min/max (17): 000000010000000000000004/0000000100000001000000CF

        full backup: 20260102-115406F
            timestamp start/stop: 2026-01-02 11:54:06+00 / 2026-01-02 11:54:16+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.4MB, database backup size: 37.4MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

        full backup: 20260102-115435F
            timestamp start/stop: 2026-01-02 11:54:35+00 / 2026-01-02 11:54:39+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.4MB, database backup size: 37.4MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

        diff backup: 20260102-115435F_20260103-030851D
            timestamp start/stop: 2026-01-03 03:08:51+00 / 2026-01-03 03:10:52+00
            wal start/stop: 0000000100000000000000A4 / 0000000100000000000000A5
            database size: 37.5MB, database backup size: 8.3MB
            repo1: backup set size: 5.7MB, backup size: 1.7MB
            backup reference total: 1 full

    db (current)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000004

        full backup: 20260103-075504F
            timestamp start/stop: 2026-01-03 07:55:04+00 / 2026-01-03 07:55:14+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         identity-sau-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/identity-sau-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up worker backups for 1 worker(s)...
[INFO] Setting up backup for: worker-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for identity-sau-main-dev...

[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO]    Version: pgBackRest 2.56.0

[INFO] 2️⃣ Creating backup directories...
[2026-01-03 07:55:15 UTC] USER=www-data EUID=0 PID=3025743 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-03 07:55:15 UTC] USER=www-data EUID=0 PID=3025752 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/identity-sau-main-dev
[2026-01-03 07:55:15 UTC] USER=www-data EUID=0 PID=3025770 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-03 07:55:15 UTC] USER=www-data EUID=0 PID=3025779 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-03 07:55:15 UTC] USER=www-data EUID=0 PID=3025788 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-03 07:55:23 UTC] USER=www-data EUID=0 PID=3025878 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-03 07:55:23 UTC] USER=www-data EUID=0 PID=3025887 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-03 07:55:23 UTC] USER=www-data EUID=0 PID=3025896 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-01-03 07:55:23 UTC] USER=www-data EUID=0 PID=3025905 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/identity-sau-main-dev
[2026-01-03 07:55:24 UTC] USER=www-data EUID=0 PID=3025921 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO] ✅ Backup directories created

[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-identity-sau-main-dev
[2026-01-03 07:55:24 UTC] USER=www-data EUID=0 PID=3025942 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-03 07:55:24 UTC] USER=www-data EUID=0 PID=3025951 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key

[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-01-03 07:55:24 UTC] USER=www-data EUID=0 PID=3025960 ACTION=fsop ARGS=find /var/lib/postgresql/17/identity-sau-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-01-03 07:55:24 UTC] USER=www-data EUID=0 PID=3025969 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed

[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-03 07:55:24 UTC] USER=www-data EUID=0 PID=3025978 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-01-03 07:55:24 UTC] USER=www-data EUID=0 PID=3025987 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-01-03 07:55:24 UTC] USER=www-data EUID=0 PID=3025998 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created

[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-03 07:55:24 UTC] USER=www-data EUID=0 PID=3026010 ACTION=passthru ARGS=sudo -u postgres test -f /var/lib/postgresql/17/identity-sau-main-dev/coordinator/PG_VERSION
[2026-01-03 07:55:24 UTC] USER=www-data EUID=0 PID=3026020 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running

[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza identity-sau-main-dev-coordinator already initialized and verified

[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

[INFO] ✅ WAL archiving configured for coordinator

[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-03 07:55:25 UTC] USER=www-data EUID=0 PID=3026091 ACTION=passthru ARGS=systemctl stop postgresql@identity-sau-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-03 07:55:28 UTC] USER=www-data EUID=0 PID=3026129 ACTION=passthru ARGS=systemctl start postgresql@identity-sau-main-dev-coordinator.service
[2026-01-03 07:55:32 UTC] USER=www-data EUID=0 PID=3026206 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@identity-sau-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled

[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-03 07:55:32 UTC] USER=www-data EUID=0 PID=3026231 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator --log-level-console=info check
2026-01-03 07:55:32.358 P00   INFO: check command begin 2.56.0: --exec-id=3026240-fd382465 --log-level-console=info --log-level-file=debug --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-03 07:55:32.397 P00   INFO: check repo1 configuration (primary)
2026-01-03 07:55:32.456 P00   INFO: check repo1 archive for WAL (primary)
2026-01-03 07:55:32.757 P00   INFO: WAL segment 000000010000000000000006 successfully archived to '/var/lib/pgbackrest/backup/identity-sau-main-dev/archive/identity-sau-main-dev-coordinator/17-28/0000000100000000/000000010000000000000006-c37e8ac857363835604bfe298fcb017ce434556b.lz4' on repo1
2026-01-03 07:55:32.757 P00   INFO: check command end: completed successfully (405ms)
[INFO] ✅ Stanza verification passed

[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-03 07:55:32 UTC] USER=www-data EUID=0 PID=3026272 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-03 07:55:32 UTC] USER=www-data EUID=0 PID=3026281 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[2026-01-03 07:55:32 UTC] USER=www-data EUID=0 PID=3026299 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[2026-01-03 07:55:32 UTC] USER=www-data EUID=0 PID=3026308 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[INFO] ✅ Backup scripts created

[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-03 07:55:33 UTC] USER=www-data EUID=0 PID=3026327 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-identity-sau-main-dev
[INFO] ✅ Cron jobs configured
[INFO]    Schedule:
[INFO]    - Full backup:         Sundays at 2:00 AM
[INFO]    - Differential backup: Mon-Sat at 2:00 AM

[INFO] 🔟 Creating restore documentation...
[2026-01-03 07:55:33 UTC] USER=www-data EUID=0 PID=3026345 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|identity-sau-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-03 07:55:33 UTC] USER=www-data EUID=0 PID=3026354 ACTION=fsop ARGS=sed -i s|__ENV_ID__|identity-sau-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-03 07:55:33 UTC] USER=www-data EUID=0 PID=3026363 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/var/lib/postgresql/17/identity-sau-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-03 07:55:33 UTC] USER=www-data EUID=0 PID=3026372 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-03 07:55:33 UTC] USER=www-data EUID=0 PID=3026383 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-03 07:55:33.396 P00   INFO: start command begin 2.56.0: --exec-id=3026409-41491424 --log-level-console=info --log-level-file=debug --stanza=identity-sau-main-dev-coordinator
2026-01-03 07:55:33.397 P00   WARN: stop file does not exist for stanza identity-sau-main-dev-coordinator
2026-01-03 07:55:33.397 P00   INFO: start command end: completed successfully (5ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-03 07:55:33.458 P00   INFO: stanza-upgrade command begin 2.56.0: --exec-id=3026420-f935af77 --log-level-console=info --log-level-file=debug --no-online --pg1-path=/var/lib/postgresql/17/identity-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-identity-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/identity-sau-main-dev --stanza=identity-sau-main-dev-coordinator
2026-01-03 07:55:33.469 P00   INFO: stanza-upgrade for stanza 'identity-sau-main-dev-coordinator' on repo1
2026-01-03 07:55:33.474 P00   INFO: stanza 'identity-sau-main-dev-coordinator' on repo1 is already up to date
2026-01-03 07:55:33.474 P00   INFO: stanza-upgrade command end: completed successfully (22ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-03 07:55:33 UTC] USER=www-data EUID=0 PID=3026434 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260103-075533.log
[2026-01-03 07:55:33 UTC] USER=www-data EUID=0 PID=3026443 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260103-075533.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-01-03 07:55:39 UTC] USER=www-data EUID=0 PID=3026530 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-3025705.log /var/log/pgbackrest/initial-backup-20260103-075533.log
[INFO] ✅ Initial full backup completed successfully
[INFO]    Log: /var/log/pgbackrest/initial-backup-20260103-075533.log
   2026-01-03 07:55:39.139 P00   INFO: repo1: remove expired backup 20260102-110544F
   2026-01-03 07:55:39.217 P00   INFO: repo1: remove archive path /var/lib/pgbackrest/backup/identity-sau-main-dev/archive/identity-sau-main-dev-coordinator/17-26
   2026-01-03 07:55:39.231 P00   INFO: repo1: 17-27 no archive to remove
   2026-01-03 07:55:39.232 P00   INFO: repo1: 17-28 no archive to remove
   2026-01-03 07:55:39.232 P00   INFO: expire command end: completed successfully (104ms)

[INFO] Current backups:
stanza: identity-sau-main-dev-coordinator
    status: ok
    cipher: aes-256-cbc

    db (prior)
        wal archive min/max (17): 000000010000000000000004/0000000100000001000000CF

        full backup: 20260102-115406F
            timestamp start/stop: 2026-01-02 11:54:06+00 / 2026-01-02 11:54:16+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.4MB, database backup size: 37.4MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

        full backup: 20260102-115435F
            timestamp start/stop: 2026-01-02 11:54:35+00 / 2026-01-02 11:54:39+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.4MB, database backup size: 37.4MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

        diff backup: 20260102-115435F_20260103-030851D
            timestamp start/stop: 2026-01-03 03:08:51+00 / 2026-01-03 03:10:52+00
            wal start/stop: 0000000100000000000000A4 / 0000000100000000000000A5
            database size: 37.5MB, database backup size: 8.3MB
            repo1: backup set size: 5.7MB, backup size: 1.7MB
            backup reference total: 1 full

    db (current)
        wal archive min/max (17): 000000010000000000000004/000000010000000000000007

        full backup: 20260103-075504F
            timestamp start/stop: 2026-01-03 07:55:04+00 / 2026-01-03 07:55:14+00
            wal start/stop: 000000010000000000000004 / 000000010000000000000004
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

        full backup: 20260103-075533F
            timestamp start/stop: 2026-01-03 07:55:33+00 / 2026-01-03 07:55:39+00
            wal start/stop: 000000010000000000000007 / 000000010000000000000007
            database size: 37.5MB, database backup size: 37.5MB
            repo1: backup set size: 5.7MB, backup size: 5.7MB

[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️  No worker identifier provided - skipping worker backup setup
[INFO]    (Run with 'worker-01', 'worker-02', etc. to configure worker backups)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ Completed steps:
[INFO]   1. pgBackRest installed and configured
[INFO]   2. WAL archiving enabled (archive_mode=on)
[INFO]   3. PostgreSQL restarted with new settings
[INFO]   4. pgBackRest stanza initialized and verified
[INFO]   5. Initial full backup completed
[INFO]   6. Automated backup cron jobs configured

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO]   Coordinator:
[INFO]     Stanza:         identity-sau-main-dev-coordinator
[INFO]     Schedule:       Full: Sun 2AM, Diff: Mon-Sat 2AM

[INFO]   Common:
[INFO]     Backup dir:     /var/lib/pgbackrest/backup/identity-sau-main-dev
[INFO]     Archive dir:    /var/lib/pgbackrest/archive/identity-sau-main-dev
[INFO]     Config:         /etc/pgbackrest/pgbackrest.conf
[INFO]     Restore guide:  /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO]   Retention:
[INFO]     Full backups:       4 (keep last 4 full backups)
[INFO]     Differential:       4 (keep last 4 diff per full)
[INFO]     Archive WAL:        Auto-managed by pgBackRest

[INFO]   Manual commands:
[INFO]     Coordinator:        sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator backup
[INFO]     List all backups:   sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ Backup setup completed for coordinator and all workers

[INFO] Skipping 06-distribute-tables-canary.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 07-distribute-tables.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:55:41 UTC] USER=unknown EUID=33 PID=3026595 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-03 07:55:41 UTC] USER=unknown EUID=33 PID=3026602 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-03 07:55:41 UTC] USER=unknown EUID=33 PID=3026609 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-03 07:55:41 UTC] USER=unknown EUID=33 PID=3026616 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] CITUS TABLE DISTRIBUTION
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🔐 Secure connection established
[INFO]    Host: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432
[INFO]    Database: fastorder_identity_sau_main_dev_db
[INFO]    SSL: verify-full (TLS 1.2+)
[INFO]    Timeouts: statement=120s, idle_tx=300s

[INFO] 🔍 Running preflight checks...
[INFO] Testing database connectivity...
[OK]   ✅ Database connection successful
[OK]   ✅ Connected to correct database: fastorder_identity_sau_main_dev_db
[INFO] Checking Citus extension in database fastorder_identity_sau_main_dev_db...
[OK]   Citus version: 13.2-1
[INFO] Checking worker registration...
[OK]   Registered workers: 1
[INFO] Worker nodes:
[INFO]                             nodename                           | nodeport | isactive | noderole 
[INFO]   -------------------------------------------------------------+----------+----------+----------
[INFO]    db-identity-sau-main-dev-postgresql-worker-01.fastorder.com |     5432 | t        | primary
[INFO]   (1 row)
[INFO]   

[INFO] 📊 Starting table distribution...

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Distributing: auth.login_account
[INFO] Description: User authentication table - distributed by region for tenant isolation
[INFO] Shard key: region_hint
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📝 Current rows: 0
[INFO] Checking constraints compatibility with Citus...
[OK]   ✅ No conflicting constraints found
[OK]   ✅ Table already distributed - skipping
[INFO]    Distribution column: region_hint
[OK]   ✅ Data integrity verified (0 rows)
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[OK]   ✅ All tables distributed successfully!
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 📊 Citus Cluster Summary:

[INFO] Distributed tables:
[INFO]            table          |   type    | shard_key | shards | size  
[INFO]   ------------------------+-----------+-----------+--------+-------
[INFO]    core.tenant            | reference | <none>    |      1 | 24 kB
[INFO]    core.realm             | local     | <none>    |      1 | 40 kB
[INFO]    core.identity          | local     | <none>    |      1 | 72 kB
[INFO]    core.device            | local     | <none>    |      1 | 48 kB
[INFO]    core.identity_account  | local     | <none>    |      1 | 48 kB
[INFO]    core.identity_mfa      | local     | <none>    |      1 | 40 kB
[INFO]    core.external_idp_link | local     | <none>    |      1 | 48 kB
[INFO]    policy.client          | local     | <none>    |      1 | 56 kB
[INFO]    policy.resource        | local     | <none>    |      1 | 48 kB
[INFO]    policy.scope           | local     | <none>    |      1 | 40 kB
[INFO]    policy.permission      | local     | <none>    |      1 | 48 kB
[INFO]    policy.role            | local     | <none>    |      1 | 56 kB
[INFO]    policy.role_permission | local     | <none>    |      1 | 24 kB
[INFO]    policy.identity_role   | local     | <none>    |      1 | 40 kB
[INFO]    policy.policy_rule     | local     | <none>    |      1 | 48 kB
[INFO]    policy.api_key         | local     | <none>    |      1 | 56 kB
[INFO]    auth.login_account     | reference | <none>    |      1 | 48 kB
[INFO]   (17 rows)
[INFO]   

[INFO] Worker capacity:
[INFO]    worker | total_shards | total_size 
[INFO]   --------+--------------+------------
[INFO]   (0 rows)
[INFO]   

[OK]   Citus table distribution complete

[INFO] Skipping 08-distribute-tables-rollback.sh (rollback script - run manually only)
[INFO] Skipping 09-distribute-tables-test.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 10-setup-cdc.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] CDC PIPELINE SETUP (Debezium + Elasticsearch Sink)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Log file: /var/log/fastorder/cdc/10-setup-cdc-*.log

[INFO] Running CDC setup for identifier: coordinator
[2026-01-03 07:55:51] ==========================================
[2026-01-03 07:55:51] CDC SETUP SCRIPT STARTED
[2026-01-03 07:55:51] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260103_075551.log
[2026-01-03 07:55:51] ==========================================
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[2026-01-03 07:55:52] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:55:52]   CDC Pipeline Setup (Debezium + ES Sink)
[2026-01-03 07:55:52] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:55:52]   Environment: identity-sau-main-dev
[2026-01-03 07:55:52]   Identifier:  coordinator
[2026-01-03 07:55:52]   Service:     identity
[2026-01-03 07:55:52] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:55:52] 📂 CDC_BASE_DIR exists: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc
[2026-01-03 07:55:52] Looking for service folder: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity
[2026-01-03 07:55:52] 
[2026-01-03 07:55:52] 📂 Found CDC configuration for service: identity
[2026-01-03 07:55:52] Scanning for subservice directories in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity
[2026-01-03 07:55:52] Found subservice: login, checking for steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps
[2026-01-03 07:55:52] 
[2026-01-03 07:55:52] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:55:52]   Setting up CDC for: identity/login
[2026-01-03 07:55:52] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-03 07:55:52] Found 7 step script(s) in /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps
[2026-01-03 07:55:52] 
[2026-01-03 07:55:52] 🔧 Running: 01-setup-debezium-auth-login.sh
[2026-01-03 07:55:52]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/01-setup-debezium-auth-login.sh
[2026-01-03 07:55:52]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Debezium CDC Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔍 Verifying Kafka infrastructure...
✅ db-identity-sau-main-dev-postgresql.fastorder.com resolves to 10.100.1.214
🔐 psql will use client cert for mTLS.
🔐 Retrieving credentials from secrets vault...
   Clearing cached credentials for coordinator...
✅ Credentials retrieved from secrets vault
🔐 Syncing debezium_user password in PostgreSQL...
✅ debezium_user password synchronized
🔍 Checking PostgreSQL SSL status...
✅ Server SSL is ON (verify-full + client cert).
🔧 Applying publication & grants over TLS…
ALTER SYSTEM
 pg_reload_conf 
----------------
 t
(1 row)

NOTICE:  publication "cdc_pub_identity" does not exist, skipping
DROP PUBLICATION
CREATE PUBLICATION
SET
NOTICE:  Added shard table auth.login_account_102024 to publication
DO
RESET
GRANT
GRANT
GRANT
✅ Publication & grants done (including Citus shard table).
⏳ Waiting for Kafka Connect @ https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors…
[2026-01-03 07:55:57] 🔗 Waiting for Kafka Connect at: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
[2026-01-03 07:55:57] ⏳ Waiting for HTTP endpoint: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
[2026-01-03 07:55:57]    Expected codes: 200,500, timeout: 300s
[2026-01-03 07:55:57] ✅ HTTP endpoint ready: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083 (code: 200, took: 0s)
[2026-01-03 07:55:57] 🔄 Testing Connect worker readiness...
[2026-01-03 07:55:57] ✅ Kafka Connect worker ready
🧹 Cleaning up existing Debezium connector and slot (if any)...
   Step 0a: Also resetting ES Sink connector offsets (required for coordinated reset)...
   → Stopping ES Sink connector pg_identity_sau_main_dev_coordinator_es_sink...
   → Deleting ES Sink connector offsets...
   ✓ ES Sink offsets deleted successfully (HTTP 200)
   → Deleting ES Sink connector (will be recreated by 02-setup-es-sink.sh)...
   ✓ ES Sink connector cleanup complete
   Step 0b: Clearing stale Debezium connector offsets from Kafka Connect...
   → Stopping connector pg_identity_sau_main_dev_debezium_postgres...
   → Deleting connector offsets (forces fresh snapshot)...
   ✓ Connector offsets deleted successfully (HTTP 200)
   Step 1: Ensuring connector is completely removed...
   Deleting connector: pg_identity_sau_main_dev_debezium_postgres (attempt 1/10)
   ✓ Connector pg_identity_sau_main_dev_debezium_postgres does not exist (HTTP 404)
   Step 2: Waiting for replication slot to become inactive...
   ✓ Slot slot_identity_sau_main_dev does not exist (clean state)
   Step 3: Dropping replication slot...
   ✓ Slot slot_identity_sau_main_dev already dropped
   Step 4: Final verification...
✅ Cleanup complete - environment is clean for fresh CDC snapshot
🔐 Checking Debezium SSL certificate permissions...
🔍 Validating Debezium SSL certificates...
🔐 Connector will use mTLS to Postgres.
  ✓ Certificate: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt
  ✓ Key: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key
  ✓ Root CA: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
ℹ️  Skipping pre-flight connectivity test (will be validated by Kafka Connect)
📤 Upserting connector: PUT https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_identity_sau_main_dev_debezium_postgres/config
   Attempt 1/5: Sending PUT request to Kafka Connect...
   (This may take up to 60s as Connect validates the configuration)
   ✅ Success (HTTP 201)

🌐 HTTP Response: 201
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Response body:
{
  "name": "pg_identity_sau_main_dev_debezium_postgres",
  "config": {
    "name": "pg_identity_sau_main_dev_debezium_postgres",
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "plugin.name": "pgoutput",
    "database.hostname": "db-identity-sau-main-dev-postgresql.fastorder.com",
    "database.port": "5432",
    "database.dbname": "fastorder_identity_sau_main_dev_db",
    "database.user": "debezium_user",
    "database.password": "tZqovvck2N0DRKeV7xws7MjaX",
    "database.sslmode": "verify-full",
    "database.sslrootcert": "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt",
    "database.sslcert": "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user.crt",
    "database.sslkey": "/etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/debezium_user_der.key",
    "publication.name": "cdc_pub_identity",
    "publication.autocreate.mode": "disabled",
    "slot.name": "slot_identity_sau_main_dev",
    "topic.prefix": "identity_sau_main_dev_cdc",
    "schema.include.list": "auth",
    "table.include.list": "auth.login_account,auth.login_account_[0-9]+",
    "transforms": "unwrap,route",
    "transforms.unwrap.add.fields": "op,ts_ms",
    "transforms.unwrap.delete.handling.mode": "rewrite",
    "transforms.unwrap.drop.tombstones": "false",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
    "transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.route.regex": "^identity_sau_main_dev_cdc\\.auth\\.login_account(_[0-9]+)?$",
    "transforms.route.replacement": "identity_sau_main_dev_account_router",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",
    "snapshot.mode": "always"
  },
  "tasks": [],
  "type": "source"
}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Connector upserted.
🔄 Verifying connector task startup...
✅ Debezium connector task is RUNNING
ℹ️  Source table auth.login_account has 0 rows.
ℹ️  Snapshot will be metadata-only; offsets may stay empty until first change.
⏳ Waiting for Debezium initial snapshot to complete...
   📊 Slot status: restart_lsn=0/8017FB8, confirmed_flush_lsn=0/8017FF0
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (0s elapsed)
   ⏳ Snapshot in progress... (5s elapsed)
   ⏳ Snapshot in progress... (10s elapsed)
   📊 Slot status: restart_lsn=0/8017FB8, confirmed_flush_lsn=0/8017FF0
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (15s elapsed)
   ⏳ Snapshot in progress... (20s elapsed)
   ⏳ Snapshot in progress... (25s elapsed)
   📊 Slot status: restart_lsn=0/8017FB8, confirmed_flush_lsn=0/8017FF0
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (30s elapsed)
   ⏳ Snapshot in progress... (35s elapsed)
   ⏳ Snapshot in progress... (40s elapsed)
   📊 Slot status: restart_lsn=0/8017FB8, confirmed_flush_lsn=0/8017FF0
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (45s elapsed)
   ⏳ Snapshot in progress... (50s elapsed)
   ⏳ Snapshot in progress... (55s elapsed)
   📊 Slot status: restart_lsn=0/8017FB8, confirmed_flush_lsn=0/8017FF0
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (60s elapsed)
   ⏳ Snapshot in progress... (65s elapsed)
   ⏳ Snapshot in progress... (70s elapsed)
   📊 Slot status: restart_lsn=0/8017FB8, confirmed_flush_lsn=0/8017FF0
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (75s elapsed)
   ⏳ Snapshot in progress... (80s elapsed)
   ⏳ Snapshot in progress... (85s elapsed)
   📊 Slot status: restart_lsn=0/8017FB8, confirmed_flush_lsn=0/8017FF0
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (90s elapsed)
   ⏳ Snapshot in progress... (95s elapsed)
   ⏳ Snapshot in progress... (100s elapsed)
   📊 Slot status: restart_lsn=0/8017FB8, confirmed_flush_lsn=0/8017FF0
   📊 Debezium snapshot status: unknown
   📊 Slot LSN advancing (activity detected, awaiting snapshot_completed)
   ⏳ Snapshot in progress... (105s elapsed)
   ⏳ Snapshot in progress... (110s elapsed)
   ⏳ Snapshot in progress... (115s elapsed)

⚠️  WARNING: Snapshot wait timeout (120s) on EMPTY table.
   Offsets are still empty, but source table has 0 rows.
   Proceeding anyway – CDC health will be verified by test inserts.

✅ Debezium connector is RUNNING after snapshot
🔍 Final verification: Checking Debezium offsets are recorded...
   ℹ️  Source table auth.login_account has 0 rows
   ℹ️  Skipping offset verification (no data to snapshot)
✅ Debezium connector verified RUNNING (empty source table)
🔄 Phase 2: Updating connector to snapshot.mode=initial...
✅ Connector updated to snapshot.mode=initial (HTTP 200)
✅ Connector verified RUNNING after Phase 2 update
✅ Debezium connector configured successfully (two-phase snapshot complete)
[2026-01-03 07:58:30] ✅ Completed: 01-setup-debezium-auth-login.sh
[2026-01-03 07:58:30] 
[2026-01-03 07:58:30] 🔧 Running: 02-setup-es-sink.sh
[2026-01-03 07:58:30]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/02-setup-es-sink.sh
[2026-01-03 07:58:30]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[WARN] Master/coordinator not found, using node-01
[INFO] Using ES domain: search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com
🔐 Retrieving keystore passwords from secrets manager...
[INFO] Retrieving Kafka truststore password...
✅ Retrieved passwords from remote backend
✅ Retrieved Kafka truststore password
[INFO] Retrieving Elasticsearch P12 password...
[INFO] 🔍 Checking secrets backend (provider: aws)...
✅ Retrieved passwords from remote backend
[INFO] ✅ Using existing passwords from backend
✅ Retrieved/generated Elasticsearch P12 password
✅ Keystore passwords retrieved successfully
   - Kafka truststore password: yOb0eqkA... (32 chars)
   - ES P12 password: wrFpsVuc... (32 chars)
[INFO] 🔐 Clearing cached ES credentials to ensure fresh retrieval...
[INFO] [INFO] ✅ Using ES password from centralized secrets vault (identifier: node-01)
[INFO] 🔐 Verifying Elasticsearch accepts client certificate...
[INFO] ✅ Elasticsearch accepting client certificate
[INFO] 🔐 Setting up ES client keystore using Kafka client certificate...
[INFO]    Certificate: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem (signed by Fastorder RA Root CA)
[INFO] 📋 Creating ES client P12 keystore from Kafka client certificate...
[2026-01-03 07:58:39 UTC] USER=www-data EUID=0 PID=3030607 ACTION=fsop ARGS=mv /tmp/es-client-3030355.p12 /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[2026-01-03 07:58:39 UTC] USER=www-data EUID=0 PID=3030618 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[2026-01-03 07:58:39 UTC] USER=www-data EUID=0 PID=3030628 ACTION=fsop ARGS=chmod 600 /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[INFO] ✅ Created ES client keystore: /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[INFO]    Using Kafka client cert signed by Fastorder RA Root CA
[INFO] ℹ️  Using Kafka truststore and adding ES CA certificate
[2026-01-03 07:58:39 UTC] USER=www-data EUID=0 PID=3030637 ACTION=fsop ARGS=test -f /opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks
[INFO] 📋 Adding ES CA certificate to truststore...
[2026-01-03 07:58:40 UTC] USER=www-data EUID=0 PID=3030694 ACTION=passthru ARGS=sudo -u kafka keytool -import -alias elasticsearch-ca -file /etc/elasticsearch/identity-sau-main-dev/node-01/certs/http_ca.crt -keystore /opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks -storepass yOb0eqkAqtj8HEWebgA7nf04YlqsLw44 -noprompt
Certificate was added to keystore
[INFO] ✅ ES CA added to truststore
[INFO] [INFO] 🔗 Waiting for Kafka Connect at: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
[INFO] [INFO] ✅ Connect HTTP ready (code 200)
[INFO] [INFO] 🔍 Verifying Debezium connector snapshot status...
[INFO] [INFO] ℹ️  Source table auth.login_account has 0 rows.
[INFO] [INFO]    Skipping Debezium snapshot wait (metadata-only snapshot on empty table).
[INFO] [INFO] 🔌 Cleaning up existing ES Sink connector: pg_identity_sau_main_dev_coordinator_es_sink
[INFO] [INFO]    → Deleting connector...
[INFO] [INFO]    HTTP 404 (404 is fine)
[INFO] [INFO] 🔐 Validating Elasticsearch credentials...
[INFO] [INFO] ✅ ES credentials validated successfully
[INFO] [INFO] 🔧 Creating required Elasticsearch ingest pipelines: identity-embed-pipeline-001
[INFO] [INFO] ✅ Pipeline identity-embed-pipeline-001 created successfully
[INFO] [INFO] 🔧 Ensuring CDC index has no default_pipeline requirement...
[INFO] [INFO] ✅ Removed default_pipeline from index (if any)
[INFO] [INFO] 🔧 Ensuring dynamic mapping is enabled...
[INFO] [INFO] ✅ Dynamic mapping enabled for identity_sau_main_dev_account_router
[DEBUG] ES_TRUSTSTORE=/opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks
[DEBUG] ES_CLIENT_P12=/opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[DEBUG] TRUSTSTORE_PASS=yOb0eqkA...
[DEBUG] P12_PASS=wrFpsVuc...
== Outgoing connector config (snippet) ==
2:  "name": "pg_identity_sau_main_dev_coordinator_es_sink",
6:  "connection.url": "https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200",
19:  "index": "identity_sau_main_dev_account_router",
[INFO] ⚠️  Skipping pre-validation - will validate on PUT...
[INFO] [INFO] ✅ Proceeding to PUT
[2026-01-03 07:58:42] [1/3] Upserting connector via PUT https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_identity_sau_main_dev_coordinator_es_sink/config
🌐 HTTP 201
✅ Connector created/updated successfully
{
  "name": "pg_identity_sau_main_dev_coordinator_es_sink",
  "config": {
    "name": "pg_identity_sau_main_dev_coordinator_es_sink",
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "tasks.max": "1",
    "topics": "identity_sau_main_dev_account_router",
    "connection.url": "https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200",
    "elastic.security.protocol": "SSL",
    "elastic.https.ssl.hostname.verification": "true",
    "elastic.https.ssl.truststore.location": "/opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks",
    "elastic.https.ssl.truststore.password": "yOb0eqkAqtj8HEWebgA7nf04YlqsLw44",
    "elastic.https.ssl.truststore.type": "JKS",
    "elastic.https.ssl.keystore.location": "/opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12",
    "elastic.https.ssl.keystore.password": "wrFpsVuculAyox6t6keqTSfFFpwsJiaI",
    "elastic.https.ssl.keystore.type": "PKCS12",
    "elastic.username": "elastic",
    "elastic.password": "JgZeXRo9EN2a7bqSBJrm",
    "connection.username": "elastic",
    "connection.password": "JgZeXRo9EN2a7bqSBJrm",
    "index": "identity_sau_main_dev_account_router",
    "key.ignore": "true",
    "schema.ignore": "true",
    "behavior.on.null.values": "delete",
    "write.method": "upsert",
    "type.name": "_doc",
    "max.in.flight.requests": "1",
    "batch.size": "2000",
    "linger.ms": "100",
    "flush.timeout.ms": "60000",
    "max.retries": "10",
    "retry.backoff.ms": "5000",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false"
  },
  "tasks": [],
  "type": "sink"
}
{
  "pg_identity_sau_main_dev_debezium_postgres": {
    "status": {
      "name": "pg_identity_sau_main_dev_debezium_postgres",
      "connector": {
        "state": "RUNNING",
        "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
      },
      "tasks": [
        {
          "id": 0,
          "state": "RUNNING",
          "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
        }
      ],
      "type": "source"
    }
  },
  "pg_identity_sau_to_universe_main_dev_es_sink": {
    "status": {
      "name": "pg_identity_sau_to_universe_main_dev_es_sink",
      "connector": {
        "state": "RUNNING",
        "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
      },
      "tasks": [
        {
          "id": 0,
          "state": "FAILED",
          "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083",
          "trace": "org.apache.kafka.common.KafkaException: Failed to load SSL keystore /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12 of type PKCS12\n\tat org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:380)\n\tat org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:352)\n\tat org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:302)\n\tat org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:162)\n\tat org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:147)\n\tat org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:100)\n\tat io.confluent.connect.elasticsearch.ConfigCallbackHandler.sslContext(ConfigCallbackHandler.java:262)\n\tat io.confluent.connect.elasticsearch.ConfigCallbackHandler.createConnectionManager(ConfigCallbackHandler.java:172)\n\tat io.confluent.connect.elasticsearch.ConfigCallbackHandler.customizeHttpClient(ConfigCallbackHandler.java:95)\n\tat org.elasticsearch.client.RestClientBuilder.createHttpClient(RestClientBuilder.java:320)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.client.RestClientBuilder.build(RestClientBuilder.java:283)\n\tat io.confluent.connect.elasticsearch.ElasticsearchClient.<init>(ElasticsearchClient.java:144)\n\tat io.confluent.connect.elasticsearch.ElasticsearchSinkTask.start(ElasticsearchSinkTask.java:82)\n\tat io.confluent.connect.elasticsearch.ElasticsearchSinkTask.start(ElasticsearchSinkTask.java:54)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:324)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doStart(WorkerTask.java:176)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:225)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:281)\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:238)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\nCaused by: java.nio.file.NoSuchFileException: /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12\n\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\tat java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)\n\tat java.base/java.nio.file.Files.newByteChannel(Files.java:380)\n\tat java.base/java.nio.file.Files.newByteChannel(Files.java:432)\n\tat java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)\n\tat java.base/java.nio.file.Files.newInputStream(Files.java:160)\n\tat org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:373)\n\t... 24 more\n"
        }
      ],
      "type": "sink"
    }
  },
  "pg_identity_sau_main_dev_coordinator_es_sink": {
    "status": {
      "name": "pg_identity_sau_main_dev_coordinator_es_sink",
      "connector": {
        "state": "RUNNING",
        "worker_id": "eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083"
      },
      "tasks": [],
      "type": "sink"
    }
  }
}
[INFO] [INFO] 🔗 Creating ES alias for application compatibility...
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (0s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (5s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (10s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (15s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (20s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (25s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (30s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (35s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (40s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (45s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (50s)
[INFO] [INFO]    ⏳ Waiting for ES index to be created... (55s)
[WARN] ⚠️  ES index not created within 60s, skipping alias creation

🔍 Final verification: Checking ES document count...
   PostgreSQL auth.login_account: 0 rows
ℹ️  PostgreSQL table is empty - skipping ES verification
✅ Done.
[2026-01-03 07:59:45] ✅ Completed: 02-setup-es-sink.sh
[2026-01-03 07:59:45] 
[2026-01-03 07:59:45] 🔧 Running: 03-setup-es-universe-sink.sh
[2026-01-03 07:59:45]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/03-setup-es-universe-sink.sh
[2026-01-03 07:59:45]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Universe Identity ES Sink Setup (Dual-Sink Pattern)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Source Zone:  sau
  Connector:      pg_identity_sau_to_universe_main_dev_es_sink
  Source Topic:   identity_sau_main_dev_account_router
  Universe ES:      search-identity-universe-main-dev.fastorder.com:9200
  Universe Index:   identity_universe_main_dev_account_router
  Zone Field:   zone: "sau" (added to each document)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Retrieving keystore passwords from secrets manager...
✅ Retrieved passwords from remote backend
✅ Retrieved Kafka truststore password
[INFO] 🔍 Checking secrets backend (provider: aws)...
✅ Retrieved passwords from remote backend
[INFO] ✅ Using existing passwords from backend
✅ Retrieved/generated Elasticsearch P12 password
🔐 Retrieving Universe ES password...
[INFO] [INFO] ✅ Retrieved Universe ES password from vault (identifier: node-01)
[INFO] 🔐 Setting up ES client keystore using Kafka client certificate...
[INFO]    Certificate: /var/www/ssl/kafka/identity-sau-main-dev/client-cert.pem (signed by Fastorder RA Root CA)
[2026-01-03 07:59:53 UTC] USER=www-data EUID=0 PID=3032190 ACTION=fsop ARGS=mv /tmp/es-client-3031977.p12 /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[2026-01-03 07:59:53 UTC] USER=www-data EUID=0 PID=3032199 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[2026-01-03 07:59:53 UTC] USER=www-data EUID=0 PID=3032208 ACTION=fsop ARGS=chmod 600 /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[INFO] ✅ Created ES client keystore: /opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12
[INFO] 🔐 Checking Universe ES CA in truststore...
[INFO] 📋 Adding Universe ES CA certificate to truststore...
[2026-01-03 07:59:55 UTC] USER=www-data EUID=0 PID=3032286 ACTION=keytool ARGS=-importcert -trustcacerts -alias identity-universe-es-ca -file /etc/elasticsearch/identity-universe-main-dev/node-01/certs/http_ca.crt -keystore /opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks -storepass yOb0eqkAqtj8HEWebgA7nf04YlqsLw44 -noprompt
Certificate was added to keystore
[INFO] ✅ Universe ES CA added to truststore via wrapper
[INFO] 🔄 Restarting Kafka Connect to load updated truststore...
[2026-01-03 07:59:56 UTC] USER=www-data EUID=0 PID=3032330 ACTION=passthru ARGS=systemctl restart confluent-connect-identity-sau-main-dev_coordinator.service
[INFO] ✅ Kafka Connect restarted: confluent-connect-identity-sau-main-dev_coordinator.service
[INFO]    Waiting for Kafka Connect to be ready (up to 120s)...
[INFO] ✅ Kafka Connect is ready after 18x2 seconds
[INFO] [INFO] ✅ Using unified PKI (Kafka client cert) for Universe ES mTLS
🔍 Verifying Universe ES connectivity...
✅ Universe ES cluster is reachable
   Cluster: fastorder-identity-universe-main-dev, Status: green
👤 Ensuring app_user exists on Universe ES...
[2026-01-03 08:00:35 UTC] USER=www-data EUID=0 PID=3040768 ACTION=passthru ARGS=grep -q ^app_user: /etc/elasticsearch/identity-universe-main-dev/node-01/users
✅ app_user already exists on Universe ES
🔧 Creating Universe ES index if not exists...
✅ Universe index already exists: identity_universe_main_dev_account_router
[INFO] 🔧 Ensuring index settings are compatible with CDC...
[INFO]    → Removing default_pipeline from index...
[INFO]    → Enabling dynamic mapping...
[INFO] ✅ Index settings updated for CDC compatibility
[INFO] [INFO] 🔗 Waiting for Kafka Connect at: https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083
[INFO] [INFO] ✅ Connect HTTP ready (code 200)
[INFO] [INFO] 🔌 Cleaning up existing Universe ES Sink connector: pg_identity_sau_to_universe_main_dev_es_sink
[INFO] [INFO]    → Stopping connector pg_identity_sau_to_universe_main_dev_es_sink...
[INFO] [INFO]    → Deleting connector offsets...
[INFO] [INFO]    → Deleting connector...

📤 Creating Universe Identity ES Sink connector...
   PUT https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_identity_sau_to_universe_main_dev_es_sink/config

   Transform: zone = "sau" (added to every document)
[1/3] Upserting connector via PUT https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_identity_sau_to_universe_main_dev_es_sink/config
🌐 HTTP 201
✅ Connector created/updated successfully
{
  "name": "pg_identity_sau_to_universe_main_dev_es_sink",
  "config": {
    "name": "pg_identity_sau_to_universe_main_dev_es_sink",
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "tasks.max": "1",
    "topics": "identity_sau_main_dev_account_router",
    "connection.url": "https://search-identity-universe-main-dev.fastorder.com:9200",
    "connection.username": "elastic",
    "connection.password": "h+yhmN0YCeA_Vt7epnWZ",
    "elastic.security.protocol": "SSL",
    "elastic.https.ssl.truststore.location": "/opt/kafka/secrets/identity-sau-main-dev/coordinator/truststore.jks",
    "elastic.https.ssl.truststore.password": "yOb0eqkAqtj8HEWebgA7nf04YlqsLw44",
    "elastic.https.ssl.truststore.type": "JKS",
    "elastic.https.ssl.keystore.location": "/opt/kafka/secrets/identity-sau-main-dev/coordinator/es-client.keystore.p12",
    "elastic.https.ssl.keystore.password": "wrFpsVuculAyox6t6keqTSfFFpwsJiaI",
    "elastic.https.ssl.keystore.type": "PKCS12",
    "key.ignore": "true",
    "schema.ignore": "true",
    "behavior.on.null.values": "delete",
    "write.method": "upsert",
    "type.name": "_doc",
    "max.in.flight.requests": "1",
    "batch.size": "2000",
    "linger.ms": "100",
    "flush.timeout.ms": "60000",
    "flush.synchronously": "true",
    "behavior.on.malformed.documents": "warn",
    "drop.invalid.message": "true",
    "max.retries": "10",
    "retry.backoff.ms": "5000",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",
    "transforms": "routeTopic,addZone,addEnv",
    "transforms.routeTopic.type": "org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.routeTopic.regex": "identity_sau_main_dev_account_router",
    "transforms.routeTopic.replacement": "identity_universe_main_dev_account_router",
    "transforms.addZone.type": "org.apache.kafka.connect.transforms.InsertField$Value",
    "transforms.addZone.static.field": "zone",
    "transforms.addZone.static.value": "sau",
    "transforms.addEnv.type": "org.apache.kafka.connect.transforms.InsertField$Value",
    "transforms.addEnv.static.field": "env",
    "transforms.addEnv.static.value": "dev"
  },
  "tasks": [],
  "type": "sink"
}

🔍 Verifying connector status...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Universe Identity ES Sink Status
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Connector: RUNNING
  Task:      RUNNING
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Universe Identity ES Sink is running

🔍 Checking Universe ES index document count...
   Universe index identity_universe_main_dev_account_router: 0 documents
   Documents from zone 'sau': 0
ℹ️  No documents yet - may take a moment for initial sync
   Run 10-test-universe-identity-index.sh to verify end-to-end

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  DUAL-SINK PATTERN SETUP COMPLETE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Connector:    pg_identity_sau_to_universe_main_dev_es_sink
  Source Topic: identity_sau_main_dev_account_router
  Universe Index: identity_universe_main_dev_account_router
  Zone Field: zone: "sau"

  Query example (filter by zone):
  GET identity_universe_main_dev_account_router/_search
  { "query": { "term": { "zone": "sau" } } }
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

✅ Done.
[2026-01-03 08:01:02] ✅ Completed: 03-setup-es-universe-sink.sh
[2026-01-03 08:01:02] 
[2026-01-03 08:01:02] 🔧 Running: 04-test-cdc-pipelines.sh
[2026-01-03 08:01:02]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/04-test-cdc-pipelines.sh
[2026-01-03 08:01:02]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
ℹ️  Using test identifier with hostname: coordinator-web-03
ℹ️  Citus: Data will be stored on coordinator
ℹ️  Test data identifier: coordinator
   - Username: cdc_user_coordinator_1767427270
   - This reflects the actual Citus storage node in Elasticsearch
→ Inserting into Postgres (auth.login_account)
   INSERT INTO auth.login_account (id, email, username, password_hash, status, region_hint, created_at, updated_at)
   VALUES ('019b82df-e6ce-7045-a500-8c548a62fb8a', 'cdc_test_coordinator_1767427270@example.com', 'cdc_user_coordinator_1767427270', crypt('testpass123', gen_salt('bf')), 'active', 'sau', NOW(), NOW())
   RETURNING id, email, username, created_at;
                  id                  |                    email                    |            username             |          created_at           
--------------------------------------+---------------------------------------------+---------------------------------+-------------------------------
 019b82df-e6ce-7045-a500-8c548a62fb8a | cdc_test_coordinator_1767427270@example.com | cdc_user_coordinator_1767427270 | 2026-01-03 08:01:10.657126+00
(1 row)

INSERT 0 1
✅ Inserted test record: cdc_test_coordinator_1767427270@example.com
ℹ️  Citus placement: Shard 102024 on db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432
→ Waiting for CDC to propagate to Elasticsearch (index: identity_sau_main_dev_account_router, max 45s)

   Polling... elapsed: 0s/45s
   Polling... elapsed: 3s/45s
   ✅ Document found after 3s!                    
→ Final search: https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/identity_sau_main_dev_account_router/_search?q=email:cdc_test_coordinator_1767427270@example.com
🎉 SUCCESS: Document is indexed in 'identity_sau_main_dev_account_router'
📄 Indexed Document (source):
{
  "mysql_id": null,
  "__ts_ms": 1767427270851,
  "updated_at": "2026-01-03T08:01:10.657126Z",
  "password_hash": "$2a$06$bg1xCbifKYBroezS1zvyFeAa5MnverN6CTwcvKy9OA/7wDZaGeLGy",
  "created_at": "2026-01-03T08:01:10.657126Z",
  "id": "019b82df-e6ce-7045-a500-8c548a62fb8a",
  "region_hint": "sau",
  "__op": "c",
  "email": "cdc_test_coordinator_1767427270@example.com",
  "username": "cdc_user_coordinator_1767427270",
  "status": "active"
}

📊 Elasticsearch Indices:
   health status index                                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
   green  open   identity_sau_main_dev_account_router-000001 Hxn4Ie-5QfOF_XmkzCTlkg   1   0          3            0     19.4kb         19.4kb       19.4kb
   
   [es-http] code=200 url=https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cat/indices?v time=0.070471s
[2026-01-03 08:01:14] ✅ Completed: 04-test-cdc-pipelines.sh
[2026-01-03 08:01:14] 
[2026-01-03 08:01:14] 🔧 Running: 05-verify-cdc-publication.sh
[2026-01-03 08:01:14]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/05-verify-cdc-publication.sh
[2026-01-03 08:01:14]    Executing directly (script is executable)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Verifying CDC Publication & Replication Slot
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔍 Checking CDC configuration...

1️⃣  Checking publication: cdc_pub_authn
   ⚠️  Publication does not exist - creating...
CREATE PUBLICATION
   ✅ Publication created successfully (with publish_via_partition_root = true)
 schemaname |   tablename   
------------+---------------
 auth       | login_account
(1 row)


2️⃣  Checking replication slot: slot_authn_sau_main_dev
   ⚠️  Replication slot does not exist
   ℹ️  This will be created automatically by Debezium when it connects

3️⃣  Checking table replica identity for CDC
   ℹ️  Replica identity: DEFAULT (only primary key changes)

   💡 CDC requires replica identity FULL for complete change capture
   🔧 Setting replica identity to FULL automatically...
ALTER TABLE
   ✅ Replica identity set to FULL

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ CDC Publication Verification Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📊 Summary:
  • Publication: ✅ Created
  • Replication Slot: ⚠️  Will be created by Debezium
  • Replica Identity: f

🔗 Next Steps:
  1. Check Kafka Connect status: curl -s https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_identity_sau_main_dev_debezium_postgres/status | jq
  2. Monitor Kafka topic: Check /var/lib/kafka/authN-af-aaaa1-dev*/authN_af_aaaa1_dev_account_router*/
  3. Verify Elasticsearch: Check the dashboard for real-time updates

[2026-01-03 08:01:17] ✅ Completed: 05-verify-cdc-publication.sh
[2026-01-03 08:01:17] 
[2026-01-03 08:01:17] 🔧 Running: 06-verify-cdc.sh
[2026-01-03 08:01:17]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/06-verify-cdc.sh
[2026-01-03 08:01:17]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  CDC Pipeline Verification
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔧 STEP 1: Testing CDC pipeline...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
ℹ️  Using test identifier with hostname: coordinator-web-03
ℹ️  Citus: Data will be stored on coordinator
ℹ️  Test data identifier: coordinator
   - Username: cdc_user_coordinator_1767427284
   - This reflects the actual Citus storage node in Elasticsearch
→ Inserting into Postgres (auth.login_account)
   INSERT INTO auth.login_account (id, email, username, password_hash, status, region_hint, created_at, updated_at)
   VALUES ('019b82e0-1ce7-7086-9d00-57272b4e8482', 'cdc_test_coordinator_1767427284@example.com', 'cdc_user_coordinator_1767427284', crypt('testpass123', gen_salt('bf')), 'active', 'sau', NOW(), NOW())
   RETURNING id, email, username, created_at;
                  id                  |                    email                    |            username             |          created_at           
--------------------------------------+---------------------------------------------+---------------------------------+-------------------------------
 019b82e0-1ce7-7086-9d00-57272b4e8482 | cdc_test_coordinator_1767427284@example.com | cdc_user_coordinator_1767427284 | 2026-01-03 08:01:24.502244+00
(1 row)

INSERT 0 1
✅ Inserted test record: cdc_test_coordinator_1767427284@example.com
ℹ️  Citus placement: Shard 102024 on db-identity-sau-main-dev-postgresql-coordinator.fastorder.com:5432
→ Waiting for CDC to propagate to Elasticsearch (index: identity_sau_main_dev_account_router, max 45s)

   Polling... elapsed: 0s/45s
   Polling... elapsed: 3s/45s
   ✅ Document found after 3s!                    
→ Final search: https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/identity_sau_main_dev_account_router/_search?q=email:cdc_test_coordinator_1767427284@example.com
🎉 SUCCESS: Document is indexed in 'identity_sau_main_dev_account_router'
📄 Indexed Document (source):
{
  "mysql_id": null,
  "__ts_ms": 1767427284571,
  "updated_at": "2026-01-03T08:01:24.502244Z",
  "password_hash": "$2a$06$kTd.loVRX4vXP8.DJKVWv.mfPa59NQeVcESnoemKzyDV2QjTReIOe",
  "created_at": "2026-01-03T08:01:24.502244Z",
  "id": "019b82e0-1ce7-7086-9d00-57272b4e8482",
  "region_hint": "sau",
  "__op": "c",
  "email": "cdc_test_coordinator_1767427284@example.com",
  "username": "cdc_user_coordinator_1767427284",
  "status": "active"
}

📊 Elasticsearch Indices:
   health status index                                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
   green  open   identity_sau_main_dev_account_router-000001 Hxn4Ie-5QfOF_XmkzCTlkg   1   0          4            0     28.5kb         28.5kb       28.5kb
   
   [es-http] code=200 url=https://search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cat/indices?v time=0.030687s

🔧 STEP 2: Verifying CDC publication...
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Verifying CDC Publication & Replication Slot
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment: identity-sau-main-dev
  Identifier:  coordinator
  Database:    fastorder_identity_sau_main_dev_db
  Host:        db-identity-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔍 Checking CDC configuration...

1️⃣  Checking publication: cdc_pub_authn
   ✅ Publication exists
   ✅ publish_via_partition_root = true (required for Citus)
   📋 Tables in publication:
 schemaname |   tablename   
------------+---------------
 auth       | login_account
(1 row)


2️⃣  Checking replication slot: slot_authn_sau_main_dev
   ⚠️  Replication slot does not exist
   ℹ️  This will be created automatically by Debezium when it connects

3️⃣  Checking table replica identity for CDC
   ✅ Replica identity: FULL (all column changes tracked)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ CDC Publication Verification Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📊 Summary:
  • Publication: ✅ Exists
  • Replication Slot: ⚠️  Will be created by Debezium
  • Replica Identity: f

🔗 Next Steps:
  1. Check Kafka Connect status: curl -s https://eventbus-identity-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_identity_sau_main_dev_debezium_postgres/status | jq
  2. Monitor Kafka topic: Check /var/lib/kafka/authN-af-aaaa1-dev*/authN_af_aaaa1_dev_account_router*/
  3. Verify Elasticsearch: Check the dashboard for real-time updates


✅ CDC Pipeline verification complete
[2026-01-03 08:01:31] ✅ Completed: 06-verify-cdc.sh
[2026-01-03 08:01:31] 
[2026-01-03 08:01:31] 🔧 Running: 07-test-universe-identity-index.sh
[2026-01-03 08:01:31]    Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/identity/login/steps/07-test-universe-identity-index.sh
[2026-01-03 08:01:31]    Executing directly (script is executable)
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
  Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
  Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
             Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
  Provider: aws
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Universe Identity Index Health Test
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Environment:     identity-sau-main-dev
  PostgreSQL:      db-identity-sau-main-dev-postgresql.fastorder.com:5432
  Zonal ES:     search-identity-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
  Universe ES:       search-identity-universe-main-dev-elasticsearch-coordinator.fastorder.com:9200
  Zonal Index:  identity_sau_main_dev_account_router
  Universe Index:    identity_universe_main_dev_account_router
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔐 Retrieving credentials...
❌ Could not retrieve PostgreSQL password
[2026-01-03 08:01:55] ❌ FAILED: 07-test-universe-identity-index.sh (exit code: 1)
[2026-01-03 08:01:55] ⚠️  Continuing with next step despite failure (non-critical step)...
[2026-01-03 08:01:55] 
[2026-01-03 08:01:55] ==========================================
[2026-01-03 08:01:55] ✅ CDC Pipeline setup complete for 1 subservice(s)
[2026-01-03 08:01:55] CDC SETUP SCRIPT FINISHED
[2026-01-03 08:01:55] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260103_075551.log
[2026-01-03 08:01:55] ==========================================
✓ ✅ CDC Pipeline setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 11-monitoring-setup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up monitoring for coordinator...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 PostgreSQL Monitoring Integration for identity-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[OK]   ✓ Observability cell is ready

[INFO] ✓ Using private IP for metrics: 10.100.1.214
[INFO] 2️⃣ Setting up postgres_exporter integration...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[INFO] Setting up postgres_exporter for identity-sau-main-dev
[2026-01-03 08:01:57 UTC] USER=www-data EUID=0 PID=3042800 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-identity-sau-main-dev.yaml /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[2026-01-03 08:01:57 UTC] USER=www-data EUID=0 PID=3042809 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[2026-01-03 08:01:57 UTC] USER=www-data EUID=0 PID=3042818 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[OK]   Custom queries file created at /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[2026-01-03 08:01:58 UTC] USER=www-data EUID=0 PID=3042828 ACTION=passthru ARGS=mv /tmp/postgres_exporter-identity-sau-main-dev.service /etc/systemd/system/postgres_exporter-identity-sau-main-dev.service
[2026-01-03 08:01:58 UTC] USER=www-data EUID=0 PID=3042837 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-03 08:01:59 UTC] USER=www-data EUID=0 PID=3042894 ACTION=passthru ARGS=systemctl enable postgres_exporter-identity-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/postgres_exporter-identity-sau-main-dev.service -> /etc/systemd/system/postgres_exporter-identity-sau-main-dev.service.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  IP Conflict Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: identity-sau-main-dev
IP Address:  10.100.1.214
Port:        9187
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔍 Checking IP conflict for identity-sau-main-dev on 10.100.1.214:9187...
✅ IP 10.100.1.214:9187 is available - no conflicts detected

🔍 Checking for orphaned processes that might conflict...
✅ No orphaned processes detected

✅ All checks passed - safe to proceed with identity-sau-main-dev setup
[2026-01-03 08:02:00 UTC] USER=www-data EUID=0 PID=3042999 ACTION=passthru ARGS=systemctl restart postgres_exporter-identity-sau-main-dev.service
[OK]   postgres_exporter configured on db-identity-sau-main-dev-postgresql.fastorder.com:9187
[INFO] Adding PostgreSQL scrape target to Prometheus config...
[OK]   PostgreSQL scrape target added
[INFO] Creating PostgreSQL alert rules...
[2026-01-03 08:02:02 UTC] USER=www-data EUID=0 PID=3043076 ACTION=fsop ARGS=mv /tmp/postgresql_alerts_identity-sau-main-dev.yml /etc/prometheus/obs-identity-sau-main-dev/rules/postgresql_alerts.yml
[OK]   PostgreSQL alert rules created: /etc/prometheus/obs-identity-sau-main-dev/rules/postgresql_alerts.yml
[INFO] Adding PostgreSQL alerts to Prometheus config...
[2026-01-03 08:02:02 UTC] USER=www-data EUID=0 PID=3043086 ACTION=fsop ARGS=sed -i /rule_files:/a\  - "rules/postgresql_alerts.yml" /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[OK]   PostgreSQL alerts registered in Prometheus
[2026-01-03 08:02:02 UTC] USER=www-data EUID=0 PID=3043096 ACTION=passthru ARGS=systemctl reload prometheus-obs-identity-sau-main-dev.service
Failed to reload prometheus-obs-identity-sau-main-dev.service: Job type reload is not applicable for unit prometheus-obs-identity-sau-main-dev.service.
[2026-01-03 08:02:02 UTC] USER=www-data EUID=0 PID=3043105 ACTION=passthru ARGS=systemctl restart prometheus-obs-identity-sau-main-dev.service
[OK]   Prometheus reloaded with PostgreSQL monitoring
[OK]   ✓ postgres_exporter integration complete
[INFO] Registering postgres_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: postgres_exporter -> 10.100.1.214:9187
[OK]   ✓ Registered postgres_exporter scrape target: 10.100.1.214:9187
[INFO]   Target file: /etc/prometheus/obs-identity-sau-main-dev/targets/postgres_exporter.yml
[OK]   ✓ postgres_exporter registered as Prometheus scrape target

[INFO] 3️⃣ Setting up pgbouncer_exporter integration...
[INFO] PgBouncer FQDN found in /etc/hosts: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.204
[INFO] PgBouncer detected: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com:6432
[OK]   ✓ pgbouncer_exporter already installed
[INFO] Getting pgbouncer_admin password (SERVICE=identity, ZONE=sau)
[OK]   ✓ pgbouncer_admin password retrieved (24 chars)
[INFO] Using pgbouncer certs from: /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[INFO] Creating pgbouncer_exporter systemd service...
[OK]   ✓ pgbouncer_exporter service file created
[INFO] Starting pgbouncer_exporter service...
[2026-01-03 08:02:06 UTC] USER=www-data EUID=0 PID=3043450 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-03 08:02:06 UTC] USER=www-data EUID=0 PID=3043507 ACTION=passthru ARGS=systemctl enable pgbouncer_exporter-identity-sau-main-dev.service
[2026-01-03 08:02:07 UTC] USER=www-data EUID=0 PID=3043562 ACTION=passthru ARGS=systemctl restart pgbouncer_exporter-identity-sau-main-dev.service
[WARN] ⚠️  pgbouncer_exporter service not running (may need manual start)
[WARN]     Run: systemctl status pgbouncer_exporter-identity-sau-main-dev.service

[INFO] 4️⃣ Registering nodes to monitoring database...
[INFO] PostgreSQL key permissions set for www-data access: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/postgres.key
[INFO] Registering PostgreSQL coordinator to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-coordinator
[INFO]   Identifier Parent: coordinator
[INFO]   IP:                10.100.1.214
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: ce097707-5ce5-40c8-a941-01512555cab8
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ PostgreSQL coordinator registered
[INFO] Registering PgBouncer to monitoring dashboard...
[INFO]   FQDN: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com, IP: 10.100.1.204, Port: 6432
[INFO]   Key permissions set for www-data access
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        identity-sau-main-dev-pgbouncer
[INFO]   Identifier Parent: pooling
[INFO]   IP:                10.100.1.204
[INFO]   Port:              6432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 426480a5-2f64-4fc0-b2b5-710f9ccb059a
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ PgBouncer registered

[INFO] 5️⃣ Creating PgBouncer professional monitoring rules...
[INFO] Creating PgBouncer recording rules...
[OK]   ✓ PgBouncer recording rules created
[INFO] Creating PgBouncer alert rules with runbook URLs...
[OK]   ✓ PgBouncer alert rules with runbook URLs created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ PostgreSQL & PgBouncer Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] postgres_exporter: http://localhost:9187/metrics
[INFO] pgbouncer_exporter: http://localhost:9127/metrics
[INFO] Prometheus: https://metrics-identity-sau-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-identity-sau-main-dev.fastorder.com
[INFO] 
[INFO] PgBouncer Monitoring:
[INFO]   • Recording rules: /etc/prometheus/obs-identity-sau-main-dev/rules/pgbouncer_recording_rules.yml
[INFO]   • Alert rules: /etc/prometheus/obs-identity-sau-main-dev/rules/pgbouncer_alerts.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up monitoring for 1 worker(s) and 1 standby(s) per worker...
[INFO] Setting up monitoring for: worker-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 PostgreSQL Monitoring Integration for identity-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[OK]   ✓ Observability cell is ready

[INFO] ✓ Using private IP for metrics: 10.100.1.214
[INFO] 2️⃣ Setting up postgres_exporter integration...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[INFO] Setting up postgres_exporter for identity-sau-main-dev
[2026-01-03 08:02:12 UTC] USER=www-data EUID=0 PID=3043780 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-identity-sau-main-dev.yaml /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[2026-01-03 08:02:12 UTC] USER=www-data EUID=0 PID=3043789 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[2026-01-03 08:02:12 UTC] USER=www-data EUID=0 PID=3043802 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[OK]   Custom queries file created at /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[OK]   postgres_exporter already running with custom queries for identity-sau-main-dev
[OK]   ✓ postgres_exporter integration complete
[INFO] Registering postgres_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: postgres_exporter -> 10.100.1.214:9187
[OK]   ✓ Registered postgres_exporter scrape target: 10.100.1.214:9187
[INFO]   Target file: /etc/prometheus/obs-identity-sau-main-dev/targets/postgres_exporter.yml
[OK]   ✓ postgres_exporter registered as Prometheus scrape target

[INFO] 3️⃣ Setting up pgbouncer_exporter integration...
[INFO] PgBouncer FQDN found in /etc/hosts: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.204
[INFO] PgBouncer detected: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com:6432
[OK]   ✓ pgbouncer_exporter already installed
[INFO] Getting pgbouncer_admin password (SERVICE=identity, ZONE=sau)
[OK]   ✓ pgbouncer_admin password retrieved (24 chars)
[INFO] Using pgbouncer certs from: /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[INFO] Creating pgbouncer_exporter systemd service...
[OK]   ✓ pgbouncer_exporter service file created
[INFO] Starting pgbouncer_exporter service...
[2026-01-03 08:02:14 UTC] USER=www-data EUID=0 PID=3043876 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-03 08:02:15 UTC] USER=www-data EUID=0 PID=3043932 ACTION=passthru ARGS=systemctl enable pgbouncer_exporter-identity-sau-main-dev.service
[2026-01-03 08:02:16 UTC] USER=www-data EUID=0 PID=3043979 ACTION=passthru ARGS=systemctl restart pgbouncer_exporter-identity-sau-main-dev.service
[WARN] ⚠️  pgbouncer_exporter service not running (may need manual start)
[WARN]     Run: systemctl status pgbouncer_exporter-identity-sau-main-dev.service

[INFO] 4️⃣ Registering nodes to monitoring database...
[INFO] PostgreSQL key permissions set for www-data access: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/postgres.key
[INFO] Registering PostgreSQL worker-01 to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-worker-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.214
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 2a8d7237-0c1b-4286-8ffc-cd46f4f7052e
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ PostgreSQL worker-01 registered
[INFO] Registering PgBouncer to monitoring dashboard...
[INFO]   FQDN: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com, IP: 10.100.1.204, Port: 6432
[INFO]   Key permissions set for www-data access
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        identity-sau-main-dev-pgbouncer
[INFO]   Identifier Parent: pooling
[INFO]   IP:                10.100.1.204
[INFO]   Port:              6432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 426480a5-2f64-4fc0-b2b5-710f9ccb059a
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ PgBouncer registered

[INFO] 5️⃣ Creating PgBouncer professional monitoring rules...
[INFO] Creating PgBouncer recording rules...
[OK]   ✓ PgBouncer recording rules created
[INFO] Creating PgBouncer alert rules with runbook URLs...
[OK]   ✓ PgBouncer alert rules with runbook URLs created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ PostgreSQL & PgBouncer Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] postgres_exporter: http://localhost:9187/metrics
[INFO] pgbouncer_exporter: http://localhost:9127/metrics
[INFO] Prometheus: https://metrics-identity-sau-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-identity-sau-main-dev.fastorder.com
[INFO] 
[INFO] PgBouncer Monitoring:
[INFO]   • Recording rules: /etc/prometheus/obs-identity-sau-main-dev/rules/pgbouncer_recording_rules.yml
[INFO]   • Alert rules: /etc/prometheus/obs-identity-sau-main-dev/rules/pgbouncer_alerts.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Setting up monitoring for standby: worker-01-standby-01
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
✓ [SECRETS] Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[SECRETS] Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[SECRETS]            Search (build_es_secret_name, get/set_es_credentials_to_vault)
[SECRETS]            Backups (build_backup_path)
[SECRETS] Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 PostgreSQL Monitoring Integration for identity-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[OK]   ✓ Observability cell is ready

[INFO] ✓ Using private IP for metrics: 10.100.1.214
[INFO] 2️⃣ Setting up postgres_exporter integration...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[INFO] Setting up postgres_exporter for identity-sau-main-dev
[2026-01-03 08:02:21 UTC] USER=www-data EUID=0 PID=3044199 ACTION=passthru ARGS=mv /tmp/postgres_exporter_queries-identity-sau-main-dev.yaml /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[2026-01-03 08:02:21 UTC] USER=www-data EUID=0 PID=3044210 ACTION=passthru ARGS=chown postgres:postgres /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[2026-01-03 08:02:21 UTC] USER=www-data EUID=0 PID=3044219 ACTION=passthru ARGS=chmod 640 /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[OK]   Custom queries file created at /etc/prometheus/postgres_exporter_queries-identity-sau-main-dev.yaml
[OK]   postgres_exporter already running with custom queries for identity-sau-main-dev
[OK]   ✓ postgres_exporter integration complete
[INFO] Registering postgres_exporter with Prometheus...
[INFO] Registering Prometheus scrape target: postgres_exporter -> 10.100.1.214:9187
[OK]   ✓ Registered postgres_exporter scrape target: 10.100.1.214:9187
[INFO]   Target file: /etc/prometheus/obs-identity-sau-main-dev/targets/postgres_exporter.yml
[OK]   ✓ postgres_exporter registered as Prometheus scrape target

[INFO] 3️⃣ Setting up pgbouncer_exporter integration...
[INFO] PgBouncer FQDN found in /etc/hosts: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.204
[INFO] PgBouncer detected: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com:6432
[OK]   ✓ pgbouncer_exporter already installed
[INFO] Getting pgbouncer_admin password (SERVICE=identity, ZONE=sau)
[OK]   ✓ pgbouncer_admin password retrieved (24 chars)
[INFO] Using pgbouncer certs from: /etc/fastorder/postgresql/certs/identity-sau-main-dev/pgbouncer
[INFO] Creating pgbouncer_exporter systemd service...
[OK]   ✓ pgbouncer_exporter service file created
[INFO] Starting pgbouncer_exporter service...
[2026-01-03 08:02:23 UTC] USER=www-data EUID=0 PID=3044286 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-03 08:02:24 UTC] USER=www-data EUID=0 PID=3044337 ACTION=passthru ARGS=systemctl enable pgbouncer_exporter-identity-sau-main-dev.service
[2026-01-03 08:02:25 UTC] USER=www-data EUID=0 PID=3044389 ACTION=passthru ARGS=systemctl restart pgbouncer_exporter-identity-sau-main-dev.service
[WARN] ⚠️  pgbouncer_exporter service not running (may need manual start)
[WARN]     Run: systemctl status pgbouncer_exporter-identity-sau-main-dev.service

[INFO] 4️⃣ Registering nodes to monitoring database...
[INFO] Registering PostgreSQL worker-01-standby-01 to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PostgreSQL
[INFO]   Identifier:        identity-sau-main-dev-postgresql-worker-01-standby-01
[INFO]   Identifier Parent: worker-01
[INFO]   IP:                10.100.1.214
[INFO]   Port:              5432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 8eaa8059-bede-4f71-ae1d-d26590a898da
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ PostgreSQL worker-01-standby-01 registered
[INFO] Registering PgBouncer to monitoring dashboard...
[INFO]   FQDN: db-identity-sau-main-dev-postgresql-bouncer.fastorder.com, IP: 10.100.1.204, Port: 6432
[INFO]   Key permissions set for www-data access
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       PgBouncer
[INFO]   Identifier:        identity-sau-main-dev-pgbouncer
[INFO]   Identifier Parent: pooling
[INFO]   IP:                10.100.1.204
[INFO]   Port:              6432
[INFO]   FQDN:              db-identity-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 426480a5-2f64-4fc0-b2b5-710f9ccb059a
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ PgBouncer registered

[INFO] 5️⃣ Creating PgBouncer professional monitoring rules...
[INFO] Creating PgBouncer recording rules...
[OK]   ✓ PgBouncer recording rules created
[INFO] Creating PgBouncer alert rules with runbook URLs...
[OK]   ✓ PgBouncer alert rules with runbook URLs created
[INFO] Reloading Prometheus configuration...
[WARN] ⚠️  Could not reload Prometheus (may need manual reload)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ PostgreSQL & PgBouncer Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] postgres_exporter: http://localhost:9187/metrics
[INFO] pgbouncer_exporter: http://localhost:9127/metrics
[INFO] Prometheus: https://metrics-identity-sau-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-identity-sau-main-dev.fastorder.com
[INFO] 
[INFO] PgBouncer Monitoring:
[INFO]   • Recording rules: /etc/prometheus/obs-identity-sau-main-dev/rules/pgbouncer_recording_rules.yml
[INFO]   • Alert rules: /etc/prometheus/obs-identity-sau-main-dev/rules/pgbouncer_alerts.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ ✅ Monitoring setup completed for coordinator, workers, and standbys

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 12-setup-offsite-backup.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Setting up offsite backup repository for identity-sau-main-dev...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Offsite Backup Repository Setup (repo2)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📋 OFFSITE BACKUP INFORMATION
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Why Offsite Backups?
[INFO]   ✓ Disaster recovery resilience (datacenter loss, hardware failure)
[INFO]   ✓ Protection against local corruption or ransomware
[INFO]   ✓ Compliance requirements (geographic redundancy)
[INFO]   ✓ Long-term archival with cost-effective storage tiers

[WARN] ⚠️  Offsite backup (repo2) is NOT ENABLED
[WARN]    Using local backups only (repo1)

[INFO] Configuration Example Location:
[INFO]   📄 /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example

[INFO] Supported Storage Backends:
[INFO]   • AWS S3 (standard, multi-region)
[INFO]   • AWS S3 Glacier (low-cost archival)
[INFO]   • MinIO (self-hosted S3-compatible)
[INFO]   • Google Cloud Storage (via S3 compatibility)
[INFO]   • Azure Blob Storage (via S3 compatibility)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📝 SETUP INSTRUCTIONS
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Step 1: Review the example configuration
[INFO]   cat /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example

[INFO] Step 2: Prepare S3 bucket and credentials
[INFO]   • Create S3 bucket (or MinIO bucket)
[INFO]   • Create IAM user with S3 permissions (PutObject, GetObject, DeleteObject, ListBucket)
[INFO]   • Note: Access Key ID and Secret Access Key

[INFO] Step 3: Add repo2 configuration to /etc/pgbackrest/pgbackrest.conf
[INFO]   • Copy repo2-* settings from example to [global] section
[INFO]   • Replace placeholders (bucket name, access keys, region)
[INFO]   • Note: Use same cipher key as repo1, or generate separate key for repo2

[INFO] Step 4: Initialize repo2 stanzas
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator stanza-create --repo=2
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-worker-01 stanza-create --repo=2
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-worker-02 stanza-create --repo=2

[INFO] Step 5: Verify repo2 configuration
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator check --repo=2

[INFO] Step 6: Take initial full backup to repo2
[INFO]   command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru sudo -u postgres pgbackrest --stanza=identity-sau-main-dev-coordinator --repo=2 --type=full backup

[INFO] Step 7: Update backup automation to include repo2
[INFO]   • Edit: /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
[INFO]   • Change: pgbackrest backup to pgbackrest --repo=1,2 backup
[INFO]   • Or: Add separate cron for repo2 backups

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🧪 TESTING
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] After configuration, run:
[INFO]   ./08-setup-offsite-backup.sh test

[INFO] This will verify:
[INFO]   ✓ S3 connectivity
[INFO]   ✓ Stanza initialization
[INFO]   ✓ Test backup and restore from repo2

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 💡 COST OPTIMIZATION
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] AWS S3 Lifecycle Policies (transition to cheaper storage):
[INFO]   • 0-30 days:   S3 Standard (~$0.023/GB/month)
[INFO]   • 30-90 days:  S3 Standard-IA (~$0.0125/GB/month)
[INFO]   • 90+ days:    S3 Glacier (~$0.004/GB/month)

[INFO] Estimated costs for 100GB backups:
[INFO]   • All Standard:     ~$2.30/month
[INFO]   • With lifecycle:   ~$1.20/month (48% savings)


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 13-setup-monitoring-alerts.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Setting up backup monitoring and alerting for identity-sau-main-dev...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Monitoring and Alerting Configuration
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] This will set up monitoring for:
  • Backup failures (cron job failures)
  • WAL archiving backlog (>100 files)
  • Repository disk space (<20% free)
  • Backup age (>25 hours)

[INFO] No alert email configured (set ALERT_EMAIL environment variable)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Creating monitoring directories...
[2026-01-03 08:02:34 UTC] USER=www-data EUID=0 PID=3044666 ACTION=fsop ARGS=mkdir -p /opt/pgbackrest-monitoring
[2026-01-03 08:02:34 UTC] USER=www-data EUID=0 PID=3044675 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest-monitoring
[2026-01-03 08:02:34 UTC] USER=www-data EUID=0 PID=3044684 ACTION=fsop ARGS=chmod 777 /opt/pgbackrest-monitoring
[2026-01-03 08:02:34 UTC] USER=www-data EUID=0 PID=3044693 ACTION=fsop ARGS=chmod 777 /var/log/pgbackrest-monitoring
[2026-01-03 08:02:34 UTC] USER=www-data EUID=0 PID=3044702 ACTION=fsop ARGS=chown postgres:postgres /opt/pgbackrest-monitoring
[2026-01-03 08:02:34 UTC] USER=www-data EUID=0 PID=3044711 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest-monitoring
[INFO] ✅ Directories created

[INFO] 2️⃣ Creating alert helper script...
[2026-01-03 08:02:34 UTC] USER=www-data EUID=0 PID=3044733 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/send-alert.sh
[INFO] ✅ Alert helper created

[INFO] 3️⃣ Creating WAL queue monitoring script...
[2026-01-03 08:02:35 UTC] USER=www-data EUID=0 PID=3044754 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-wal-queue.sh
[INFO] ✅ WAL queue monitor created

[INFO] 4️⃣ Creating backup age monitoring script...
[2026-01-03 08:02:35 UTC] USER=www-data EUID=0 PID=3044774 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-backup-age.sh
[INFO] ✅ Backup age monitor created

[INFO] 5️⃣ Creating repository disk space monitoring script...
[2026-01-03 08:02:35 UTC] USER=www-data EUID=0 PID=3044794 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-repo-space.sh
[INFO] ✅ Disk space monitor created

[INFO] 6️⃣ Creating backup failure detection script...
[2026-01-03 08:02:35 UTC] USER=www-data EUID=0 PID=3044814 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/check-backup-failures.sh
[INFO] ✅ Backup failure detector created

[INFO] 7️⃣ Creating master monitoring script...
[2026-01-03 08:02:35 UTC] USER=www-data EUID=0 PID=3044832 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO] ✅ Master monitoring script created

[INFO] 8️⃣ Installing mailutils for email alerts...
[INFO] ✅ mailutils already installed

[INFO] 9️⃣ Installing jq for JSON parsing...
[INFO] ✅ jq already installed

[INFO] 🔟 Setting up monitoring cron jobs...
[2026-01-03 08:02:35 UTC] USER=www-data EUID=0 PID=3044850 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-monitoring-identity-sau-main-dev
[INFO] ✅ Monitoring cron jobs configured
[INFO]    Checks run every 15 minutes

[INFO] 1️⃣1️⃣ Creating monitoring dashboard...
[2026-01-03 08:02:35 UTC] USER=www-data EUID=0 PID=3044870 ACTION=fsop ARGS=chmod 755 /opt/pgbackrest-monitoring/dashboard.sh
[INFO] ✅ Monitoring dashboard created

[INFO] 1️⃣2️⃣ Running initial monitoring check...

[2026-01-03 08:02:35 UTC] USER=www-data EUID=0 PID=3044879 ACTION=passthru ARGS=bash /opt/pgbackrest-monitoring/run-all-checks.sh
grep: write error: Broken pipe

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup monitoring setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Monitoring Configuration:
[INFO]   Alert Email:        
[INFO]   Slack Webhook:      Not configured

[INFO] Monitoring Checks:
[INFO]   • WAL Queue:        Every 15 minutes (threshold: >100 files)
[INFO]   • Backup Age:       Every 15 minutes (threshold: >25 hours)
[INFO]   • Disk Space:       Every 15 minutes (threshold: <20% free)
[INFO]   • Backup Failures:  Every 15 minutes (log analysis)

[INFO] Scripts Created:
[INFO]   Monitoring dir:     /opt/pgbackrest-monitoring
[INFO]   Log dir:            /var/log/pgbackrest-monitoring
[INFO]   Dashboard:          /opt/pgbackrest-monitoring/dashboard.sh
[INFO]   Master check:       /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO]   Alert sender:       /opt/pgbackrest-monitoring/send-alert.sh

[INFO] Useful Commands:
[INFO]   View dashboard:     /usr/local/bin/fastorder-provisioning-wrapper.sh /opt/pgbackrest-monitoring/dashboard.sh
[INFO]   Run checks now:     /usr/local/bin/fastorder-provisioning-wrapper.sh /opt/pgbackrest-monitoring/run-all-checks.sh
[INFO]   View alerts:        tail -f /var/log/pgbackrest-monitoring/alerts.log
[INFO]   View monitoring:    tail -f /var/log/pgbackrest-monitoring/monitoring.log

[INFO] Cron Schedule:
[INFO]   All checks:         Every 15 minutes
[INFO]   Log rotation:       Weekly (keep 7 days)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 14-vault-cipher-key.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ✅ Using permanent AWS credentials from /home/ab/.aws/credentials [default] profile
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔐 PostgreSQL Cipher Key Vaulting
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO]   Environment:       identity-sau-main-dev
[INFO]   AWS Region:        me-central-1
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣  Verifying AWS setup...
[INFO] ✅ AWS authentication successful

[INFO] 2️⃣  Verifying cipher key...
[INFO] ✅ Cipher key found
[INFO]    Location: /etc/pgbackrest/.cipher-key-identity-sau-main-dev
[INFO]    Hash (MD5): a2c56e403b3aaec22fcda5af6fc58831
[INFO]    Size: 194 bytes

[INFO] 3️⃣  Vaulting cipher key to AWS Secrets Manager...
[INFO]    Secret name: fastorder/db/identity/sau/main/dev/postgresql/pgbackrest/cipher-key
[INFO]    Secret exists, updating value...
[INFO] ✅ Cipher key updated in AWS Secrets Manager
[INFO]    Verifying storage...
[INFO] ✅ Verification successful - key matches

[INFO] 4️⃣  Creating local encrypted backup...
[2026-01-03 08:02:48 UTC] USER=www-data EUID=0 PID=3045248 ACTION=fsop ARGS=mv /tmp/cipher-key-backup-3045041.enc /root/.pgbackrest-cipher-key-identity-sau-main-dev.enc
[2026-01-03 08:02:48 UTC] USER=www-data EUID=0 PID=3045257 ACTION=fsop ARGS=chmod 600 /root/.pgbackrest-cipher-key-identity-sau-main-dev.enc
[2026-01-03 08:02:49 UTC] USER=www-data EUID=0 PID=3045276 ACTION=fsop ARGS=chmod 600 /root/.pgbackrest-cipher-key-passphrase-identity-sau-main-dev.txt
[INFO] ✅ Local encrypted backup created
[INFO]    Backup file: /root/.pgbackrest-cipher-key-identity-sau-main-dev.enc
[INFO]    Passphrase: /root/.pgbackrest-cipher-key-passphrase-identity-sau-main-dev.txt

[INFO] 5️⃣  Vaulting backup passphrase...
[INFO] ✅ Backup passphrase updated

[INFO] 6️⃣  Creating recovery documentation...
[2026-01-03 08:02:53 UTC] USER=www-data EUID=0 PID=3045350 ACTION=fsop ARGS=chmod 640 /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_identity-sau-main-dev.md
[2026-01-03 08:02:53 UTC] USER=www-data EUID=0 PID=3045359 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_identity-sau-main-dev.md
[INFO] ✅ Recovery documentation: /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_identity-sau-main-dev.md

[INFO] 7️⃣  Storing backup metadata...
[INFO] ✅ Backup metadata stored in AWS Secrets Manager
[INFO]    Secret: fastorder/db/identity/sau/main/dev/postgresql/backup/metadata-20260103-080253

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Cipher Key Vaulting Complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO]   Environment:          identity-sau-main-dev
[INFO]   Key Hash:             a2c56e403b3aaec22fcda5af6fc58831

[INFO] AWS Secrets:
[INFO]   Cipher Key:           fastorder/db/identity/sau/main/dev/postgresql/pgbackrest/cipher-key
[INFO]   Passphrase:           fastorder/db/identity/sau/main/dev/postgresql/pgbackrest/cipher-key-passphrase
[INFO]   Backup Metadata:      fastorder/db/identity/sau/main/dev/postgresql/backup/metadata-20260103-080253

[INFO] Local Backups:
[INFO]   Encrypted File:       /root/.pgbackrest-cipher-key-identity-sau-main-dev.enc
[INFO]   Passphrase File:      /root/.pgbackrest-cipher-key-passphrase-identity-sau-main-dev.txt

[INFO] Recovery Doc:           /var/lib/pgbackrest/AWS_SECRETS_RECOVERY_identity-sau-main-dev.md
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Skipping 15-backup-restore-test.sh (test script - set RUN_TESTS=true to enable)
[INFO] Skipping 16-test-recovery.sh (test script - set RUN_TESTS=true to enable)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 17-verification.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)

[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] PostgreSQL Production Readiness Verification
[INFO] ═══════════════════════════════════════════════════════════════════════════════
[INFO] 
[INFO] This script verifies 3 CRITICAL checks for production readiness:
[INFO]   1. Citus Cluster Operational (coordinator + workers)
[INFO]   2. SSL/TLS Enforced (certificates valid, connections secure)
[INFO]   3. Coordinator Backups Configured (pgBackRest functional)
[INFO] 
[INFO] 📖 Documentation: /tmp/VERIFICATION_RUNBOOK.md
[INFO] 🔐 Security: Uses sudo for certificate checks (maintains strict permissions)
[INFO] 📊 Exit Code: 0 = production ready, 1 = critical checks failed
[INFO] ═══════════════════════════════════════════════════════════════════════════════

[INFO] 🕐 Ensuring all PostgreSQL services are ready...
[ OK ] ✅ All PostgreSQL services are ready

[INFO] 🔍 Starting PostgreSQL verification...
[INFO] Environment: identity-sau-main-dev
[INFO] Citus: yes

[INFO] Citus mode ENABLED
[INFO] → Coordinator + 1 worker(s) + 3 HA node(s) per worker

[INFO] Verifying 1 worker(s)...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying: worker-01 (type: worker-01)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Starting PostgreSQL verification for identity-sau-main-dev-worker-01...

[INFO] 1️⃣ Checking systemd service status...
[OK]   ✅ Service postgresql@identity-sau-main-dev-worker-01.service is active

[INFO] 2️⃣ Checking PostgreSQL process...
[OK]   ✅ PostgreSQL process is running

[INFO] 3️⃣ Checking socket directory...
[OK]   ✅ Socket directory exists: /var/run/postgresql-identity-sau-main-dev-worker-01
total 4
drwxrwsr-x  2 postgres postgres   80 Jan  3 07:47 .
drwxr-xr-x 56 root     root     1460 Jan  3 07:59 ..
srwxrwxrwx  1 postgres postgres    0 Jan  3 07:47 .s.PGSQL.5432
-rw-------  1 postgres postgres  131 Jan  3 07:47 .s.PGSQL.5432.lock

[INFO] 4️⃣ Testing connection via Unix socket...
[OK]   ✅ Socket connection successful
                                                              version                                                              
-----------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.6 (Ubuntu 17.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

[INFO] 5️⃣ Checking SSL certificates...
[2026-01-03 08:02:59 UTC] USER=www-data EUID=0 PID=3045564 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[OK]   ✅ Server certificate exists: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt
[2026-01-03 08:02:59 UTC] USER=www-data EUID=0 PID=3045573 ACTION=fsop ARGS=openssl x509 -in /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/server.crt -noout -checkend 86400
Certificate will not expire
[OK]   ✅ Server certificate is valid
[2026-01-03 08:02:59 UTC] USER=www-data EUID=0 PID=3045584 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[OK]   ✅ CA certificate exists: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01/ca.crt
[INFO] ℹ️  Client certificates not found at /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/postgres.crt
[INFO]     (This is OK if using password authentication)

[INFO] 6️⃣ Checking PostgreSQL settings...
[OK]   ✅ SSL is enabled worker-01 worker-01
[OK]   ✅ Max connections: 100
[OK]   ✅ Listen addresses: 10.100.1.215
[OK]   ✅ WAL level: logical
[OK]   ✅ Shared preload libraries: shared_preload_libraries

[INFO] 7️⃣ Checking replication configuration...
[INFO] ℹ️  No synchronous standbys configured (single node or async replication)
[INFO] Checking replication slots...
      slot_name       | slot_type | active | restart_lsn 
----------------------+-----------+--------+-------------
 worker_01_standby_01 | physical  | f      | 
(1 row)
[OK]   ✅ Replication slot naming uses underscores (correct)
[INFO] Checking active replication connections...
 application_name | client_addr | state | sync_state 
------------------+-------------+-------+------------
(0 rows)
[INFO] ℹ️  No active replication connections
[INFO] ℹ️  This is a PRIMARY node (no standby.signal)

[INFO] 8️⃣ Checking pg_hba.conf for replication rules...
[WARN] ⚠️ pg_hba.conf not found at /var/lib/postgresql/17/identity-sau-main-dev/worker-01/pg_hba.conf

[INFO] 9️⃣ Checking Citus configuration...
[OK]   ✅ Citus extension is installed
[OK]   ✅ Citus version: Citus 13.2.0
[OK]   ✅ max_prepared_transactions: 100 (adequate for Citus)
[INFO] Citus active worker nodes:
                          node_name                          | node_port 
-------------------------------------------------------------+-----------
 db-identity-sau-main-dev-postgresql-worker-01.fastorder.com |      5432
(1 row)



[INFO] 🔟 Checking data directory...
[OK]   ✅ Data directory exists: /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[OK]   ✅ Data directory size: 4.0K

[INFO] 1️⃣1️⃣ Checking PgBouncer configuration...
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini' as root on web-03.
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/%i/pgbouncer.ini' as root on web-03.
[OK]   ✅ PgBouncer is installed
[INFO]    Version: 1.24.1
2.1.12-stable
c-ares
OpenSSL
yes
Failed to print table: Broken pipe
[INFO] ℹ️  PgBouncer service not configured for this environment

[INFO] 1️⃣2️⃣ Enhanced PgBouncer Admin Console Verification...
Failed to print table: Broken pipe
[INFO] ℹ️  PgBouncer not configured for enhanced verification

[INFO] 1️⃣3️⃣ Replicator User Connection Verification...
[INFO] Found 1 replication slot(s) - verifying replicator connectivity...
[WARN] ⚠️ Replicator certificates not found at /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01
[INFO]    Expected files:
[INFO]    - /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/root.crt
[INFO]    - /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.crt
[INFO]    - /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01/replicator.key
[INFO] Checking pg_hba.conf replicator rules...
[OK]   ✅ Replicator HBA rules found:
 line_number |  type   |   database    |  user_name   |   address    |  auth_method  | options | error 
-------------+---------+---------------+--------------+--------------+---------------+---------+-------
          20 | hostssl | {replication} | {replicator} | 10.100.1.216 | scram-sha-256 |         | 
          21 | hostssl | {replication} | {replicator} | 10.100.1.215 | scram-sha-256 |         | 
(2 rows)
[INFO] Checking active replicator connections in pg_stat_activity...
[WARN] ⚠️ No active replicator connections in pg_stat_activity
[WARN]    This is expected if standbys are not currently connected

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ PostgreSQL verification completed successfully!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Instance:       identity-sau-main-dev-worker-01
[INFO] Service:        postgresql@identity-sau-main-dev-worker-01.service
[INFO] Socket:         /var/run/postgresql-identity-sau-main-dev-worker-01
[INFO] Data Directory: /var/lib/postgresql/17/identity-sau-main-dev/worker-01
[INFO] Hostname:       db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO] Port:           5432
[INFO] SSL:            on
[INFO] WAL Level:      logical
[INFO] Citus:          yes
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 💡 OPTIMIZATION OPPORTUNITIES (Optional Enhancements)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1. Review connection limits for production workload
[INFO]    🔌 Current: max_connections = 100 (PostgreSQL default)
[INFO]    💡 Consider: Increasing to 200-500 for production applications
[INFO]    ⚙️  Alternative: Use PgBouncer connection pooling (lower PostgreSQL limit, higher client capacity)
[INFO]    🔧 Action: Adjust max_connections in postgresql.conf based on workload analysis
[INFO]    ⚠️  Note: Each connection consumes ~10MB RAM; tune based on available memory
[INFO]    📚 Docs: https://www.postgresql.org/docs/current/runtime-config-connection.html

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ℹ️  These are optional enhancements for production-scale deployments
[INFO] ℹ️  Current configuration is fully functional and ready for production
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] ✅ Verification passed for worker-01

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying: worker-01-standby-01 (type: worker-01)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Starting PostgreSQL verification for identity-sau-main-dev-worker-01-standby-01...

[INFO] 1️⃣ Checking systemd service status...
[OK]   ✅ Service postgresql@identity-sau-main-dev-worker-01-standby-01.service is active

[INFO] 2️⃣ Checking PostgreSQL process...
[OK]   ✅ PostgreSQL process is running

[INFO] 3️⃣ Checking socket directory...
[OK]   ✅ Socket directory exists: /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
total 4
drwxrwsr-x  2 postgres postgres   80 Jan  3 07:52 .
drwxr-xr-x 56 root     root     1460 Jan  3 07:59 ..
srwxrwxrwx  1 postgres postgres    0 Jan  3 07:52 .s.PGSQL.5432
-rw-------  1 postgres postgres  153 Jan  3 07:52 .s.PGSQL.5432.lock

[INFO] 4️⃣ Testing connection via Unix socket...
[OK]   ✅ Socket connection successful
                                                              version                                                              
-----------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.6 (Ubuntu 17.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

[INFO] 5️⃣ Checking SSL certificates...
[2026-01-03 08:03:21 UTC] USER=www-data EUID=0 PID=3046425 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[OK]   ✅ Server certificate exists: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-03 08:03:21 UTC] USER=www-data EUID=0 PID=3046434 ACTION=fsop ARGS=openssl x509 -in /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/server.crt -noout -checkend 86400
Certificate will not expire
[OK]   ✅ Server certificate is valid
[2026-01-03 08:03:21 UTC] USER=www-data EUID=0 PID=3046443 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[OK]   ✅ CA certificate exists: /etc/fastorder/postgresql/certs/identity-sau-main-dev/worker-01-standby-01/ca.crt
[INFO] ℹ️  Client certificates not found at /home/postgres/ssl/.postgresql/identity-sau-main-dev/worker-01-standby-01/postgres.crt
[INFO]     (This is OK if using password authentication)

[INFO] 6️⃣ Checking PostgreSQL settings...
[OK]   ✅ SSL is enabled worker-01 worker-01-standby-01
[OK]   ✅ Max connections: 100
[OK]   ✅ Listen addresses: 10.100.1.216
[OK]   ✅ WAL level: logical
[OK]   ✅ Shared preload libraries: shared_preload_libraries

[INFO] 7️⃣ Checking replication configuration...
[INFO] ℹ️  No synchronous standbys configured (single node or async replication)
[INFO] Checking replication slots...
 slot_name | slot_type | active | restart_lsn 
-----------+-----------+--------+-------------
(0 rows)
[OK]   ✅ Replication slot naming uses underscores (correct)
[INFO] Checking active replication connections...
 application_name | client_addr | state | sync_state 
------------------+-------------+-------+------------
(0 rows)
[INFO] ℹ️  No active replication connections
[INFO] ℹ️  This is a PRIMARY node (no standby.signal)

[INFO] 8️⃣ Checking pg_hba.conf for replication rules...
[WARN] ⚠️ pg_hba.conf not found at /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01/pg_hba.conf

[INFO] 9️⃣ Checking Citus configuration...
[INFO] ℹ️  Citus extension not needed on standby (will inherit from primary via replication)

[INFO] 🔟 Checking data directory...
[OK]   ✅ Data directory exists: /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[OK]   ✅ Data directory size: 4.0K

[INFO] 1️⃣1️⃣ Checking PgBouncer configuration...
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini' as root on web-03.
Failed to print table: Broken pipe
[OK]   ✅ PgBouncer is installed
[INFO]    Version: 1.24.1
2.1.12-stable
c-ares
OpenSSL
yes
[OK]   ✅ PgBouncer service is active: pgbouncer@identity-sau-main-dev.service
[OK]   ✅ PgBouncer IP service is active: pgbouncer-ip@identity-sau-main-dev.service
[OK]   ✅ PgBouncer IP: 10.100.1.204
[OK]   ✅ PgBouncer IP is bound to network interface
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini' as root on web-03.
[WARN] ⚠️ PgBouncer config not found: /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/identity-sau-main-dev/userlist.txt' as root on web-03.
[WARN] ⚠️ PgBouncer auth file not found: /etc/pgbouncer/identity-sau-main-dev/userlist.txt
[OK]   ✅ PgBouncer is listening on port 6432

[INFO] 1️⃣2️⃣ Enhanced PgBouncer Admin Console Verification...
Failed to print table: Broken pipe
[INFO] ℹ️  PgBouncer not configured for enhanced verification

[INFO] 1️⃣3️⃣ Replicator User Connection Verification...
[INFO] ℹ️  No replication slots configured - skipping replicator verification

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ PostgreSQL verification completed successfully!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Instance:       identity-sau-main-dev-worker-01-standby-01
[INFO] Service:        postgresql@identity-sau-main-dev-worker-01-standby-01.service
[INFO] Socket:         /var/run/postgresql-identity-sau-main-dev-worker-01-standby-01
[INFO] Data Directory: /var/lib/postgresql/17/identity-sau-main-dev/worker-01-standby-01
[INFO] Hostname:       db-identity-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
[INFO] Port:           5432
[INFO] SSL:            on
[INFO] WAL Level:      logical
[INFO] Citus:          yes
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] ✅ Verification passed for worker-01-standby-01

[INFO] Skipping worker-01-standby-02 - service not configured
[INFO] Skipping worker-01-standby-03 - service not configured
[INFO] Verifying coordinator...
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Verifying: coordinator (type: coordinator)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Starting PostgreSQL verification for identity-sau-main-dev-coordinator...

[INFO] 1️⃣ Checking systemd service status...
[OK]   ✅ Service postgresql@identity-sau-main-dev-coordinator.service is active

[INFO] 2️⃣ Checking PostgreSQL process...
[OK]   ✅ PostgreSQL process is running

[INFO] 3️⃣ Checking socket directory...
[OK]   ✅ Socket directory exists: /var/run/postgresql-identity-sau-main-dev-coordinator
total 4
drwxrwsr-x  2 postgres postgres   80 Jan  3 07:55 .
drwxr-xr-x 56 root     root     1460 Jan  3 07:59 ..
srwxrwxrwx  1 postgres postgres    0 Jan  3 07:55 .s.PGSQL.5432
-rw-------  1 postgres postgres  135 Jan  3 07:55 .s.PGSQL.5432.lock

[INFO] 4️⃣ Testing connection via Unix socket...
[OK]   ✅ Socket connection successful
                                                              version                                                              
-----------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.6 (Ubuntu 17.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

[INFO] 5️⃣ Checking SSL certificates...
[2026-01-03 08:03:42 UTC] USER=www-data EUID=0 PID=3046937 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[OK]   ✅ Server certificate exists: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt
[2026-01-03 08:03:42 UTC] USER=www-data EUID=0 PID=3046946 ACTION=fsop ARGS=openssl x509 -in /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/server.crt -noout -checkend 86400
Certificate will not expire
[OK]   ✅ Server certificate is valid
[2026-01-03 08:03:42 UTC] USER=www-data EUID=0 PID=3046955 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[OK]   ✅ CA certificate exists: /etc/fastorder/postgresql/certs/identity-sau-main-dev/coordinator/ca.crt
[INFO] ℹ️  Client certificates not found at /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/postgres.crt
[INFO]     (This is OK if using password authentication)

[INFO] 6️⃣ Checking PostgreSQL settings...
[OK]   ✅ SSL is enabled coordinator coordinator
[OK]   ✅ Max connections: 150
[OK]   ✅ Listen addresses: 10.100.1.214
[OK]   ✅ WAL level: logical
[OK]   ✅ Shared preload libraries: shared_preload_libraries

[INFO] 7️⃣ Checking replication configuration...
[INFO] ℹ️  No synchronous standbys configured (single node or async replication)
[INFO] Checking replication slots...
         slot_name          | slot_type | active | restart_lsn 
----------------------------+-----------+--------+-------------
 slot_identity_sau_main_dev | logical   | t      | 0/80198F0
(1 row)
[OK]   ✅ Replication slot naming uses underscores (correct)
[INFO] Checking active replication connections...
  application_name  | client_addr  |   state   | sync_state 
--------------------+--------------+-----------+------------
 Debezium Streaming | 10.100.1.214 | streaming | async
(1 row)
[INFO] ℹ️  Async replication is active
[INFO] ℹ️  This is a PRIMARY node (no standby.signal)

[INFO] 8️⃣ Checking pg_hba.conf for replication rules...
[WARN] ⚠️ pg_hba.conf not found at /var/lib/postgresql/17/identity-sau-main-dev/coordinator/pg_hba.conf

[INFO] 9️⃣ Checking Citus configuration...
[OK]   ✅ Citus extension is installed
[OK]   ✅ Citus version: Citus 13.2.0
[OK]   ✅ max_prepared_transactions: 100 (adequate for Citus)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] COORDINATOR-SPECIFIC CHECKS
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Checking registered workers...
[INFO] ℹ️  Coordinator role verified via pg_dist_node (1 workers registered)
[INFO] Checking coordinator hostname configuration...
[OK]   ✅ Coordinator hostname: ---------------------------------------------------------------:----------
[INFO] Checking for stuck prepared transactions...
[OK]   ✅ No stuck Citus prepared transactions
[INFO] Expected workers: 1
[INFO] Registered workers: 1
[OK]   ✅ All 1 worker(s) successfully registered
[INFO] Registered worker nodes:
                           nodename                            | nodeport | groupid | isactive | noderole | shouldhaveshards 
---------------------------------------------------------------+----------+---------+----------+----------+------------------
 db-identity-sau-main-dev-postgresql-coordinator.fastorder.com |     5432 |       0 | t        | primary  | f
 db-identity-sau-main-dev-postgresql-worker-01.fastorder.com   |     5432 |       1 | t        | primary  | t
(2 rows)

[INFO] Note: groupid=0 is the coordinator, groupid>0 are workers

[INFO] Citus active worker nodes:
                          node_name                          | node_port 
-------------------------------------------------------------+-----------
 db-identity-sau-main-dev-postgresql-worker-01.fastorder.com |      5432
(1 row)


[INFO] Verifying Citus workers...
[INFO] Checking worker: db-identity-sau-main-dev-postgresql-worker-01.fastorder.com
 citus_add_node 
----------------
              2
(1 row)


[INFO] Testing Citus distributed table setup...
[INFO] Checking for blocking locks...
    SELECT pg_terminate_backend(pid)
    FROM pg_stat_activity
    WHERE pid <> pg_backend_pid()
      AND state = 'idle in transaction'
      AND query_start < NOW() - INTERVAL '30 seconds'
      AND datname = current_database();
  
 pg_terminate_backend 
----------------------
(0 rows)

[INFO] Creating demo schema (if needed)...
CREATE SCHEMA
[OK]   ✅ Demo schema ready
[INFO] Creating distributed table 'demo.events'...
CREATE TABLE
[OK]   ✅ Table is already distributed
[INFO] Inserting test data...
INSERT 0 1
[OK]   ✅ Distributed table contains 1 row(s)
[INFO] Checking shard distribution...
[OK]   ✅ Table has 1 shard(s)
[INFO] Shard placement across workers (first 10 shards):
 shardid | nodename | nodeport | shardstate 
---------+----------+----------+------------
(0 rows)
[OK]   ✅ Verified 3 shard placement(s)
[INFO] Testing query routing (EXPLAIN for user_id=42)...
[INFO]    Query plan:         QUERY PLAN        
--------------------------
 Seq Scan on events
   Filter: (user_id = 42)
(2 rows)


[INFO] 🔟 Checking data directory...
[OK]   ✅ Data directory exists: /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[OK]   ✅ Data directory size: 4.0K

[INFO] 1️⃣1️⃣ Checking PgBouncer configuration...
Sorry, user www-data is not allowed to execute '/usr/bin/test -f /etc/pgbouncer/identity-sau-main-dev/pgbouncer.ini' as root on web-03.
Failed to print table: Broken pipe
[OK]   ✅ PgBouncer is installed
[INFO]    Version: 1.24.1
2.1.12-stable
c-ares
OpenSSL
yes
Failed to print table: Broken pipe
[INFO] ℹ️  PgBouncer service not configured for this environment

[INFO] 1️⃣2️⃣ Enhanced PgBouncer Admin Console Verification...
[INFO] ℹ️  PgBouncer password not found

[INFO] 1️⃣3️⃣ Replicator User Connection Verification...
[INFO] Found 1 replication slot(s) - verifying replicator connectivity...
[WARN] ⚠️ Replicator certificates not found at /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator
[INFO]    Expected files:
[INFO]    - /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/root.crt
[INFO]    - /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/replicator.crt
[INFO]    - /home/postgres/ssl/.postgresql/identity-sau-main-dev/coordinator/replicator.key
[INFO] Checking pg_hba.conf replicator rules...
[OK]   ✅ Replicator HBA rules found:
 line_number | type | database | user_name | address | auth_method | options | error 
-------------+------+----------+-----------+---------+-------------+---------+-------
(0 rows)
[INFO] Checking active replicator connections in pg_stat_activity...
[WARN] ⚠️ No active replicator connections in pg_stat_activity
[WARN]    This is expected if standbys are not currently connected

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK]   ✅ PostgreSQL verification completed successfully!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Instance:       identity-sau-main-dev-coordinator
[INFO] Service:        postgresql@identity-sau-main-dev-coordinator.service
[INFO] Socket:         /var/run/postgresql-identity-sau-main-dev-coordinator
[INFO] Data Directory: /var/lib/postgresql/17/identity-sau-main-dev/coordinator
[INFO] Hostname:       db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[INFO] Port:           5432
[INFO] SSL:            on
[INFO] WAL Level:      logical
[INFO] Citus:          yes
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Failed to print table: Broken pipe

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 💡 OPTIMIZATION OPPORTUNITIES (Optional Enhancements)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1. Enable PgBouncer connection pooling
[INFO]    📦 Benefit: Reduces connection overhead for high-concurrency workloads
[INFO]    ⚡ Use case: When facing connection exhaustion or frequent connect/disconnect cycles
[INFO]    🔧 Action: Enable and configure pgbouncer@identity-sau-main-dev.service
[INFO]    📚 Docs: https://www.pgbouncer.org/config.html

[INFO] 2. Enable synchronous replication for zero-data-loss (RPO=0)
[INFO]    🛡️  Benefit: Guaranteed no data loss on primary failure (zero RPO)
[INFO]    ⚖️  Trade-off: Slightly higher write latency (~1-5ms) for durability guarantee
[INFO]    🎯 Use case: Critical data requiring absolute durability across availability zones
[INFO]    🔧 Action: ALTER SYSTEM SET synchronous_standby_names = 'ANY 1 (coordinator_standby_01, coordinator_standby_02)';
[INFO]    ⚠️  Note: Requires at least one standby to be available for writes to commit
[INFO]    📚 Docs: https://www.postgresql.org/docs/current/warm-standby.html#SYNCHRONOUS-REPLICATION

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ℹ️  These are optional enhancements for production-scale deployments
[INFO] ℹ️  Current configuration is fully functional and ready for production
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] ✅ Verification passed for coordinator


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📊 PRODUCTION READINESS CHECKS (Step 04 & 05)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 🔍 Checking Monitoring Setup (postgres_exporter or observability cell)...

[INFO] ℹ️  Monitoring can be configured via:
[INFO]    • Local postgres_exporter (step 04-monitoring-setup.sh)
[INFO]    • Observability Cell integration (step 02-observability-cell)

[ OK ] ✅ postgres_exporter is installed
[INFO]    Version: 0.10.1-1ubuntu0.22.04.3
[ OK ] ✅ postgres_exporter-identity-sau-main-dev.service is running
[WARN] ⚠️  Metrics endpoint not responding
[INFO] ℹ️  Monitoring user 'postgres_exporter' not found in PostgreSQL
[INFO]    This is expected if using observability cell remote monitoring
[INFO] ℹ️  Monitoring check passed (local or observability cell)

[INFO] 🔍 Checking Backup Setup (pgBackRest + WAL archiving)...

[ OK ] ✅ pgBackRest is installed
[INFO]    Version: pgBackRest 2.56.0
[ OK ] ✅ WAL archiving is enabled (archive_mode=on)
[ OK ] ✅ archive_command is configured for pgBackRest
[INFO]    Command: timeout 30 /usr/bin/pgbackrest --stanza=identity-sau-main-dev-coordinator archive-push %p
[ OK ] ✅ pgBackRest configuration exists
[ OK ] ✅ pgBackRest stanza 'identity-sau-main-dev-coordinator' is initialized
[ OK ] ✅ Backups exist (4 full backup(s))
[INFO]    Latest backup info:
                 timestamp start/stop: 2026-01-03 07:55:04+00 / 2026-01-03 07:55:14+00
                 wal start/stop: 000000010000000000000004 / 000000010000000000000004
                 database size: 37.5MB, database backup size: 37.5MB
                 repo1: backup set size: 5.7MB, backup size: 5.7MB
     
             full backup: 20260103-075533F
                 timestamp start/stop: 2026-01-03 07:55:33+00 / 2026-01-03 07:55:39+00
                 wal start/stop: 000000010000000000000007 / 000000010000000000000007
                 database size: 37.5MB, database backup size: 37.5MB
                 repo1: backup set size: 5.7MB, backup size: 5.7MB
[ OK ] ✅ Automated backup cron jobs are configured
[INFO]    Schedule:
     0 2 * * 0 root /usr/local/bin/pgbackrest-full-backup-identity-sau-main-dev.sh
     0 2 * * 1-6 root /usr/local/bin/pgbackrest-diff-backup-identity-sau-main-dev.sh
[ OK ] ✅ Backup directory exists: /var/lib/pgbackrest
[INFO]    Total backup size: 2.3G

[INFO] 🔍 Checking Worker Backup Coverage...

[INFO] ℹ️  Worker backups are optional for development environments
[INFO]    For production, ensure all workers have backup coverage

[INFO] Checking worker 1/1: worker-01...
[WARN] ⚠️  Worker worker-01 stanza exists but status unknown
[INFO] ℹ️  Incomplete worker backup coverage (0/1) - OK for dev

[INFO] 🔍 Checking Synchronous Replication (RPO=0)...

[INFO] ℹ️  Synchronous replication (RPO=0) is optional for development
[INFO]    For production with zero data loss requirement, enable sync replication

[INFO] ℹ️  Worker worker-01 synchronous replication NOT configured
[INFO]    └─ synchronous_commit: on
[INFO]    └─ synchronous_standby_names: 
[ OK ] ✅ All workers have synchronous replication (RPO=0)

[INFO] 🔍 Checking Connection and Memory Optimization...

[ OK ] ✅ Coordinator max_connections optimized: 150
[ OK ] ✅ Coordinator work_mem optimized: 8MB
[ OK ] ✅ Worker worker-01 max_connections optimized: 100
[ OK ] ✅ Worker worker-01 work_mem optimized: 8MB
[ OK ] ✅ All instances have optimized connection and memory settings

[INFO] 🔍 Checking Optimizations...

[ OK ] ✅ Citus coordinator host configured: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com
[ OK ] ✅ Periodic integrity checks configured
[INFO]    └─ Daily checks: 3, Weekly verify: 3
[WARN] ⚠️  Backup schedule NOT staggered (all at :00)
[INFO]    Optimize with: ./setup/04-postgresql/steps/04-production-optimizations.sh
[2026-01-03 08:04:15 UTC] USER=www-data EUID=0 PID=3048358 ACTION=fsop ARGS=test -f /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[2026-01-03 08:04:15 UTC] USER=www-data EUID=0 PID=3048368 ACTION=fsop ARGS=grep -q ## Cipher Key Management /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
[WARN] ⚠️  Cipher key management documentation missing
[INFO]    Add with: ./setup/04-postgresql/steps/04-production-optimizations.sh
[INFO] ℹ️  Offsite backup (repo2) not configured (optional for production)
[INFO]    Setup guide: ./setup/05-db/engine/postgresql/steps/14-setup-offsite-backup.sh
[WARN] ⚠️  Some production optimizations incomplete

[INFO] 🔍 Checking Citus Maintenance Daemon Health...

[INFO] Checking for stuck Citus Maintenance Daemons...
[ OK ] ✅ Citus Maintenance Daemons are healthy
[INFO] Checking for stuck distributed table operations...
[ OK ] ✅ No stuck distributed table operations
[INFO] Testing distributed table operations (10s timeout)...
[WARN] ⚠️  CRITICAL: Distributed table test TIMED OUT (10s)
[WARN]    Citus cluster is NOT operational - distributed tables cannot be created
[WARN]    This confirms maintenance daemons are stuck
[WARN]    
[WARN]    🔧 ACTION REQUIRED: Restart coordinator before using Citus
[WARN]       sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev-coordinator.service
[INFO] Checking for lock contention...
[ OK ] ✅ No lock contention detected
[INFO] Checking for lingering prepared transactions...
[ OK ] ✅ No lingering prepared transactions

[WARN] ⚠️  Citus cluster has health issues - see warnings above
[WARN]    
[WARN]    ⚡ IMMEDIATE ACTION: Restart coordinator to restore Citus functionality
[WARN]       sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@identity-sau-main-dev.service

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 📋 PRODUCTION READINESS SUMMARY
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Component                 Status          Production Ready?   
───────────────────────── ─────────────── ────────────────────
Citus Cluster             ✅ Operational YES                 
High Availability         ✅ Configured  YES                 
SSL/TLS Security          ✅ Enabled     YES                 
PgBouncer                 ✅ Running     YES                 
Monitoring                ✅ Operational YES                 
Backups (Coordinator)     ✅ Configured  YES                 
Backups (Workers)         ✅ Configured  YES                 
Sync Replication (RPO=0)  ✅ Enabled     YES                 
Connection Optimization   ✅ Configured  YES                 
Optimizations             ⚠️  Incomplete OPTIONAL            

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[ OK ] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ OK ] 🎉 PRODUCTION READY: 100% (3/3 critical checks passed)
[ OK ] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] ✅ All critical components are operational and production-ready

[INFO] Next steps:
[INFO]   1. Configure Prometheus to scrape metrics: http://localhost:9230/metrics
[INFO]   2. Import Grafana dashboards for PostgreSQL + Citus monitoring
[INFO]   3. Setup alerting rules for critical metrics
[INFO]   4. Schedule regular restore drills (monthly)
[INFO]   5. Review /var/www/html/skeleton.dev.fastorder.com/fixing/scripts/PRODUCTION_READINESS.md

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✓ Verification process completed successfully
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Executing step: 18-production-optimizations.sh
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] PostgreSQL Production Optimizations
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Environment: identity-sau-main-dev
[INFO] Enable Sync Replication: --auto

[INFO] 1️⃣ Configuring Citus coordinator hostname...
[ OK ] ✅ Coordinator hostname already configured: db-identity-sau-main-dev-postgresql-coordinator.fastorder.com

[INFO] 2️⃣ Configuring synchronous replication for RPO=0...
[INFO] Synchronous replication NOT enabled (use './04-production-optimizations.sh yes' to enable)
[INFO] Current configuration: async replication (RPO > 0)
[INFO] 
[INFO] To enable safely after deployment:
[INFO]   /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/enable_sync_replication_safe.sh \
[INFO]     /var/run/postgresql-identity-sau-main-dev-worker-01 worker_01_standby_01

[INFO] 3️⃣ Adding periodic integrity check cron jobs...
[2026-01-03 08:04:18 UTC] USER=www-data EUID=0 PID=3048543 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-integrity-identity-sau-main-dev
[ OK ] ✅ Integrity check cron jobs configured
[INFO]    Daily checks: 02:15, 03:30, 04:45 (coordinator, worker-01, worker-02)
[INFO]    Weekly verify: Sundays at same times

[INFO] 4️⃣ Updating backup schedule with staggered timing...
[ OK ] ✅ Backup schedule staggered:
[INFO]    Coordinator: 02:05 (full: Sun, diff: Mon-Sat)
[INFO]    Worker-01:   03:10 (full: Sun, diff: Mon-Sat)
[INFO]    Worker-02:   04:15 (full: Sun, diff: Mon-Sat)

[INFO] 5️⃣ Documenting cipher key backup procedures...
[2026-01-03 08:04:18 UTC] USER=www-data EUID=0 PID=3048572 ACTION=fsop ARGS=test -f /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md
Sorry, user www-data is not allowed to execute '/usr/bin/grep -q ## Cipher Key Management /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md' as root on web-03.
[ OK ] ✅ Cipher key documentation added to /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_identity-sau-main-dev.md

[INFO] 6️⃣ Checking offsite backup configuration...
[INFO] ℹ️  Offsite backup (repo2) is NOT configured
[INFO]    Configuration example: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[INFO]    Setup instructions: ./setup/04-postgresql/steps/08-setup-offsite-backup.sh
[ OK ] ✅ Offsite backup example available: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Production Optimizations Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[ OK ] Applied optimizations:
[ OK ]   1. ✅ Citus coordinator hostname
[ OK ]   2. ⏭️  Synchronous replication (RPO=0)
[ OK ]   3. ✅ Periodic integrity checks (daily + weekly)
[ OK ]   4. ✅ Staggered backup schedule (reduced load spikes)
[ OK ]   5. ✅ Cipher key backup documentation
[ OK ]   6. ✅ Offsite backup (repo2) example configuration

[INFO] Next steps:
[INFO]   1. Backup cipher keys to secure vault immediately
[INFO]   2. Set up S3/MinIO for offsite backups:
[INFO]      - Instructions: ./setup/04-postgresql/steps/08-setup-offsite-backup.sh
[INFO]      - Example config: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/aws-s3/pgbackrest.conf.example
[INFO]   3. Configure alerting for backup failures (cron exit codes)
[INFO]   4. Test restore drill from offsite repository
[INFO]   5. Enable RPO=0 if needed: ./04-production-optimizations.sh yes

[ OK ] System is now production-grade! 🎉

✓ ✔ PostgreSQL creation completed
[INFO] Cleaning up temporary files...
[INFO] Starting cleanup of temporary files...
[INFO] Cleaning up SSL temp files for identity-sau-main-dev...
[INFO] Cleaning up old provisioning logs...
[SUCCESS] Removed 2 old log files
[INFO] Cleaning up old configuration backups...
✓ ✔ Cleanup completed

✓ ✅ Database infrastructure (postgresql) setup completed successfully

[INFO] Loaded from topology.json: identity-sau-main-dev
[2026-01-03 08:04:20] Loaded environment: identity-sau-main-dev
[2026-01-03 08:04:20] Service: identity, Zone: sau, Branch: main, Env: dev
[2026-01-03 08:04:20] VM IP: 142.93.238.16, Interface: eth0:16
[2026-01-03 08:04:20] Elasticsearch Nodes: 1, PostgreSQL Workers: 1
[2026-01-03 08:04:20] PostgreSQL HA Nodes: 1, Citus Enabled: yes
✓ Environment initialized successfully (mode: general)
[INFO] Starting finalizing setup process...
[INFO] Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps
[INFO] Environment: identity-sau-main-dev

[INFO] Found 3 step(s) to execute

[INFO] 📦 Step 1/3: enable_disable_all_applications...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
════════════════════════════════════════════════════════════════════════════════
  Environment Services Management
════════════════════════════════════════════════════════════════════════════════
  Environment:  identity-sau-main-dev
  Action:       enable
  Triggered by: false
════════════════════════════════════════════════════════════════════════════════

🔍 Scanning for environment-specific services...
✅ Found 8 services for environment: identity-sau-main-dev

📋 Services to enable:
────────────────────────────────────────────────────────────────────────────────
  • confluent-connect-identity-sau-main-dev_coordinator.service  [active/unmasked/enabled]
  • confluent-kraft-identity-sau-main-dev_coordinator.service    [active/unmasked/enabled]
  • elasticsearch@identity-sau-main-dev-node-01.service          [active/unmasked/enabled]
  • pgbouncer-ip@identity-sau-main-dev.service                   [active/unmasked/enabled]
  • pgbouncer@identity-sau-main-dev.service                      [active/unmasked/enabled]
  • postgresql@identity-sau-main-dev-coordinator.service         [active/unmasked/enabled]
  • postgresql@identity-sau-main-dev-worker-01-standby-01.service [active/unmasked/enabled]
  • postgresql@identity-sau-main-dev-worker-01.service           [active/unmasked/enabled]
────────────────────────────────────────────────────────────────────────────────


❌ Cancelled by user
[OK] ✅ Step 1 completed: 01-enable_disable_all_applications.sh

[INFO] 📦 Step 2/3: verify monitoring...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Monitoring Verification for identity-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Detecting installed services...
Failed to print table: Broken pipe
[OK] ✓ PostgreSQL detected
Failed to print table: Broken pipe
[OK] ✓ Elasticsearch detected
Failed to print table: Broken pipe
[OK] ✓ Kafka detected
Failed to print table: Broken pipe
[OK] ✓ PgBouncer detected

[INFO] Services to verify: postgresql elasticsearch kafka pgbouncer

[INFO] 2️⃣ Verifying exporters are running...
[OK] ✓ PostgreSQL exporter is running
[OK] ✓ Elasticsearch exporter is running
[OK] ✓ Kafka JMX exporter is running
[WARN] ⚠️  PgBouncer exporter is not running (may not be configured)

[INFO] 3️⃣ Verifying Prometheus configuration...
[2026-01-03 08:04:23 UTC] USER=www-data EUID=0 PID=3049050 ACTION=passthru ARGS=grep -q job_name: 'postgresql' /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[OK] ✓ postgresql is configured in Prometheus
[2026-01-03 08:04:23 UTC] USER=www-data EUID=0 PID=3049071 ACTION=passthru ARGS=grep -q job_name: 'elasticsearch' /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[OK] ✓ elasticsearch is configured in Prometheus
[2026-01-03 08:04:23 UTC] USER=www-data EUID=0 PID=3049097 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[WARN] ⚠️  kafka is not configured in Prometheus scrape targets
[2026-01-03 08:04:23 UTC] USER=www-data EUID=0 PID=3049122 ACTION=passthru ARGS=grep -q job_name: 'pgbouncer' /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[WARN] ⚠️  pgbouncer is not configured in Prometheus scrape targets

[INFO] 4️⃣ Verifying Prometheus is actively scraping...
[OK] ✓ Prometheus is running
[OK] ✓ postgresql target is UP in Prometheus
[OK] ✓ elasticsearch target is UP in Prometheus
[WARN] ⚠️  kafka target is not UP in Prometheus (may still be initializing)
[WARN] ⚠️  pgbouncer target is not UP in Prometheus (may still be initializing)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Monitoring Verification Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[WARN] Some monitoring issues were detected:

[WARN] Prometheus Configuration Issues:
  - kafka not configured in Prometheus
  - pgbouncer not configured in Prometheus

[WARN] Automatically running monitoring setup scripts to fix issues...

[INFO] Running Kafka monitoring setup...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Kafka Monitoring Integration for identity-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[OK]   ✓ Observability cell is ready

[INFO] 2️⃣ Setting up Kafka JMX exporter integration...
[INFO] Checking observability cell readiness: obs-identity-sau-main-dev
[OK]   Observability cell endpoints registered for identity-sau-main-dev
[INFO] Setting up Kafka JMX exporter for identity-sau-main-dev
[INFO] JMX Prometheus Java Agent already exists at /opt/kafka/libs/jmx_prometheus_javaagent.jar
[2026-01-03 08:04:30 UTC] USER=www-data EUID=0 PID=3049266 ACTION=passthru ARGS=mv /tmp/jmx_exporter.yml /opt/kafka/config/jmx_exporter.yml
[2026-01-03 08:04:30 UTC] USER=www-data EUID=0 PID=3049275 ACTION=passthru ARGS=chmod 644 /opt/kafka/config/jmx_exporter.yml
[OK]   JMX exporter configuration created at /opt/kafka/config/jmx_exporter.yml
[OK]   JMX exporter configuration created
[INFO] Configuring Kafka systemd services to use JMX exporter...
[2026-01-03 08:04:30 UTC] USER=www-data EUID=0 PID=3049299 ACTION=fsop ARGS=test -f /etc/systemd/system/[2026-01-03
[INFO] All Kafka services already configured with JMX exporter
[OK]   Kafka JMX exporter integration complete
[INFO] Metrics endpoint: http://142.93.238.16:9308/metrics
[INFO] Prometheus will automatically scrape: https://metrics-identity-sau-main-dev.fastorder.com:9090
[INFO] View dashboards at: https://dashboards-identity-sau-main-dev.fastorder.com
[OK]   ✓ Kafka JMX exporter integration complete
[INFO] Configuring KAFKA_OPTS environment variable for kafka user...
[2026-01-03 08:04:30 UTC] USER=www-data EUID=0 PID=3049320 ACTION=passthru ARGS=grep -q KAFKA_OPTS.*javaagent.*jmx_prometheus_javaagent /home/kafka/.bashrc
[OK]   ✓ KAFKA_OPTS already configured
[INFO] 2.5️⃣ Enabling JMX exporter in Kafka systemd service...
[2026-01-03 08:04:30 UTC] USER=www-data EUID=0 PID=3049342 ACTION=passthru ARGS=grep -q javaagent.*jmx_prometheus_javaagent /etc/systemd/system/confluent-kraft-identity-sau-main-dev_coordinator.service
[OK]   ✓ JMX exporter already enabled in Kafka systemd services
[INFO] 2.6️⃣ Configuring Prometheus to scrape Kafka metrics...
[2026-01-03 08:04:30 UTC] USER=www-data EUID=0 PID=3049363 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[INFO] Adding Kafka scrape target to Prometheus configuration...
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[2026-01-03 08:04:30 UTC] USER=www-data EUID=0 PID=3049402 ACTION=passthru ARGS=sed -i /# Prometheus self-monitoring/r /tmp/prometheus_kafka_add.yml /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[ERROR] Invalid Prometheus configuration - rolling back
[2026-01-03 08:04:31 UTC] USER=www-data EUID=0 PID=3049435 ACTION=passthru ARGS=sed -i /job_name: 'kafka'/,+6d /etc/prometheus/obs-identity-sau-main-dev/prometheus.yml
[2026-01-03 08:04:31 UTC] USER=www-data EUID=0 PID=3049457 ACTION=fsop ARGS=rm -f /tmp/prometheus_kafka_add.yml

[INFO] 3️⃣ Registering Kafka nodes to monitoring database...
[INFO] Registering Kafka Broker to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Kafka Broker
[INFO]   Identifier:        identity-sau-main-dev-broker-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                142.93.238.16
[INFO]   Port:              9092
[INFO]   FQDN:              eventbus-identity-sau-main-dev-kafka-broker-01.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 1a310579-24b9-4091-8626-7335f80305c3
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ Kafka broker registered
[INFO] Registering Kafka Connect to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Kafka Connect
[INFO]   Identifier:        identity-sau-main-dev-connect-01
[INFO]   Identifier Parent: cluster
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8083
[INFO]   FQDN:              eventbus-identity-sau-main-dev-kafka-connect.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 71719f62-65ea-4a2b-a0ed-4a8d3f80403b
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
[OK]   ✓ Kafka Connect registered
[INFO] Schema Registry not running, skipping registration

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Kafka Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Metrics: http://localhost:9308/metrics
[INFO] Prometheus: https://metrics-identity-sau-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-identity-sau-main-dev.fastorder.com
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✓ Kafka monitoring setup completed

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Step 2 completed: 02-verify-monitoring.sh

[INFO] 📦 Step 3/3: register backup infrastructure...
[INFO] Loaded environment: identity-sau-main-dev (svc=identity zone=sau env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔧 Registering Core Services & Backup Infrastructure for identity-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] 1️⃣ Registering Main App...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Main App
[INFO]   Identifier:        identity-sau-main-dev-main-app
[INFO]   Identifier Parent: application
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8080
[INFO]   FQDN:              app-identity-sau-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 6781a0e7-58d6-4224-ae59-10b617367a2a
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
/opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps/03-register-backup-infrastructure.sh: line 70: ok: command not found

[INFO] 2️⃣ Registering Audit Service...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO]   Application:       Audit Service
[INFO]   Identifier:        identity-sau-main-dev-audit
[INFO]   Identifier Parent: application
[INFO]   IP:                142.93.238.16
[INFO]   Port:              8081
[INFO]   FQDN:              audit-identity-sau-main-dev.fastorder.com
[INFO]   Status:            running
[INFO]   Environment:       identity-sau-main-dev (service=identity, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 203de866-853b-49eb-80a0-dffa65ac5d16
[SUCCESS] Environment UUID: 82a0dcd2-dcf2-422e-a830-b2dd51514393
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/82a0dcd2-dcf2-422e-a830-b2dd51514393
/opt/fastorder/bash/scripts/env_app_setup/setup/06-finalizing/steps/03-register-backup-infrastructure.sh: line 85: ok: command not found

[INFO] 3️⃣ Registering PostgreSQL Backup Node...
[ERROR] Invalid identifier format: backup-db
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register PostgreSQL backup node (non-blocking)

[INFO] 4️⃣ Registering Elasticsearch Backup Node...
[ERROR] Invalid identifier format: backup-search
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Elasticsearch backup node (non-blocking)

[INFO] 5️⃣ Registering Kafka Backup Node...
[ERROR] Invalid identifier format: backup-eventbus
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Kafka backup node (non-blocking)

[INFO] 6️⃣ Registering Backup Orchestrator...
[ERROR] Invalid identifier format: backup-orchestrator
[ERROR] Expected formats:
[ERROR]   SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., authN-by-main144-dev-node-01)
[ERROR]   iam-DOMAIN-ZONE-BRANCH-ENV_NODE_TYPE (e.g., iam-identity-universe-main-dev_keycloak_main)
[ERROR]   obs-SERVICE-zone-BRANCH-ENV-NODE_TYPE (e.g., obs-authN-sau-main-dev-alertmanager)
[WARN] ⚠️  Failed to register Backup orchestrator (non-blocking)

[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Core Services & Backup Infrastructure Registration Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Registered core services:
[INFO]   🚀 main-app            → Core application service
[INFO]   📋 audit               → Centralized audit logging (WORM)

[INFO] Registered backup nodes:
[INFO]   📦 backup-db           → PostgreSQL backup (pgBackRest, PITR)
[INFO]   📦 backup-search       → Elasticsearch snapshots (ILM, S3)
[INFO]   📦 backup-eventbus     → Kafka log segments (replication)
[INFO]   📦 backup-orchestrator → Central backup coordination

[INFO] Dashboard: https://skeleton.dev.fastorder.com/dashboard/monitoring
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Step 3 completed: 03-register-backup-infrastructure.sh


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ finalizing setup completed successfully!
[OK] Executed all 3 steps
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Environment: identity-sau-main-dev
[INFO] Service: identity
[INFO] Zone: sau
[INFO] Branch: main
[INFO] Env: dev