Environment: User Sau Main Dev on web-03
"{\"env\": \"dev\", \"zone\": \"sau\", \"branch\": \"main\", \"db_app\": \"postgresql\", \"service\": \"user\", \"es_nodes\": 1, \"db_enabled\": true, \"pg_standby\": 1, \"pg_workers\": 1, \"search_app\": \"elasticsearch\", \"description\": \"\", \"iam_enabled\": false, \"worker_1_ip\": \"10.100.1.42\", \"eventbus_app\": \"kafka\", \"es_https_mode\": \"direct\", \"service_es_ip\": \"10.100.1.4\", \"worker_1_fqdn\": \"db-user-sau-main-dev-postgresql-worker-01.fastorder.com\", \"search_enabled\": true, \"service_app_ip\": \"10.100.1.2\", \"service_obs_ip\": \"10.100.1.18\", \"service_es_fqdn\": \"search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com\", \"service_otlp_ip\": \"10.100.1.30\", \"eventbus_enabled\": true, \"service_app_fqdn\": \"app-user-sau-main-dev.fastorder.com\", \"service_audit_ip\": \"10.100.1.32\", \"service_obs_fqdn\": \"obs-user-sau-main-dev.fastorder.com\", \"service_tempo_ip\": \"10.100.1.28\", \"service_endpoints\": \"[{\\\"ip\\\":\\\"10.100.1.3\\\",\\\"fqdn\\\":\\\"app-user-sau-main-dev.fastorder.com\\\",\\\"service\\\":\\\"app\\\"},{\\\"ip\\\":\\\"10.100.1.5\\\",\\\"fqdn\\\":\\\"search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com\\\",\\\"service\\\":\\\"es_coordinator\\\"},{\\\"ip\\\":\\\"10.100.1.7\\\",\\\"fqdn\\\":\\\"search-user-sau-main-dev-elasticsearch-node-01.fastorder.com\\\",\\\"service\\\":\\\"es_node_1\\\"},{\\\"ip\\\":\\\"10.100.1.9\\\",\\\"fqdn\\\":\\\"eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com\\\",\\\"service\\\":\\\"kafka_broker_1\\\"},{\\\"ip\\\":\\\"10.100.1.11\\\",\\\"fqdn\\\":\\\"eventbus-user-sau-main-dev-kafka-connect.fastorder.com\\\",\\\"service\\\":\\\"kafka_connect\\\"},{\\\"ip\\\":\\\"10.100.1.13\\\",\\\"fqdn\\\":\\\"schema-user-sau-main-dev-kafka-registry.fastorder.com\\\",\\\"service\\\":\\\"kafka_registry\\\"},{\\\"ip\\\":\\\"10.100.1.15\\\",\\\"fqdn\\\":\\\"db-user-sau-main-dev-postgresql-coordinator.fastorder.com\\\",\\\"service\\\":\\\"pg_coordinator\\\"},{\\\"ip\\\":\\\"10.100.1.17\\\",\\\"fqdn\\\":\\\"db-user-sau-main-dev-postgresql-bouncer.fastorder.com\\\",\\\"service\\\":\\\"pgbouncer\\\"},{\\\"ip\\\":\\\"10.100.1.19\\\",\\\"fqdn\\\":\\\"obs-user-sau-main-dev.fastorder.com\\\",\\\"service\\\":\\\"obs\\\"},{\\\"ip\\\":\\\"10.100.1.21\\\",\\\"fqdn\\\":\\\"metrics-user-sau-main-dev-prometheus.fastorder.com\\\",\\\"service\\\":\\\"metrics\\\"},{\\\"ip\\\":\\\"10.100.1.23\\\",\\\"fqdn\\\":\\\"dashboards-user-sau-main-dev-grafana.fastorder.com\\\",\\\"service\\\":\\\"dashboards\\\"},{\\\"ip\\\":\\\"10.100.1.25\\\",\\\"fqdn\\\":\\\"alerts-user-sau-main-dev-alertmanager.fastorder.com\\\",\\\"service\\\":\\\"alerts\\\"},{\\\"ip\\\":\\\"10.100.1.27\\\",\\\"fqdn\\\":\\\"logstore-user-sau-main-dev-clickhouse.fastorder.com\\\",\\\"service\\\":\\\"logs\\\"},{\\\"ip\\\":\\\"10.100.1.29\\\",\\\"fqdn\\\":\\\"traces-user-sau-main-dev-tempo.fastorder.com\\\",\\\"service\\\":\\\"traces\\\"},{\\\"ip\\\":\\\"10.100.1.31\\\",\\\"fqdn\\\":\\\"telemetry-user-sau-main-dev-opentelemetry.fastorder.com\\\",\\\"service\\\":\\\"telemetry\\\"},{\\\"ip\\\":\\\"10.100.1.33\\\",\\\"fqdn\\\":\\\"audit-user-sau-main-dev.fastorder.com\\\",\\\"service\\\":\\\"audit\\\"},{\\\"ip\\\":\\\"10.100.1.35\\\",\\\"fqdn\\\":\\\"backup-user-sau-main-dev-db-postgresql.fastorder.com\\\",\\\"service\\\":\\\"backup_pg\\\"},{\\\"ip\\\":\\\"10.100.1.37\\\",\\\"fqdn\\\":\\\"backup-user-sau-main-dev-eventbus-kafka.fastorder.com\\\",\\\"service\\\":\\\"backup_kafka\\\"},{\\\"ip\\\":\\\"10.100.1.39\\\",\\\"fqdn\\\":\\\"backup-user-sau-main-dev-search-elasticsearch.fastorder.com\\\",\\\"service\\\":\\\"backup_es\\\"},{\\\"ip\\\":\\\"10.100.1.41\\\",\\\"fqdn\\\":\\\"backup-user-sau-main-dev-orchestrator.fastorder.com\\\",\\\"service\\\":\\\"backup_orchestrator\\\"}]\", \"service_otlp_fqdn\": \"telemetry-user-sau-main-dev-opentelemetry.fastorder.com\", \"postgresql_enabled\": true, \"service_audit_fqdn\": \"audit-user-sau-main-dev.fastorder.com\", \"service_grafana_ip\": \"10.100.1.22\", \"service_tempo_fqdn\": \"traces-user-sau-main-dev-tempo.fastorder.com\", \"service_backup_es_ip\": \"10.100.1.38\", \"service_backup_pg_ip\": \"10.100.1.34\", \"service_es_node_1_ip\": \"10.100.1.6\", \"service_grafana_fqdn\": \"dashboards-user-sau-main-dev-grafana.fastorder.com\", \"service_pgbouncer_ip\": \"10.100.1.16\", \"service_prometheus_ip\": \"10.100.1.20\", \"worker_1_standby_1_ip\": \"10.100.1.43\", \"service_backup_es_fqdn\": \"backup-user-sau-main-dev-search-elasticsearch.fastorder.com\", \"service_backup_pg_fqdn\": \"backup-user-sau-main-dev-db-postgresql.fastorder.com\", \"service_es_node_1_fqdn\": \"search-user-sau-main-dev-elasticsearch-node-01.fastorder.com\", \"service_log_backend_ip\": \"10.100.1.26\", \"service_pgbouncer_fqdn\": \"db-user-sau-main-dev-postgresql-bouncer.fastorder.com\", \"service_alertmanager_ip\": \"10.100.1.24\", \"service_backup_kafka_ip\": \"10.100.1.36\", \"service_prometheus_fqdn\": \"metrics-user-sau-main-dev-prometheus.fastorder.com\", \"worker_1_standby_1_fqdn\": \"db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com\", \"service_kafka_connect_ip\": \"10.100.1.10\", \"service_log_backend_fqdn\": \"logstore-user-sau-main-dev-clickhouse.fastorder.com\", \"service_alertmanager_fqdn\": \"alerts-user-sau-main-dev-alertmanager.fastorder.com\", \"service_backup_kafka_fqdn\": \"backup-user-sau-main-dev-eventbus-kafka.fastorder.com\", \"service_kafka_broker_1_ip\": \"10.100.1.8\", \"service_kafka_registry_ip\": \"10.100.1.12\", \"service_pg_coordinator_ip\": \"10.100.1.14\", \"service_kafka_connect_fqdn\": \"eventbus-user-sau-main-dev-kafka-connect.fastorder.com\", \"postgresql_run_verification\": true, \"service_kafka_broker_1_fqdn\": \"eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com\", \"service_kafka_registry_fqdn\": \"schema-user-sau-main-dev-kafka-registry.fastorder.com\", \"service_pg_coordinator_fqdn\": \"db-user-sau-main-dev-postgresql-coordinator.fastorder.com\", \"service_backup_orchestrator_ip\": \"10.100.1.40\", \"service_backup_orchestrator_fqdn\": \"backup-user-sau-main-dev-orchestrator.fastorder.com\"}"
This job encountered an error. You can restart from the failed step.
This job has been restarted. You are viewing an older attempt. The logs and status shown below are from the latest retry.
This job failed at one of the steps below. You can resume from where it failed to save time and avoid re-running successful steps.
[1m════════════════════════════════════════════════════════════════[0m
[1m FastOrder Pre-Flight Validation Checks[0m
[1m════════════════════════════════════════════════════════════════[0m
[0;34m[INFO][0m Checking SSH connectivity to target host...
[0;32m[✓][0m Target is localhost, skipping SSH check
[0;34m[INFO][0m Checking available disk space...
[0;34m[INFO][0m Checking /data disk (mounted separately for data storage)
[0;32m[✓][0m Disk space sufficient: 267GB available (required: 50GB)
[0;34m[INFO][0m Checking available memory...
[1;33m[⚠][0m Memory limited: 15GB (recommended: 16GB)
→ Consider reducing Elasticsearch nodes or PostgreSQL workers
[0;34m[INFO][0m Checking critical port availability...
[0;32m[✓][0m Port 5432 in use on specific IP (10.100.1.190:5432) - OK, can use different IP
[0;32m[✓][0m Port 9200 in use on specific IP ([::ffff:10.100.1.179]) - OK, can use different IP
[0;32m[✓][0m Port 9300 in use on specific IP ([::ffff:10.100.1.179]) - OK, can use different IP
[0;32m[✓][0m Port 9092 in use on specific IP ([::ffff:10.100.1.235]) - OK, can use different IP
[0;32m[✓][0m Port 2181 available (Zookeeper)
[0;34m[INFO][0m Checking DNS resolution...
[0;32m[✓][0m DNS resolution working: google.com
[0;32m[✓][0m DNS resolution working: github.com
[0;32m[✓][0m DNS resolution working: archive.ubuntu.com
[0;34m[INFO][0m Checking required system commands...
[0;32m[✓][0m Command available: curl
[0;32m[✓][0m Command available: wget
[0;32m[✓][0m Command available: git
[0;32m[✓][0m Command available: sudo
[0;32m[✓][0m Command available: systemctl
[0;32m[✓][0m Command available: apt-get
[0;34m[INFO][0m Checking current system load...
[1;33m[⚠][0m System load elevated: 3.41 (4 CPUs)
→ Provisioning may be slower than expected
[0;34m[INFO][0m Checking for existing environment conflicts...
[0;32m[✓][0m No conflicting services found for: user-uae-main-dev
[1m════════════════════════════════════════════════════════════════[0m
[1m Pre-Flight Check Summary[0m
[1m════════════════════════════════════════════════════════════════[0m
[1;33m[⚠][0m 2 warning(s) detected
⚠️ Environment can proceed with caution
Review warnings above and consider remediation
[INFO] Using web-provided environment: user-sau-main-dev
[INFO] Auto-creating state directory for user-sau-main-dev...
[ OK ] Created topology.json for user-sau-main-dev
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=10.100.1.51)
[0;36m[2026-01-18_23:21:46][0m Starting Terraform provisioning step
[0;36m[2026-01-18_23:21:46][0m Service: user
[0;36m[2026-01-18_23:21:46][0m Zone: sau
[0;36m[2026-01-18_23:21:46][0m Environment: dev
[0;36m[2026-01-18_23:21:46][0m Resource: web-03
[0;36m[2026-01-18_23:21:46][0m Terraform binary: /home/ab/bin/terraform
[0;36m[2026-01-18_23:21:46][0m HOME: /home/www-data
[0;36m[2026-01-18_23:21:46][0m AWS Config: /home/ab/.aws/config
[0;36m[2026-01-18_23:21:46][0m AWS Credentials: /home/ab/.aws/credentials
[0;36m[2026-01-18_23:21:46][0m Terraform directory: /opt/fastorder/cli/terraform/examples/citus-production
[0;36m[2026-01-18_23:21:46][0m Running terraform init...
[0m[1mInitializing the backend...[0m
[0m[1mUpgrading modules...[0m
- citus_cluster in ../../modules/citus_cluster
[0m[1mInitializing provider plugins...[0m
- Finding hashicorp/aws versions matching "~> 5.0"...
- Using previously-installed hashicorp/aws v5.100.0
[0m[1m[32mTerraform has been successfully initialized![0m[32m[0m
[0m[32m
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.[0m
[0;32m[2026-01-18_23:21:50] ✓[0m Terraform init succeeded
[0;36m[2026-01-18_23:21:50][0m Running terraform validate...
[32m[1mSuccess![0m The configuration is valid.
[0m
[0;32m[2026-01-18_23:21:53] ✓[0m Terraform validate succeeded
[0;36m[2026-01-18_23:21:53][0m Running terraform plan...
[0m[1mmodule.citus_cluster.data.aws_caller_identity.current: Reading...[0m[0m
[0m[1mmodule.citus_cluster.data.aws_caller_identity.current: Read complete after 0s [id=464621692046][0m
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
[32m+[0m create[0m
Terraform will perform the following actions:
[1m # module.citus_cluster.aws_iam_instance_profile.citus[0m will be created
[0m [32m+[0m[0m resource "aws_iam_instance_profile" "citus" {
[32m+[0m[0m arn = (known after apply)
[32m+[0m[0m create_date = (known after apply)
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m name = (known after apply)
[32m+[0m[0m name_prefix = "citus-prod-"
[32m+[0m[0m path = "/"
[32m+[0m[0m role = (known after apply)
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-prod"
}
[32m+[0m[0m tags_all = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "ManagedBy" = "Terraform"
[32m+[0m[0m "Name" = "citus-prod"
[32m+[0m[0m "Owner" = "Platform Team"
[32m+[0m[0m "Project" = "FastOrder"
}
[32m+[0m[0m unique_id = (known after apply)
}
[1m # module.citus_cluster.aws_iam_role.citus[0m will be created
[0m [32m+[0m[0m resource "aws_iam_role" "citus" {
[32m+[0m[0m arn = (known after apply)
[32m+[0m[0m assume_role_policy = jsonencode(
{
[32m+[0m[0m Statement = [
[32m+[0m[0m {
[32m+[0m[0m Action = "sts:AssumeRole"
[32m+[0m[0m Effect = "Allow"
[32m+[0m[0m Principal = {
[32m+[0m[0m Service = "ec2.amazonaws.com"
}
},
]
[32m+[0m[0m Version = "2012-10-17"
}
)
[32m+[0m[0m create_date = (known after apply)
[32m+[0m[0m force_detach_policies = false
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m managed_policy_arns = (known after apply)
[32m+[0m[0m max_session_duration = 3600
[32m+[0m[0m name = (known after apply)
[32m+[0m[0m name_prefix = "citus-prod-"
[32m+[0m[0m path = "/"
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-prod"
}
[32m+[0m[0m tags_all = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "ManagedBy" = "Terraform"
[32m+[0m[0m "Name" = "citus-prod"
[32m+[0m[0m "Owner" = "Platform Team"
[32m+[0m[0m "Project" = "FastOrder"
}
[32m+[0m[0m unique_id = (known after apply)
}
[1m # module.citus_cluster.aws_iam_role_policy.secrets_manager[0][0m will be created
[0m [32m+[0m[0m resource "aws_iam_role_policy" "secrets_manager" {
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m name = (known after apply)
[32m+[0m[0m name_prefix = "secrets-access-"
[32m+[0m[0m policy = jsonencode(
{
[32m+[0m[0m Statement = [
[32m+[0m[0m {
[32m+[0m[0m Action = [
[32m+[0m[0m "secretsmanager:GetSecretValue",
[32m+[0m[0m "secretsmanager:DescribeSecret",
]
[32m+[0m[0m Effect = "Allow"
[32m+[0m[0m Resource = "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/db/web/ksa/main/dev/postgresqladmin/ksa/prod*"
},
]
[32m+[0m[0m Version = "2012-10-17"
}
)
[32m+[0m[0m role = (known after apply)
}
[1m # module.citus_cluster.aws_iam_role_policy_attachment.cloudwatch[0m will be created
[0m [32m+[0m[0m resource "aws_iam_role_policy_attachment" "cloudwatch" {
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
[32m+[0m[0m role = (known after apply)
}
[1m # module.citus_cluster.aws_iam_role_policy_attachment.ssm[0m will be created
[0m [32m+[0m[0m resource "aws_iam_role_policy_attachment" "ssm" {
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
[32m+[0m[0m role = (known after apply)
}
[1m # module.citus_cluster.aws_instance.coordinator[0m will be created
[0m [32m+[0m[0m resource "aws_instance" "coordinator" {
[32m+[0m[0m ami = "ami-0b2aae5f4283c0df2"
[32m+[0m[0m arn = (known after apply)
[32m+[0m[0m associate_public_ip_address = (known after apply)
[32m+[0m[0m availability_zone = (known after apply)
[32m+[0m[0m cpu_core_count = (known after apply)
[32m+[0m[0m cpu_threads_per_core = (known after apply)
[32m+[0m[0m disable_api_stop = (known after apply)
[32m+[0m[0m disable_api_termination = (known after apply)
[32m+[0m[0m ebs_optimized = (known after apply)
[32m+[0m[0m enable_primary_ipv6 = (known after apply)
[32m+[0m[0m get_password_data = false
[32m+[0m[0m host_id = (known after apply)
[32m+[0m[0m host_resource_group_arn = (known after apply)
[32m+[0m[0m iam_instance_profile = (known after apply)
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m instance_initiated_shutdown_behavior = (known after apply)
[32m+[0m[0m instance_lifecycle = (known after apply)
[32m+[0m[0m instance_state = (known after apply)
[32m+[0m[0m instance_type = "r6i.2xlarge"
[32m+[0m[0m ipv6_address_count = (known after apply)
[32m+[0m[0m ipv6_addresses = (known after apply)
[32m+[0m[0m key_name = (known after apply)
[32m+[0m[0m monitoring = (known after apply)
[32m+[0m[0m outpost_arn = (known after apply)
[32m+[0m[0m password_data = (known after apply)
[32m+[0m[0m placement_group = (known after apply)
[32m+[0m[0m placement_partition_number = (known after apply)
[32m+[0m[0m primary_network_interface_id = (known after apply)
[32m+[0m[0m private_dns = (known after apply)
[32m+[0m[0m private_ip = (known after apply)
[32m+[0m[0m public_dns = (known after apply)
[32m+[0m[0m public_ip = (known after apply)
[32m+[0m[0m secondary_private_ips = (known after apply)
[32m+[0m[0m security_groups = (known after apply)
[32m+[0m[0m source_dest_check = true
[32m+[0m[0m spot_instance_request_id = (known after apply)
[32m+[0m[0m subnet_id = "subnet-0a1f5a9a74ed030cf"
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-coordinator-prod"
[32m+[0m[0m "Role" = "coordinator"
[32m+[0m[0m "Service" = "citus"
}
[32m+[0m[0m tags_all = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "ManagedBy" = "Terraform"
[32m+[0m[0m "Name" = "citus-coordinator-prod"
[32m+[0m[0m "Owner" = "Platform Team"
[32m+[0m[0m "Project" = "FastOrder"
[32m+[0m[0m "Role" = "coordinator"
[32m+[0m[0m "Service" = "citus"
}
[32m+[0m[0m tenancy = (known after apply)
[32m+[0m[0m user_data = "2a9e41ea765dcf3b3046ee10d2f458c18f00e430"
[32m+[0m[0m user_data_base64 = (known after apply)
[32m+[0m[0m user_data_replace_on_change = false
[32m+[0m[0m vpc_security_group_ids = (known after apply)
[32m+[0m[0m ebs_block_device {
[32m+[0m[0m delete_on_termination = false
[32m+[0m[0m device_name = "/dev/sdf"
[32m+[0m[0m encrypted = true
[32m+[0m[0m iops = 3000
[32m+[0m[0m kms_key_id = (known after apply)
[32m+[0m[0m snapshot_id = (known after apply)
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-coordinator-prod-data"
}
[32m+[0m[0m tags_all = (known after apply)
[32m+[0m[0m throughput = 125
[32m+[0m[0m volume_id = (known after apply)
[32m+[0m[0m volume_size = 500
[32m+[0m[0m volume_type = "gp3"
}
[32m+[0m[0m root_block_device {
[32m+[0m[0m delete_on_termination = false
[32m+[0m[0m device_name = (known after apply)
[32m+[0m[0m encrypted = true
[32m+[0m[0m iops = (known after apply)
[32m+[0m[0m kms_key_id = (known after apply)
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-coordinator-prod-root"
}
[32m+[0m[0m tags_all = (known after apply)
[32m+[0m[0m throughput = (known after apply)
[32m+[0m[0m volume_id = (known after apply)
[32m+[0m[0m volume_size = 100
[32m+[0m[0m volume_type = "gp3"
}
}
[1m # module.citus_cluster.aws_instance.workers[0][0m will be created
[0m [32m+[0m[0m resource "aws_instance" "workers" {
[32m+[0m[0m ami = "ami-0b2aae5f4283c0df2"
[32m+[0m[0m arn = (known after apply)
[32m+[0m[0m associate_public_ip_address = (known after apply)
[32m+[0m[0m availability_zone = (known after apply)
[32m+[0m[0m cpu_core_count = (known after apply)
[32m+[0m[0m cpu_threads_per_core = (known after apply)
[32m+[0m[0m disable_api_stop = (known after apply)
[32m+[0m[0m disable_api_termination = (known after apply)
[32m+[0m[0m ebs_optimized = (known after apply)
[32m+[0m[0m enable_primary_ipv6 = (known after apply)
[32m+[0m[0m get_password_data = false
[32m+[0m[0m host_id = (known after apply)
[32m+[0m[0m host_resource_group_arn = (known after apply)
[32m+[0m[0m iam_instance_profile = (known after apply)
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m instance_initiated_shutdown_behavior = (known after apply)
[32m+[0m[0m instance_lifecycle = (known after apply)
[32m+[0m[0m instance_state = (known after apply)
[32m+[0m[0m instance_type = "r6i.2xlarge"
[32m+[0m[0m ipv6_address_count = (known after apply)
[32m+[0m[0m ipv6_addresses = (known after apply)
[32m+[0m[0m key_name = (known after apply)
[32m+[0m[0m monitoring = (known after apply)
[32m+[0m[0m outpost_arn = (known after apply)
[32m+[0m[0m password_data = (known after apply)
[32m+[0m[0m placement_group = (known after apply)
[32m+[0m[0m placement_partition_number = (known after apply)
[32m+[0m[0m primary_network_interface_id = (known after apply)
[32m+[0m[0m private_dns = (known after apply)
[32m+[0m[0m private_ip = (known after apply)
[32m+[0m[0m public_dns = (known after apply)
[32m+[0m[0m public_ip = (known after apply)
[32m+[0m[0m secondary_private_ips = (known after apply)
[32m+[0m[0m security_groups = (known after apply)
[32m+[0m[0m source_dest_check = true
[32m+[0m[0m spot_instance_request_id = (known after apply)
[32m+[0m[0m subnet_id = "subnet-0a1f5a9a74ed030cf"
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-worker-0-prod"
[32m+[0m[0m "Role" = "worker"
[32m+[0m[0m "Service" = "citus"
[32m+[0m[0m "WorkerIndex" = "0"
}
[32m+[0m[0m tags_all = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "ManagedBy" = "Terraform"
[32m+[0m[0m "Name" = "citus-worker-0-prod"
[32m+[0m[0m "Owner" = "Platform Team"
[32m+[0m[0m "Project" = "FastOrder"
[32m+[0m[0m "Role" = "worker"
[32m+[0m[0m "Service" = "citus"
[32m+[0m[0m "WorkerIndex" = "0"
}
[32m+[0m[0m tenancy = (known after apply)
[32m+[0m[0m user_data = "7b4bd87c9982aab7fa463c8d12e99399661f8bde"
[32m+[0m[0m user_data_base64 = (known after apply)
[32m+[0m[0m user_data_replace_on_change = false
[32m+[0m[0m vpc_security_group_ids = (known after apply)
[32m+[0m[0m ebs_block_device {
[32m+[0m[0m delete_on_termination = false
[32m+[0m[0m device_name = "/dev/sdf"
[32m+[0m[0m encrypted = true
[32m+[0m[0m iops = 3000
[32m+[0m[0m kms_key_id = (known after apply)
[32m+[0m[0m snapshot_id = (known after apply)
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-worker-0-prod-data"
}
[32m+[0m[0m tags_all = (known after apply)
[32m+[0m[0m throughput = 125
[32m+[0m[0m volume_id = (known after apply)
[32m+[0m[0m volume_size = 500
[32m+[0m[0m volume_type = "gp3"
}
[32m+[0m[0m root_block_device {
[32m+[0m[0m delete_on_termination = false
[32m+[0m[0m device_name = (known after apply)
[32m+[0m[0m encrypted = true
[32m+[0m[0m iops = (known after apply)
[32m+[0m[0m kms_key_id = (known after apply)
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-worker-0-prod-root"
}
[32m+[0m[0m tags_all = (known after apply)
[32m+[0m[0m throughput = (known after apply)
[32m+[0m[0m volume_id = (known after apply)
[32m+[0m[0m volume_size = 100
[32m+[0m[0m volume_type = "gp3"
}
}
[1m # module.citus_cluster.aws_instance.workers[1][0m will be created
[0m [32m+[0m[0m resource "aws_instance" "workers" {
[32m+[0m[0m ami = "ami-0b2aae5f4283c0df2"
[32m+[0m[0m arn = (known after apply)
[32m+[0m[0m associate_public_ip_address = (known after apply)
[32m+[0m[0m availability_zone = (known after apply)
[32m+[0m[0m cpu_core_count = (known after apply)
[32m+[0m[0m cpu_threads_per_core = (known after apply)
[32m+[0m[0m disable_api_stop = (known after apply)
[32m+[0m[0m disable_api_termination = (known after apply)
[32m+[0m[0m ebs_optimized = (known after apply)
[32m+[0m[0m enable_primary_ipv6 = (known after apply)
[32m+[0m[0m get_password_data = false
[32m+[0m[0m host_id = (known after apply)
[32m+[0m[0m host_resource_group_arn = (known after apply)
[32m+[0m[0m iam_instance_profile = (known after apply)
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m instance_initiated_shutdown_behavior = (known after apply)
[32m+[0m[0m instance_lifecycle = (known after apply)
[32m+[0m[0m instance_state = (known after apply)
[32m+[0m[0m instance_type = "r6i.2xlarge"
[32m+[0m[0m ipv6_address_count = (known after apply)
[32m+[0m[0m ipv6_addresses = (known after apply)
[32m+[0m[0m key_name = (known after apply)
[32m+[0m[0m monitoring = (known after apply)
[32m+[0m[0m outpost_arn = (known after apply)
[32m+[0m[0m password_data = (known after apply)
[32m+[0m[0m placement_group = (known after apply)
[32m+[0m[0m placement_partition_number = (known after apply)
[32m+[0m[0m primary_network_interface_id = (known after apply)
[32m+[0m[0m private_dns = (known after apply)
[32m+[0m[0m private_ip = (known after apply)
[32m+[0m[0m public_dns = (known after apply)
[32m+[0m[0m public_ip = (known after apply)
[32m+[0m[0m secondary_private_ips = (known after apply)
[32m+[0m[0m security_groups = (known after apply)
[32m+[0m[0m source_dest_check = true
[32m+[0m[0m spot_instance_request_id = (known after apply)
[32m+[0m[0m subnet_id = "subnet-02c930351cde1e9c3"
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-worker-1-prod"
[32m+[0m[0m "Role" = "worker"
[32m+[0m[0m "Service" = "citus"
[32m+[0m[0m "WorkerIndex" = "1"
}
[32m+[0m[0m tags_all = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "ManagedBy" = "Terraform"
[32m+[0m[0m "Name" = "citus-worker-1-prod"
[32m+[0m[0m "Owner" = "Platform Team"
[32m+[0m[0m "Project" = "FastOrder"
[32m+[0m[0m "Role" = "worker"
[32m+[0m[0m "Service" = "citus"
[32m+[0m[0m "WorkerIndex" = "1"
}
[32m+[0m[0m tenancy = (known after apply)
[32m+[0m[0m user_data = "7b4bd87c9982aab7fa463c8d12e99399661f8bde"
[32m+[0m[0m user_data_base64 = (known after apply)
[32m+[0m[0m user_data_replace_on_change = false
[32m+[0m[0m vpc_security_group_ids = (known after apply)
[32m+[0m[0m ebs_block_device {
[32m+[0m[0m delete_on_termination = false
[32m+[0m[0m device_name = "/dev/sdf"
[32m+[0m[0m encrypted = true
[32m+[0m[0m iops = 3000
[32m+[0m[0m kms_key_id = (known after apply)
[32m+[0m[0m snapshot_id = (known after apply)
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-worker-1-prod-data"
}
[32m+[0m[0m tags_all = (known after apply)
[32m+[0m[0m throughput = 125
[32m+[0m[0m volume_id = (known after apply)
[32m+[0m[0m volume_size = 500
[32m+[0m[0m volume_type = "gp3"
}
[32m+[0m[0m root_block_device {
[32m+[0m[0m delete_on_termination = false
[32m+[0m[0m device_name = (known after apply)
[32m+[0m[0m encrypted = true
[32m+[0m[0m iops = (known after apply)
[32m+[0m[0m kms_key_id = (known after apply)
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-worker-1-prod-root"
}
[32m+[0m[0m tags_all = (known after apply)
[32m+[0m[0m throughput = (known after apply)
[32m+[0m[0m volume_id = (known after apply)
[32m+[0m[0m volume_size = 100
[32m+[0m[0m volume_type = "gp3"
}
}
[1m # module.citus_cluster.aws_security_group.citus[0m will be created
[0m [32m+[0m[0m resource "aws_security_group" "citus" {
[32m+[0m[0m arn = (known after apply)
[32m+[0m[0m description = "Security group for Citus cluster"
[32m+[0m[0m egress = [
[32m+[0m[0m {
[32m+[0m[0m cidr_blocks = [
[32m+[0m[0m "0.0.0.0/0",
]
[32m+[0m[0m description = "Allow all outbound"
[32m+[0m[0m from_port = 0
[32m+[0m[0m ipv6_cidr_blocks = []
[32m+[0m[0m prefix_list_ids = []
[32m+[0m[0m protocol = "-1"
[32m+[0m[0m security_groups = []
[32m+[0m[0m self = false
[32m+[0m[0m to_port = 0
},
]
[32m+[0m[0m id = (known after apply)
[32m+[0m[0m ingress = [
[32m+[0m[0m {
[32m+[0m[0m cidr_blocks = [
[32m+[0m[0m "10.0.0.0/8",
]
[32m+[0m[0m description = "PgBouncer access"
[32m+[0m[0m from_port = 6432
[32m+[0m[0m ipv6_cidr_blocks = []
[32m+[0m[0m prefix_list_ids = []
[32m+[0m[0m protocol = "tcp"
[32m+[0m[0m security_groups = []
[32m+[0m[0m self = false
[32m+[0m[0m to_port = 6432
},
[32m+[0m[0m {
[32m+[0m[0m cidr_blocks = [
[32m+[0m[0m "10.0.0.0/8",
]
[32m+[0m[0m description = "PostgreSQL access"
[32m+[0m[0m from_port = 5432
[32m+[0m[0m ipv6_cidr_blocks = []
[32m+[0m[0m prefix_list_ids = []
[32m+[0m[0m protocol = "tcp"
[32m+[0m[0m security_groups = []
[32m+[0m[0m self = false
[32m+[0m[0m to_port = 5432
},
[32m+[0m[0m {
[32m+[0m[0m cidr_blocks = [
[32m+[0m[0m "10.0.0.0/8",
]
[32m+[0m[0m description = "SSH access"
[32m+[0m[0m from_port = 22
[32m+[0m[0m ipv6_cidr_blocks = []
[32m+[0m[0m prefix_list_ids = []
[32m+[0m[0m protocol = "tcp"
[32m+[0m[0m security_groups = []
[32m+[0m[0m self = false
[32m+[0m[0m to_port = 22
},
[32m+[0m[0m {
[32m+[0m[0m cidr_blocks = []
[32m+[0m[0m description = "Internal cluster communication"
[32m+[0m[0m from_port = 0
[32m+[0m[0m ipv6_cidr_blocks = []
[32m+[0m[0m prefix_list_ids = []
[32m+[0m[0m protocol = "tcp"
[32m+[0m[0m security_groups = []
[32m+[0m[0m self = true
[32m+[0m[0m to_port = 65535
},
]
[32m+[0m[0m name = (known after apply)
[32m+[0m[0m name_prefix = "citus-prod-"
[32m+[0m[0m owner_id = (known after apply)
[32m+[0m[0m revoke_rules_on_delete = false
[32m+[0m[0m tags = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "Name" = "citus-prod"
[32m+[0m[0m "Service" = "citus"
}
[32m+[0m[0m tags_all = {
[32m+[0m[0m "Backup" = "Required"
[32m+[0m[0m "CostCenter" = "Platform"
[32m+[0m[0m "Environment" = "prod"
[32m+[0m[0m "ManagedBy" = "Terraform"
[32m+[0m[0m "Name" = "citus-prod"
[32m+[0m[0m "Owner" = "Platform Team"
[32m+[0m[0m "Project" = "FastOrder"
[32m+[0m[0m "Service" = "citus"
}
[32m+[0m[0m vpc_id = "vpc-0af7da1e7d94d62bd"
}
[1mPlan:[0m 9 to add, 0 to change, 0 to destroy.
[0m
Changes to Outputs:
[32m+[0m[0m connection_string = (sensitive value)
[32m+[0m[0m coordinator_ip = (known after apply)
[32m+[0m[0m worker_ips = [
[32m+[0m[0m (known after apply),
[32m+[0m[0m (known after apply),
]
[90m
─────────────────────────────────────────────────────────────────────────────[0m
Saved the plan to: tfplan
To perform exactly these actions, run the following command to apply:
terraform apply "tfplan"
[0;32m[2026-01-18_23:21:57] ✓[0m Terraform plan succeeded
[0;36m[2026-01-18_23:21:57][0m Generating plan JSON...
[0;32m[2026-01-18_23:22:00] ✓[0m Terraform provisioning step completed successfully
Next step: Review the plan and apply with 'terraform apply tfplan'
[INFO] FastOrder Environment Preparation
[INFO] Service: user
[INFO] Zone: sau
[INFO] Environment: dev
[INFO] Branch: main
[INFO] State Directory: /opt/fastorder/bash/scripts/env_app_setup/state
[INFO] Library: /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] IP: 142.93.238.16 (specified)
[INFO] Creating environment using fo-env...
[INFO] Creating new FastOrder environment (v1 topology)
[INFO] Generated environment ID: user-sau-main-dev
[INFO] Using provided IP: 142.93.238.16
[INFO] Allocated interface: eth0:16
[INFO] Configuring network interface for VM IP: 142.93.238.16
[INFO] VM IP 142.93.238.16 is already configured on eth0:16
[CONFIG] No web configuration found for environment: user-sau-main-dev
[CONFIG] Using defaults: ES_NODES=1, PG_WORKERS=1
[INFO] Service enabled flags: db=yes, eventbus=yes, search=yes
[ OK ] Created topology.json at /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
[ OK ] Generated overlay configurations in /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/generated/
[ OK ] Updated environments.json
[ OK ] Updated setup.json
[ OK ] Environment created successfully!
[INFO]
[INFO] Environment Details:
[INFO] ID: user-sau-main-dev
[INFO] Service: user
[INFO] zone: sau
[INFO] Environment: dev
[INFO] Branch: main
[INFO] IP: 142.93.238.16
[INFO] Interface: eth0:16
[INFO]
[INFO] Configuration files:
[INFO] Topology: /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
[INFO] Generated: /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/generated/*.env
[INFO] Overrides: /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/overrides/*.env
[INFO]
[INFO] To use this environment:
[INFO] export ENV_ID="user-sau-main-dev"
[INFO] source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO] init_environment
[ OK ] Environment preparation completed successfully!
[INFO] Creating topology from web form submission...
[INFO] Using environment from web interface: user-sau-main-dev
[0;32m[2026-01-18 23:22:02][0m Using web-provided environment: user-sau-main-dev
[0;32m[2026-01-18 23:22:02][0m Service: user, Zone: sau, Branch: main, Env: dev
[ OK ] Environment initialized successfully (mode: general)
[INFO] Creating topology.json from web form submission...
[INFO] DEBUG: Service enabled flags...
[INFO] DB_ENABLED=yes
[INFO] EVENTBUS_ENABLED=yes
[INFO] SEARCH_ENABLED=yes
[INFO] DEBUG: Checking for form submission variables...
[INFO] service_es_ip=10.100.1.4
[INFO] service_es_fqdn=search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com
[INFO] service_pg_coordinator_ip=10.100.1.14
[WARN] IP 10.100.1.4 is already allocated, allocating new IP for search
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Adding search: search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.89) [reallocated from 10.100.1.4]
[WARN] IP 10.100.1.6 is already allocated, allocating new IP for search-node-01
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Adding search-node-01: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.152) [reallocated from 10.100.1.6]
[WARN] IP 10.100.1.8 is already allocated, allocating new IP for eventbus-broker-01
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Adding eventbus-broker-01: eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com (10.100.1.131) [reallocated from 10.100.1.8]
[WARN] IP 10.100.1.10 is already allocated, allocating new IP for eventbus-connect
[INFO] Adding eventbus-connect: eventbus-user-sau-main-dev-kafka-connect.fastorder.com (10.100.1.183) [reallocated from 10.100.1.10]
[WARN] IP 10.100.1.12 is already allocated, allocating new IP for schema-registry
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Adding schema-registry: schema-user-sau-main-dev-kafka-registry.fastorder.com (10.100.1.93) [reallocated from 10.100.1.12]
[WARN] IP 10.100.1.14 is already allocated, allocating new IP for pg-coordinator
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Adding pg-coordinator: db-user-sau-main-dev-postgresql-coordinator.fastorder.com (10.100.1.95) [reallocated from 10.100.1.14]
[WARN] IP 10.100.1.16 is already allocated, allocating new IP for pgbouncer
[INFO] Adding pgbouncer: db-user-sau-main-dev-postgresql-bouncer.fastorder.com (10.100.1.184) [reallocated from 10.100.1.16]
[WARN] IP 10.100.1.18 is already allocated, allocating new IP for obs
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Adding obs: obs-user-sau-main-dev.fastorder.com (10.100.1.166) [reallocated from 10.100.1.18]
[ OK ] Topology created from form data
[INFO] Applications registered:
✓ eventbus-broker-01: eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com (10.100.1.131)
✓ eventbus-connect: eventbus-user-sau-main-dev-kafka-connect.fastorder.com (10.100.1.183)
✓ obs: obs-user-sau-main-dev.fastorder.com (10.100.1.166)
✓ pg-coordinator: db-user-sau-main-dev-postgresql-coordinator.fastorder.com (10.100.1.95)
✓ pgbouncer: db-user-sau-main-dev-postgresql-bouncer.fastorder.com (10.100.1.184)
✓ schema-registry: schema-user-sau-main-dev-kafka-registry.fastorder.com (10.100.1.93)
✓ search: search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.89)
✓ search-node-01: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.152)
[ OK ] Topology created from form data
[INFO] Next steps:
[INFO] 1. Review the generated topology.json and configurations
[INFO] 2. Customize overrides/*.env files if needed
[INFO] 3. Run subsequent installation steps (02-install-postgresql, etc.)
[INFO] To use this environment in other scripts:
[INFO] export ENV_ID="$(fo-env list | tail -n1 | awk '{print $1}')"
[INFO] source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO] init_environment
⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 🚀 OBSERVABILITY CELL PROVISIONING STARTED
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Script: 02-observability-cell/run.sh
[0;34m[INFO][0m Timestamp: 2026-01-18 23:22:10 UTC
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Ensuring correct permissions for observability deployment...
[2026-01-18 23:22:10 UTC] USER=www-data EUID=0 PID=3976930 ACTION=fsop ARGS=chmod 775 /var/log/fastorder
[2026-01-18 23:22:10 UTC] USER=www-data EUID=0 PID=3976939 ACTION=fsop ARGS=chown www-data:www-data /var/log/fastorder
[2026-01-18 23:22:10 UTC] USER=www-data EUID=0 PID=3976948 ACTION=fsop ARGS=touch /var/log/fastorder/provisioning-elevated.log
[2026-01-18 23:22:10 UTC] USER=www-data EUID=0 PID=3976957 ACTION=fsop ARGS=chmod 666 /var/log/fastorder/provisioning-elevated.log
[2026-01-18 23:22:10 UTC] USER=www-data EUID=0 PID=3976966 ACTION=fsop ARGS=chown www-data:www-data /var/log/fastorder/provisioning-elevated.log
[0;32m[OK][0m Log directory: /var/log/fastorder (775)
[0;32m[OK][0m Log file: provisioning-elevated.log (666)
[2026-01-18 23:22:10 UTC] USER=www-data EUID=0 PID=3976975 ACTION=fsop ARGS=chmod 775 /opt/fastorder/bash/scripts/env_app_setup/state
[0;32m[OK][0m State directory: 775
[2026-01-18 23:22:10 UTC] USER=www-data EUID=0 PID=3976984 ACTION=fsop ARGS=mkdir -p /etc/fastorder/observability/certs
[2026-01-18 23:22:10 UTC] USER=www-data EUID=0 PID=3976993 ACTION=fsop ARGS=chmod 750 /etc/fastorder/observability/certs
[0;32m[OK][0m Cert directory: /etc/fastorder/observability/certs (750 - secure)
[0;32m[OK][0m Lib scripts: executable (755)
[0;32m[OK][0m All deployment scripts: executable (755)
[0;32m[OK][0m All directories: accessible (755)
[0;32m[OK][0m ✅ All permissions verified and fixed
[0;34m[CREDS][0m Using AWS credentials from: /var/www/.aws/credentials
[0;34m[CREDS][0m Credential management library loaded (region: me-central-1)
[INFO] Using web-provided environment: user-sau-main-dev
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
═══════════════════════════════════════════════════════════════════════════════
OBSERVABILITY CELL PROVISIONING
═══════════════════════════════════════════════════════════════════════════════
[INFO] Application Cell: user-sau-main-dev
[INFO] Observability Cell: obs-user-sau-main-dev
[INFO] Service: user | Zone: sau | Env: dev
[INFO] Step 1/10: Provisioning network infrastructure...
[INFO] Using existing IP for obs: 10.100.1.166
[INFO] Allocated new IP for metrics: 10.100.1.187
[2026-01-18 23:22:11 UTC] USER=www-data EUID=0 PID=3977508 ACTION=fsop ARGS=cp /tmp/tmp.3cZdv0tT4L /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
[INFO] Allocated new IP for dashboards: 10.100.1.188
[2026-01-18 23:22:11 UTC] USER=www-data EUID=0 PID=3977527 ACTION=fsop ARGS=cp /tmp/tmp.CDHUQwNSSZ /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
[INFO] Allocated new IP for logstore: 10.100.1.217
[2026-01-18 23:22:12 UTC] USER=www-data EUID=0 PID=3977600 ACTION=fsop ARGS=cp /tmp/tmp.VMAJAlMiPq /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
[INFO] Allocated new IP for traces: 10.100.1.227
[2026-01-18 23:22:12 UTC] USER=www-data EUID=0 PID=3977636 ACTION=fsop ARGS=cp /tmp/tmp.bEVUYXDKFV /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
[INFO] Allocated new IP for alerts: 10.100.1.228
[2026-01-18 23:22:12 UTC] USER=www-data EUID=0 PID=3977653 ACTION=fsop ARGS=cp /tmp/tmp.SI44YhxIoS /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
[INFO] Allocated new IP for telemetry: 10.100.1.229
[2026-01-18 23:22:12 UTC] USER=www-data EUID=0 PID=3977670 ACTION=fsop ARGS=cp /tmp/tmp.NqGqa6mY40 /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
[INFO] Allocated observability IPs:
[INFO] metrics: 10.100.1.187
[INFO] alerts: 10.100.1.228
[INFO] dashboards: 10.100.1.188
[INFO] traces: 10.100.1.227
[INFO] telemetry: 10.100.1.229
[INFO] logstore: 10.100.1.217
[INFO] proxy: 10.100.1.166
[INFO] obs: 10.100.1.166
[ OK ] Network infrastructure allocated
[INFO] Cleaning up ports from previous environments...
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking and cleaning ports for observability cell: obs-user-sau-main-dev
[0;34m[INFO][0m IP Address: 10.100.1.166
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking for conflicting observability services...
[0;34m[INFO][0m Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Found 5 observability service(s) (all belong to current cell)
[0;34m[INFO][0m Checking for remaining processes on IP 10.100.1.166...
[0;34m[INFO][0m Scanning 15 ports...
[0;32m[OK][0m ✅ All 15 ports are FREE - ready for installation
[0;32m[OK][0m Port cleanup completed successfully
[0;34m[INFO][0m Configuring IP aliases on network interface...
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING NETWORK IP ALIASES
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Interface: lo
[0;34m[INFO][0m IP Count: 8
[0;34m[INFO][0m Configuring: metrics → 10.100.1.187
[0;34m[INFO][0m IP 10.100.1.187 already configured on network interface
[0;34m[INFO][0m Configuring: alerts → 10.100.1.228
[0;34m[INFO][0m IP 10.100.1.228 already configured on network interface
[0;34m[INFO][0m Configuring: dashboards → 10.100.1.188
[0;34m[INFO][0m IP 10.100.1.188 already configured on network interface
[0;34m[INFO][0m Configuring: traces → 10.100.1.227
[0;34m[INFO][0m IP 10.100.1.227 already configured on network interface
[0;34m[INFO][0m Configuring: telemetry → 10.100.1.229
[0;34m[INFO][0m IP 10.100.1.229 already configured on network interface
[0;34m[INFO][0m Configuring: logstore → 10.100.1.217
[0;34m[INFO][0m IP 10.100.1.217 already configured on network interface
[0;34m[INFO][0m Configuring: proxy → 10.100.1.166
[0;34m[INFO][0m Configuring IP alias: 10.100.1.166/32 on lo
[0;32m[OK][0m ✅ IP 10.100.1.166 configured successfully on lo
[0;32m[OK][0m ✅ IP 10.100.1.166 verified on network interface
[0;34m[INFO][0m Configuring: obs → 10.100.1.166
[0;34m[INFO][0m IP 10.100.1.166 already configured on network interface
[0;32m[OK][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ All IP aliases configured successfully
[0;32m[OK][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Current IP configuration on lo:
inet 127.0.0.1/8 scope host lo
inet 10.100.1.204/32 scope global lo:pgbouncer
inet 10.100.1.192/32 scope global lo:pgbouncer
inet 10.100.60.2/32 scope global lo
inet 10.100.1.155/32 scope global lo
inet 10.100.1.183/32 scope global lo
inet 10.100.1.187/32 scope global lo
inet 10.100.1.217/32 scope global lo
inet 10.100.1.184/32 scope global lo
inet 10.100.1.111/32 scope global lo
inet 10.100.1.181/32 scope global lo
inet 10.100.1.221/32 scope global lo
inet 10.100.1.225/32 scope global lo
inet 10.100.1.222/32 scope global lo
inet 10.100.1.224/32 scope global lo
inet 10.100.1.226/32 scope global lo
inet 10.100.1.223/32 scope global lo
inet 10.100.1.220/32 scope global lo
inet 10.100.1.219/32 scope global lo
inet 10.100.1.228/32 scope global lo
inet 10.100.1.232/32 scope global lo
inet 10.100.1.229/32 scope global lo
inet 10.100.1.231/32 scope global lo
inet 10.100.1.233/32 scope global lo
inet 10.100.1.230/32 scope global lo
inet 10.100.1.81/32 scope global lo
inet 10.100.1.122/32 scope global lo:pgbouncer
inet 10.100.1.236/32 scope global lo
inet 10.100.1.237/32 scope global lo
inet 10.100.1.227/32 scope global lo
inet 10.100.1.172/32 scope global lo
inet 10.100.1.91/32 scope global lo
inet 10.100.1.188/32 scope global lo:pgbouncer
inet 10.100.1.95/32 scope global lo:pgbouncer
inet 10.100.1.61/32 scope global lo
inet 10.100.1.90/32 scope global lo
inet 10.100.1.145/32 scope global lo
inet 10.100.1.177/32 scope global lo:pgbouncer
inet 10.100.1.50/32 scope global lo
inet 10.100.1.235/32 scope global lo
inet 10.100.1.73/32 scope global lo:pgbouncer
inet 10.100.1.58/32 scope global lo
inet 10.100.1.166/32 scope global lo
[0;32m[OK][0m IP aliases configured on network interface
[0;34m[INFO][0m Step 2/10: Creating DNS entries...
[0;34m[INFO][0m Configuring DNS entries in /etc/hosts...
[0;34m[INFO][0m Added: metrics-user-sau-main-dev-prometheus.fastorder.com → 10.100.1.187
[0;34m[INFO][0m Added: alerts-user-sau-main-dev-alertmanager.fastorder.com → 10.100.1.228
[0;34m[INFO][0m Added: dashboards-user-sau-main-dev-grafana.fastorder.com → 10.100.1.188
[0;34m[INFO][0m Added: traces-user-sau-main-dev-tempo.fastorder.com → 10.100.1.227
[0;34m[INFO][0m Added: telemetry-user-sau-main-dev-opentelemetry.fastorder.com → 10.100.1.229
[0;34m[INFO][0m Added: logstore-user-sau-main-dev-clickhouse.fastorder.com → 10.100.1.217
[0;34m[INFO][0m Added: observe-user-sau-main-dev.fastorder.com → 10.100.1.166
[0;34m[INFO][0m Adding observability integration aliases...
[0;34m[INFO][0m Added alias: metrics-user-sau-main-dev.fastorder.com → 10.100.1.187
[0;34m[INFO][0m Added alias: alerts-user-sau-main-dev.fastorder.com → 10.100.1.228
[0;34m[INFO][0m Added alias: dashboards-user-sau-main-dev.fastorder.com → 10.100.1.188
[0;34m[INFO][0m Added alias: traces-user-sau-main-dev.fastorder.com → 10.100.1.227
[0;34m[INFO][0m Added alias: telemetry-user-sau-main-dev.fastorder.com → 10.100.1.229
[0;34m[INFO][0m Added alias: logstore-user-sau-main-dev.fastorder.com → 10.100.1.217
[2026-01-18 23:22:13 UTC] USER=www-data EUID=0 PID=3977978 ACTION=fsop ARGS=sed -i /observe-user-sau-main-dev.fastorder.com/d /etc/hosts
[0;34m[INFO][0m Added alias: observe-user-sau-main-dev.fastorder.com → 10.100.1.166
[0;32m[OK][0m DNS entries created
[0;34m[INFO][0m Step 3/10: Creating AWS Secrets Manager structure...
[INFO] Creating AWS Secrets Manager structure
[INFO] Base path: fastorder/observability/user/sau/dev
[INFO] Observability Cell: obs-user-sau-main-dev
[INFO] Application Cell: user-sau-main-dev
[INFO] Exists: fastorder/observability/user/sau/dev/metrics
[INFO] Exists: fastorder/observability/user/sau/dev/dashboards
[INFO] Exists: fastorder/observability/user/sau/dev/logstore
[INFO] Exists: fastorder/observability/user/sau/dev/traces
[INFO] Exists: fastorder/observability/user/sau/dev/telemetry
[INFO] Exists: fastorder/observability/user/sau/dev/alerts
[INFO] Secrets structure created successfully
[0;32m[OK][0m Secrets structure created
[0;34m[INFO][0m Step 4/10: Generating mTLS certificates...
[INFO] Generating mTLS certificates for observability cell
[INFO] Observability Cell: obs-user-sau-main-dev
[INFO] Components: prometheus,grafana,loki,tempo,otlp_collector,clickhouse,alertmanager
[INFO] Creating certificate directory: /etc/fastorder/observability/certs/obs-user-sau-main-dev
[2026-01-18 23:22:21 UTC] USER=www-data EUID=0 PID=3978169 ACTION=fsop ARGS=mkdir -p /etc/fastorder/observability/certs/obs-user-sau-main-dev
[2026-01-18 23:22:21 UTC] USER=www-data EUID=0 PID=3978180 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-user-sau-main-dev
[INFO] Generating CA certificate for obs-user-sau-main-dev
[2026-01-18 23:22:21 UTC] USER=www-data EUID=0 PID=3978199 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-key.pem 4096
[2026-01-18 23:22:23 UTC] USER=www-data EUID=0 PID=3978240 ACTION=fsop ARGS=openssl req -new -x509 -days 3650 -key /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-key.pem -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=obs-user-sau-main-dev-ca
[2026-01-18 23:22:23 UTC] USER=www-data EUID=0 PID=3978256 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-key.pem
[2026-01-18 23:22:23 UTC] USER=www-data EUID=0 PID=3978265 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem
[INFO] CA certificate created: /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem
[INFO] Generating certificate for: prometheus
[2026-01-18 23:22:23 UTC] USER=www-data EUID=0 PID=3978274 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/prometheus-key.pem 2048
[2026-01-18 23:22:23 UTC] USER=www-data EUID=0 PID=3978283 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-sau-main-dev/prometheus-key.pem -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/prometheus-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=prometheus.obs-user-sau-main-dev
[2026-01-18 23:22:23 UTC] USER=www-data EUID=0 PID=3978292 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-sau-main-dev/prometheus-csr.pem -CA /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/prometheus-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = prometheus.obs-user-sau-main-dev
[2026-01-18 23:22:23 UTC] USER=www-data EUID=0 PID=3978301 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-user-sau-main-dev/prometheus-key.pem
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978310 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/prometheus-cert.pem
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978319 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-user-sau-main-dev/prometheus-csr.pem
[INFO] Certificate created: /etc/fastorder/observability/certs/obs-user-sau-main-dev/prometheus-cert.pem
[INFO] Generating certificate for: grafana
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978328 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/grafana-key.pem 2048
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978337 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-sau-main-dev/grafana-key.pem -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/grafana-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=grafana.obs-user-sau-main-dev
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978346 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-sau-main-dev/grafana-csr.pem -CA /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/grafana-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = grafana.obs-user-sau-main-dev
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978355 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-user-sau-main-dev/grafana-key.pem
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978364 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/grafana-cert.pem
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978373 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-user-sau-main-dev/grafana-csr.pem
[INFO] Certificate created: /etc/fastorder/observability/certs/obs-user-sau-main-dev/grafana-cert.pem
[INFO] Generating certificate for: loki
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978382 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/loki-key.pem 2048
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978393 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-sau-main-dev/loki-key.pem -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/loki-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=loki.obs-user-sau-main-dev
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978402 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-sau-main-dev/loki-csr.pem -CA /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/loki-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = loki.obs-user-sau-main-dev
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978411 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-user-sau-main-dev/loki-key.pem
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978420 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/loki-cert.pem
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978429 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-user-sau-main-dev/loki-csr.pem
[INFO] Certificate created: /etc/fastorder/observability/certs/obs-user-sau-main-dev/loki-cert.pem
[INFO] Generating certificate for: tempo
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978438 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/tempo-key.pem 2048
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978448 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-sau-main-dev/tempo-key.pem -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/tempo-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=tempo.obs-user-sau-main-dev
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978457 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-sau-main-dev/tempo-csr.pem -CA /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/tempo-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = tempo.obs-user-sau-main-dev
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978467 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-user-sau-main-dev/tempo-key.pem
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978476 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/tempo-cert.pem
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978485 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-user-sau-main-dev/tempo-csr.pem
[INFO] Certificate created: /etc/fastorder/observability/certs/obs-user-sau-main-dev/tempo-cert.pem
[INFO] Generating certificate for: otlp_collector
[2026-01-18 23:22:24 UTC] USER=www-data EUID=0 PID=3978494 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/otlp_collector-key.pem 2048
[2026-01-18 23:22:25 UTC] USER=www-data EUID=0 PID=3978528 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-sau-main-dev/otlp_collector-key.pem -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/otlp_collector-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=otlp_collector.obs-user-sau-main-dev
[2026-01-18 23:22:25 UTC] USER=www-data EUID=0 PID=3978545 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-sau-main-dev/otlp_collector-csr.pem -CA /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/otlp_collector-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = otlp_collector.obs-user-sau-main-dev
[2026-01-18 23:22:25 UTC] USER=www-data EUID=0 PID=3978563 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-user-sau-main-dev/otlp_collector-key.pem
[2026-01-18 23:22:25 UTC] USER=www-data EUID=0 PID=3978572 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/otlp_collector-cert.pem
[2026-01-18 23:22:25 UTC] USER=www-data EUID=0 PID=3978582 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-user-sau-main-dev/otlp_collector-csr.pem
[INFO] Certificate created: /etc/fastorder/observability/certs/obs-user-sau-main-dev/otlp_collector-cert.pem
[INFO] Generating certificate for: clickhouse
[2026-01-18 23:22:25 UTC] USER=www-data EUID=0 PID=3978592 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/clickhouse-key.pem 2048
[2026-01-18 23:22:25 UTC] USER=www-data EUID=0 PID=3978602 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-sau-main-dev/clickhouse-key.pem -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/clickhouse-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=clickhouse.obs-user-sau-main-dev
[2026-01-18 23:22:25 UTC] USER=www-data EUID=0 PID=3978611 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-sau-main-dev/clickhouse-csr.pem -CA /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/clickhouse-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = clickhouse.obs-user-sau-main-dev
[2026-01-18 23:22:25 UTC] USER=www-data EUID=0 PID=3978620 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-user-sau-main-dev/clickhouse-key.pem
[2026-01-18 23:22:25 UTC] USER=www-data EUID=0 PID=3978629 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/clickhouse-cert.pem
[2026-01-18 23:22:25 UTC] USER=www-data EUID=0 PID=3978638 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-user-sau-main-dev/clickhouse-csr.pem
[INFO] Certificate created: /etc/fastorder/observability/certs/obs-user-sau-main-dev/clickhouse-cert.pem
[INFO] Generating certificate for: alertmanager
[2026-01-18 23:22:25 UTC] USER=www-data EUID=0 PID=3978647 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/alertmanager-key.pem 2048
[2026-01-18 23:22:26 UTC] USER=www-data EUID=0 PID=3978661 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-sau-main-dev/alertmanager-key.pem -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/alertmanager-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Observability/CN=alertmanager.obs-user-sau-main-dev
[2026-01-18 23:22:26 UTC] USER=www-data EUID=0 PID=3978670 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-sau-main-dev/alertmanager-csr.pem -CA /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/alertmanager-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Observability, CN = alertmanager.obs-user-sau-main-dev
[2026-01-18 23:22:26 UTC] USER=www-data EUID=0 PID=3978679 ACTION=fsop ARGS=chmod 600 /etc/fastorder/observability/certs/obs-user-sau-main-dev/alertmanager-key.pem
[2026-01-18 23:22:26 UTC] USER=www-data EUID=0 PID=3978688 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/alertmanager-cert.pem
[INFO] Certificate created: /etc/fastorder/observability/certs/obs-user-sau-main-dev/alertmanager-cert.pem
[INFO] Generating PHP client certificate for metrics service...
[2026-01-18 23:22:26 UTC] USER=www-data EUID=0 PID=3978707 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/php-client-key.pem 2048
[2026-01-18 23:22:26 UTC] USER=www-data EUID=0 PID=3978744 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-sau-main-dev/php-client-key.pem -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/php-client-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=Dashboard/CN=php-metrics-client.obs-user-sau-main-dev
[2026-01-18 23:22:26 UTC] USER=www-data EUID=0 PID=3978754 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-sau-main-dev/php-client-csr.pem -CA /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/php-client-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = Dashboard, CN = php-metrics-client.obs-user-sau-main-dev
[2026-01-18 23:22:27 UTC] USER=www-data EUID=0 PID=3978763 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-user-sau-main-dev/php-client-key.pem
[2026-01-18 23:22:27 UTC] USER=www-data EUID=0 PID=3978772 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/php-client-cert.pem
[2026-01-18 23:22:27 UTC] USER=www-data EUID=0 PID=3978781 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-user-sau-main-dev/php-client-key.pem
[2026-01-18 23:22:27 UTC] USER=www-data EUID=0 PID=3978790 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-user-sau-main-dev/php-client-cert.pem
[2026-01-18 23:22:27 UTC] USER=www-data EUID=0 PID=3978799 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-user-sau-main-dev/php-client-csr.pem
[INFO] PHP client certificate created: /etc/fastorder/observability/certs/obs-user-sau-main-dev/php-client-cert.pem
[INFO] Generating Apache client certificate for mTLS reverse proxy...
[2026-01-18 23:22:27 UTC] USER=www-data EUID=0 PID=3978808 ACTION=fsop ARGS=openssl genrsa -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/apache-client-key.pem 2048
[2026-01-18 23:22:27 UTC] USER=www-data EUID=0 PID=3978823 ACTION=fsop ARGS=openssl req -new -key /etc/fastorder/observability/certs/obs-user-sau-main-dev/apache-client-key.pem -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/apache-client-csr.pem -subj /C=US/ST=State/L=City/O=FastOrder/OU=ReverseProxy/CN=apache-proxy.obs-user-sau-main-dev
[2026-01-18 23:22:27 UTC] USER=www-data EUID=0 PID=3978833 ACTION=fsop ARGS=openssl x509 -req -in /etc/fastorder/observability/certs/obs-user-sau-main-dev/apache-client-csr.pem -CA /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem -CAkey /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-key.pem -CAcreateserial -out /etc/fastorder/observability/certs/obs-user-sau-main-dev/apache-client-cert.pem -days 730
Certificate request self-signature ok
subject=C = US, ST = State, L = City, O = FastOrder, OU = ReverseProxy, CN = apache-proxy.obs-user-sau-main-dev
[2026-01-18 23:22:27 UTC] USER=www-data EUID=0 PID=3978861 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-user-sau-main-dev/apache-client-key.pem
[2026-01-18 23:22:27 UTC] USER=www-data EUID=0 PID=3978874 ACTION=fsop ARGS=chmod 640 /etc/fastorder/observability/certs/obs-user-sau-main-dev/apache-client-combined.pem
[2026-01-18 23:22:27 UTC] USER=www-data EUID=0 PID=3978891 ACTION=fsop ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/apache-client-cert.pem
[2026-01-18 23:22:27 UTC] USER=www-data EUID=0 PID=3978900 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-user-sau-main-dev/apache-client-key.pem
[2026-01-18 23:22:27 UTC] USER=www-data EUID=0 PID=3978909 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-user-sau-main-dev/apache-client-cert.pem
[2026-01-18 23:22:27 UTC] USER=www-data EUID=0 PID=3978918 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/observability/certs/obs-user-sau-main-dev/apache-client-combined.pem
[2026-01-18 23:22:27 UTC] USER=www-data EUID=0 PID=3978927 ACTION=fsop ARGS=rm -f /etc/fastorder/observability/certs/obs-user-sau-main-dev/apache-client-csr.pem
[INFO] Apache client certificate created: /etc/fastorder/observability/certs/obs-user-sau-main-dev/apache-client-cert.pem
[INFO] Apache combined cert+key: /etc/fastorder/observability/certs/obs-user-sau-main-dev/apache-client-combined.pem
[INFO] Storing mTLS certificates in AWS Secrets Manager...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/user/sau/main/dev/mtls/php-client-JEBoMj",
"Name": "fastorder/observability/user/sau/main/dev/mtls/php-client",
"VersionId": "a255ee33-7a23-480c-a841-33cd034b1b54"
}
[INFO] mTLS certificates stored in Secrets Manager: fastorder/observability/user/sau/main/dev/mtls/php-client
[INFO] mTLS certificates generated successfully
[INFO] Certificate directory: /etc/fastorder/observability/certs/obs-user-sau-main-dev
[INFO] PHP client cert: /etc/fastorder/observability/certs/obs-user-sau-main-dev/php-client-cert.pem
[INFO] PHP client key: /etc/fastorder/observability/certs/obs-user-sau-main-dev/php-client-key.pem
[INFO] Apache client cert: /etc/fastorder/observability/certs/obs-user-sau-main-dev/apache-client-cert.pem
[INFO] Apache combined (for SSLProxyMachineCertificateFile): /etc/fastorder/observability/certs/obs-user-sau-main-dev/apache-client-combined.pem
[0;32m[OK][0m mTLS certificates generated
[0;34m[INFO][0m Step 5/10: Deploying log storage backend...
[0;34m[INFO][0m Provider: clickhouse (selected)
[0;34m[INFO][0m Note: Deployed before telemetry (OtelCol depends on log storage)
[0;34m[INFO][0m FQDN: logstore-user-sau-main-dev-clickhouse.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.217
[0;34m[INFO][0m Deploying log backend: clickhouse...
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m LOG STORAGE BACKEND DEPLOYMENT
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: clickhouse
[0;34m[INFO][0m Observability Cell: obs-user-sau-main-dev
[0;34m[INFO][0m FQDN: logstore-user-sau-main-dev-clickhouse.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.217
[0;34m[INFO][0m S3 Bucket: fastorder-logs-sau-dev
[0;34m[INFO][0m Retention: 90 days
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[2026-01-18 23:22:30 UTC] USER=unknown EUID=33 PID=3978994 ACTION=fsop ARGS=chmod +x /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh
/bin/chmod: changing permissions of '/opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh': Operation not permitted
[0;34m[INFO][0m Using provider: clickhouse
[0;34m[INFO][0m Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/clickhouse.sh
[0;34m[INFO][0m Executing provider deployment script...
[0;34m[INFO][0m Parsed: SERVICE=user, ZONE=sau, BRANCH=main, ENV=dev
[0;34m[INFO][0m Checking and cleaning ports before installation...
[0;34m[INFO][0m Initializing certificate directory for obs-user-sau-main-dev...
[2026-01-18 23:22:30 UTC] USER=www-data EUID=0 PID=3979011 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-18 23:22:30 UTC] USER=www-data EUID=0 PID=3979020 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-18 23:22:30 UTC] USER=www-data EUID=0 PID=3979029 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-18 23:22:30 UTC] USER=www-data EUID=0 PID=3979038 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-user-sau-main-dev
[0;32m[OK][0m Certificate directory initialized: /etc/fastorder/observability/certs/obs-user-sau-main-dev
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking and cleaning ports for observability cell: obs-user-sau-main-dev
[0;34m[INFO][0m IP Address: 10.100.1.217
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking for conflicting observability services...
[0;34m[INFO][0m Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Found 5 observability service(s) (all belong to current cell)
[0;34m[INFO][0m Checking for remaining processes on IP 10.100.1.217...
[0;34m[INFO][0m Scanning 15 ports...
[0;32m[OK][0m ✅ All 15 ports are FREE - ready for installation
[0;32m[OK][0m Port cleanup successful on attempt 1
[0;34m[INFO][0m Binding ClickHouse to allocated IP: 10.100.1.217
[0;34m[INFO][0m Deploying ClickHouse for obs-user-sau-main-dev
[0;34m[INFO][0m FQDN: logstore-user-sau-main-dev-clickhouse.fastorder.com
[0;34m[INFO][0m Allocated IP: 10.100.1.217
[0;34m[INFO][0m VM IP: 10.100.1.217
[0;34m[INFO][0m Ports: HTTP=8123 TCP=9000 Interserver=9009
[0;34m[INFO][0m S3 Bucket: fastorder-logs-sau-dev (region=me-central-1)
[0;34m[INFO][0m Retention: 90 days
[0;34m[INFO][0m Checking if ClickHouse is installed...
[0;32m[OK][0m ClickHouse already installed
[2026-01-18 23:22:30 UTC] USER=www-data EUID=0 PID=3979159 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-server-obs-user-sau-main-dev/config.d
[2026-01-18 23:22:30 UTC] USER=www-data EUID=0 PID=3979169 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-server-obs-user-sau-main-dev/users.d
[2026-01-18 23:22:30 UTC] USER=www-data EUID=0 PID=3979178 ACTION=fsop ARGS=mkdir -p /var/lib/clickhouse-obs-user-sau-main-dev
[2026-01-18 23:22:30 UTC] USER=www-data EUID=0 PID=3979187 ACTION=fsop ARGS=mkdir -p /var/log/clickhouse-server-obs-user-sau-main-dev
[2026-01-18 23:22:30 UTC] USER=www-data EUID=0 PID=3979196 ACTION=passthru ARGS=chmod 755 /etc/clickhouse-server-obs-user-sau-main-dev
[2026-01-18 23:22:30 UTC] USER=www-data EUID=0 PID=3979205 ACTION=passthru ARGS=chmod 700 /var/lib/clickhouse-obs-user-sau-main-dev
[2026-01-18 23:22:30 UTC] USER=www-data EUID=0 PID=3979214 ACTION=passthru ARGS=chmod 750 /var/log/clickhouse-server-obs-user-sau-main-dev
[0;34m[INFO][0m Found existing logs_writer credentials in Secrets Manager - reusing to maintain sync
[0;34m[INFO][0m Found existing metrics_reader credentials in Secrets Manager - reusing to maintain sync
[0;34m[INFO][0m TLS configuration exported for clickhouse
[0;34m[INFO][0m Cert: /etc/fastorder/observability/certs/obs-user-sau-main-dev/clickhouse-cert.pem
[0;34m[INFO][0m Key: /etc/fastorder/observability/certs/obs-user-sau-main-dev/clickhouse-key.pem
[0;34m[INFO][0m CA: /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem
[0;34m[INFO][0m Configuring certificate permissions for clickhouse (user: clickhouse)
[0;34m[INFO][0m Initializing certificate directory for obs-user-sau-main-dev...
[2026-01-18 23:22:33 UTC] USER=www-data EUID=0 PID=3979319 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-18 23:22:33 UTC] USER=www-data EUID=0 PID=3979328 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-18 23:22:33 UTC] USER=www-data EUID=0 PID=3979337 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-18 23:22:33 UTC] USER=www-data EUID=0 PID=3979346 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-user-sau-main-dev
[0;32m[OK][0m Certificate directory initialized: /etc/fastorder/observability/certs/obs-user-sau-main-dev
[0;34m[INFO][0m Setting file permissions...
[2026-01-18 23:22:33 UTC] USER=www-data EUID=0 PID=3979356 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/clickhouse-cert.pem
[2026-01-18 23:22:33 UTC] USER=www-data EUID=0 PID=3979366 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem
[2026-01-18 23:22:33 UTC] USER=www-data EUID=0 PID=3979375 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-user-sau-main-dev/clickhouse-key.pem
[0;34m[INFO][0m Setting file ownership...
[2026-01-18 23:22:33 UTC] USER=www-data EUID=0 PID=3979384 ACTION=passthru ARGS=chown root:clickhouse /etc/fastorder/observability/certs/obs-user-sau-main-dev/clickhouse-key.pem
[2026-01-18 23:22:33 UTC] USER=www-data EUID=0 PID=3979393 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-user-sau-main-dev/clickhouse-cert.pem /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem
[0;34m[INFO][0m Permission configuration completed
[0;34m[INFO][0m (Verification skipped - running via wrapper, trust chmod/chown success)
[0;32m[OK][0m ✅ Certificate permissions configured successfully for clickhouse
[0;34m[INFO][0m Creating ClickHouse configuration...
[2026-01-18 23:22:33 UTC] USER=www-data EUID=0 PID=3979439 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /etc/clickhouse-server-obs-user-sau-main-dev
[2026-01-18 23:22:33 UTC] USER=www-data EUID=0 PID=3979448 ACTION=passthru ARGS=bash -c chmod 640 /etc/clickhouse-server-obs-user-sau-main-dev/*.xml
[0;32m[OK][0m ClickHouse configuration created
[0;34m[INFO][0m Creating logs table schema...
[2026-01-18 23:22:33 UTC] USER=www-data EUID=0 PID=3979466 ACTION=passthru ARGS=sed -i s/__RETENTION_DAYS__/90/g /etc/clickhouse-server-obs-user-sau-main-dev/logs_schema.sql
[2026-01-18 23:22:33 UTC] USER=www-data EUID=0 PID=3979475 ACTION=passthru ARGS=chmod 644 /etc/clickhouse-server-obs-user-sau-main-dev/logs_schema.sql
[0;32m[OK][0m Logs schema created
[0;34m[INFO][0m Creating systemd service...
[2026-01-18 23:22:33 UTC] USER=www-data EUID=0 PID=3979493 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /var/lib/clickhouse-obs-user-sau-main-dev
[2026-01-18 23:22:34 UTC] USER=www-data EUID=0 PID=3979502 ACTION=passthru ARGS=chown -R clickhouse:clickhouse /var/log/clickhouse-server-obs-user-sau-main-dev
[2026-01-18 23:22:34 UTC] USER=www-data EUID=0 PID=3979511 ACTION=passthru ARGS=chmod 700 /var/lib/clickhouse-obs-user-sau-main-dev
[0;32m[OK][0m Systemd service created
[0;34m[INFO][0m Starting ClickHouse service...
[2026-01-18 23:22:34 UTC] USER=www-data EUID=0 PID=3979520 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:22:34 UTC] USER=www-data EUID=0 PID=3979565 ACTION=passthru ARGS=systemctl enable clickhouse-server-obs-user-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/clickhouse-server-obs-user-sau-main-dev.service → /etc/systemd/system/clickhouse-server-obs-user-sau-main-dev.service.
[2026-01-18 23:22:34 UTC] USER=www-data EUID=0 PID=3979648 ACTION=passthru ARGS=systemctl start clickhouse-server-obs-user-sau-main-dev.service
[0;34m[INFO][0m Waiting for ClickHouse to be ready...
[0;32m[OK][0m ClickHouse is ready
[0;34m[INFO][0m Initializing database schema...
[0;32m[OK][0m Schema initialized
[0;34m[INFO][0m Storing ClickHouse credentials in AWS Secrets Manager...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/user/sau/main/dev/clickhouse/server/logs_writer-rLASjU",
"Name": "fastorder/observability/user/sau/main/dev/clickhouse/server/logs_writer",
"VersionId": "7fdeb284-280c-466b-ab75-6c14e6f0a9bd"
}
[0;32m[OK][0m logs_writer credentials stored and verified in Secrets Manager
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/user/sau/main/dev/clickhouse/server/metrics_reader-tp67G5",
"Name": "fastorder/observability/user/sau/main/dev/clickhouse/server/metrics_reader",
"VersionId": "7b1968be-2b59-48c8-b112-eb85c3fac6d4"
}
[0;32m[OK][0m metrics_reader credentials stored and verified in Secrets Manager
[0;34m[INFO][0m Validating ClickHouse deployment...
[0;34m[INFO][0m ClickHouse version: 25.10.1.3832
[0;34m[INFO][0m Tables created: .inner_id.2f86f613-ad12-405d-b929-bb2be0ccf8a6
.inner_id.9dd338c2-eb72-420c-a808-56963dc7829c
application_logs
error_logs_mv
iam_audit_event
metrics_all
otel_logs
request_logs_mv
security_access
[0;34m[INFO][0m Test log inserted. Total logs: 1
[0;32m[OK][0m ✅ ClickHouse deployment validated
[0;34m[INFO][0m Setting up clickhouse-backup for backup management...
[0;32m[OK][0m clickhouse-backup already installed
[0;34m[INFO][0m Creating clickhouse-backup configuration...
[2026-01-18 23:22:46 UTC] USER=www-data EUID=0 PID=3980639 ACTION=fsop ARGS=mkdir -p /etc/clickhouse-backup
[2026-01-18 23:22:46 UTC] USER=www-data EUID=0 PID=3980669 ACTION=passthru ARGS=chmod 750 /etc/clickhouse-backup
[2026-01-18 23:22:46 UTC] USER=www-data EUID=0 PID=3980687 ACTION=passthru ARGS=chown root:clickhouse /etc/clickhouse-backup/config-obs-user-sau-main-dev.yml
[2026-01-18 23:22:46 UTC] USER=www-data EUID=0 PID=3980696 ACTION=passthru ARGS=chmod 640 /etc/clickhouse-backup/config-obs-user-sau-main-dev.yml
[2026-01-18 23:22:46 UTC] USER=www-data EUID=0 PID=3980732 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:22:46 UTC] USER=www-data EUID=0 PID=3980779 ACTION=passthru ARGS=systemctl enable clickhouse-backup-api-obs-user-sau-main-dev.service
[2026-01-18 23:22:47 UTC] USER=www-data EUID=0 PID=3980835 ACTION=passthru ARGS=systemctl start clickhouse-backup-api-obs-user-sau-main-dev.service
[2026-01-18 23:22:47 UTC] USER=www-data EUID=0 PID=3980845 ACTION=passthru ARGS=systemctl enable clickhouse-backup@obs-user-sau-main-dev.timer
[2026-01-18 23:22:47 UTC] USER=www-data EUID=0 PID=3980890 ACTION=passthru ARGS=systemctl start clickhouse-backup@obs-user-sau-main-dev.timer
[0;32m[OK][0m clickhouse-backup configured and started
[0;34m[INFO][0m Setting up ClickHouse exporter for Prometheus...
[0;32m[OK][0m clickhouse_exporter already installed
[2026-01-18 23:22:47 UTC] USER=www-data EUID=0 PID=3980911 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:22:48 UTC] USER=www-data EUID=0 PID=3980956 ACTION=passthru ARGS=systemctl enable clickhouse_exporter-obs-user-sau-main-dev.service
[2026-01-18 23:22:48 UTC] USER=www-data EUID=0 PID=3981002 ACTION=passthru ARGS=systemctl start clickhouse_exporter-obs-user-sau-main-dev.service
[0;32m[OK][0m clickhouse_exporter configured and started
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ ClickHouse Deployed Successfully
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m FQDN: logstore-user-sau-main-dev-clickhouse.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.217
[0;34m[INFO][0m HTTP Port: 8123
[0;34m[INFO][0m Native Port: 9000
[0;34m[INFO][0m Database: logs
[0;34m[INFO][0m Retention: 90 days
[0;34m[INFO][0m Storage: Tiered (Local → S3: fastorder-logs-sau-dev in me-central-1)
[0;34m[INFO][0m
[0;34m[INFO][0m Backup & Monitoring:
[0;34m[INFO][0m clickhouse-backup API: http://10.100.1.217:7171
[0;34m[INFO][0m clickhouse_exporter: http://10.100.1.217:9116/metrics
[0;34m[INFO][0m Backup Schedule: Daily at 2:00 AM
[0;34m[INFO][0m Local Backups Retained: 7
[0;34m[INFO][0m
[0;34m[INFO][0m Credentials stored in AWS Secrets Manager:
[0;34m[INFO][0m Writers: fastorder/observability/user/sau/main/dev/clickhouse/server/logs_writer
[0;34m[INFO][0m Readers: fastorder/observability/user/sau/main/dev/clickhouse/server/metrics_reader (for PHP metrics service)
[0;34m[INFO][0m
[0;34m[INFO][0m Example queries (using credentials from Secrets Manager):
[0;34m[INFO][0m # Write logs:
[0;34m[INFO][0m clickhouse-client --host logstore-user-sau-main-dev-clickhouse.fastorder.com --port 9000 --user logs_writer --password '***' --query 'SELECT 1'
[0;34m[INFO][0m
[0;34m[INFO][0m # Read metrics (PHP metrics service):
[0;34m[INFO][0m clickhouse-client --host logstore-user-sau-main-dev-clickhouse.fastorder.com --port 9000 --user metrics_reader --password '***' --query 'SELECT * FROM system.metrics'
[0;34m[INFO][0m
[0;34m[INFO][0m HTTPS Setup (run on web-03/skeleton server):
[0;34m[INFO][0m # Set up HTTPS reverse proxy with Let's Encrypt:
[0;34m[INFO][0m OBS_CELL=obs-user-sau-main-dev BACKEND_IP=10.100.1.217 sudo bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/provider/../https/setup-clickhouse-https.sh
[0;34m[INFO][0m
[0;34m[INFO][0m # Or add --setup-https flag when running this script
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Log Storage Backend Deployed Successfully
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: clickhouse
[0;34m[INFO][0m FQDN: logstore-user-sau-main-dev-clickhouse.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.217
[0;34m[INFO][0m Retention: 90 days
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Registering ClickHouse in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: ClickHouse
[INFO] Identifier: user-sau-main-dev-clickhouse
[INFO] Identifier Parent: cluster
[INFO] IP: 10.100.1.217
[INFO] Port: 8443
[INFO] FQDN: logstore-user-sau-main-dev-clickhouse.fastorder.com
[INFO] Status: running
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ INVALID REQUEST
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] Response: {"success":false,"error":"Invalid JSON: Control character error, possibly incorrectly encoded"}
[ERROR]
[ERROR] Request payload:
{
"env_id": "user-sau-main-dev",
"application": "ClickHouse",
"identifier": "user-sau-main-dev-clickhouse",
"identifier_parent": "cluster",
"ip": "10.100.1.217",
"port": 8443,
"fqdn": "logstore-user-sau-main-dev-clickhouse.fastorder.com",
"status": "running",
"meta": {
"role": "log_storage",
"provider": "clickhouse",
"version": "25.10
1.3832",
"http_port": 8123,
"native_port": 9000,
"https_port": 8443,
"protocol": "https",
"metrics_enabled": true,
"metrics_port": 8123,
"metrics_path": "/metrics",
"health_endpoint": "https://logstore-user-sau-main-dev-clickhouse.fastorder.com/ping",
"retention_days": 90,
"s3_bucket": "fastorder-logs-sau-dev"
}
}
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[1;33m[WARN][0m ⚠️ Failed to register ClickHouse (service is running)
[0;32m[OK][0m clickhouse deployed successfully
[0;32m[OK][0m Log storage backend deployed
[0;34m[INFO][0m Step 6/10: Deploying telemetry collector...
[0;34m[INFO][0m Provider: otlp (backend implementation - internal)
[0;34m[INFO][0m Endpoint: telemetry-user-sau-main-dev-opentelemetry.fastorder.com (stable, exposed to clients)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m TELEMETRY COLLECTOR DEPLOYMENT
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: otlp
[0;34m[INFO][0m Observability Cell: obs-user-sau-main-dev
[0;34m[INFO][0m FQDN: telemetry-user-sau-main-dev-opentelemetry.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.229
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Using provider: otlp
[0;34m[INFO][0m Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Telemetry/provider/otlp.sh
[0;34m[INFO][0m Executing provider deployment script...
[0;34m[INFO][0m Parsed: SERVICE=user, ZONE=sau, BRANCH=main, ENV=dev
[0;34m[INFO][0m Checking and cleaning ports before installation...
[0;34m[INFO][0m Initializing certificate directory for obs-user-sau-main-dev...
[2026-01-18 23:22:49 UTC] USER=www-data EUID=0 PID=3981110 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-18 23:22:49 UTC] USER=www-data EUID=0 PID=3981119 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-18 23:22:49 UTC] USER=www-data EUID=0 PID=3981128 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-18 23:22:49 UTC] USER=www-data EUID=0 PID=3981137 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-user-sau-main-dev
[0;32m[OK][0m Certificate directory initialized: /etc/fastorder/observability/certs/obs-user-sau-main-dev
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking and cleaning ports for observability cell: obs-user-sau-main-dev
[0;34m[INFO][0m IP Address: 10.100.1.229
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking for conflicting observability services...
[0;34m[INFO][0m Service clickhouse-server-obs-user-sau-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Service clickhouse-server@obs-user-sau-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Found 7 observability service(s) (all belong to current cell)
[0;34m[INFO][0m Checking for remaining processes on IP 10.100.1.229...
[0;34m[INFO][0m Scanning 15 ports...
[0;32m[OK][0m ✅ All 15 ports are FREE - ready for installation
[0;32m[OK][0m Port cleanup successful on attempt 1
[0;34m[INFO][0m Binding to allocated IP: 10.100.1.229
[0;34m[INFO][0m Deploying OpenTelemetry Collector for observability cell: obs-user-sau-main-dev
[0;34m[INFO][0m FQDN: telemetry-user-sau-main-dev-opentelemetry.fastorder.com
[0;34m[INFO][0m Allocated IP: 10.100.1.229
[0;34m[INFO][0m VM IP: 10.100.1.229
[0;34m[INFO][0m Ports: gRPC=4317 HTTP=4318 Metrics=8888 Prom=8889
[0;34m[VERSION][0m Fetching latest version for otel_collector from GitHub (open-telemetry/opentelemetry-collector-releases)...
[0;32m[VERSION][0m Latest otel_collector version: 0.143.1
[0;34m[INFO][0m Resolved OpenTelemetry Collector version: 0.143.1
[0;32m[OK][0m User 'otelcol' already exists
[0;34m[INFO][0m Checking if OpenTelemetry Collector is installed...
[0;32m[OK][0m OpenTelemetry Collector already installed at /usr/local/bin/otelcol-contrib
[0;34m[INFO][0m Creating configuration/data directories...
[2026-01-18 23:22:50 UTC] USER=www-data EUID=0 PID=3981282 ACTION=passthru ARGS=mkdir -p /etc/otelcol/obs-user-sau-main-dev
[2026-01-18 23:22:50 UTC] USER=www-data EUID=0 PID=3981298 ACTION=passthru ARGS=mkdir -p /var/lib/otelcol/obs-user-sau-main-dev
[2026-01-18 23:22:50 UTC] USER=www-data EUID=0 PID=3981308 ACTION=passthru ARGS=chown -R otelcol:otelcol /etc/otelcol/obs-user-sau-main-dev /var/lib/otelcol/obs-user-sau-main-dev
[2026-01-18 23:22:50 UTC] USER=www-data EUID=0 PID=3981326 ACTION=passthru ARGS=chmod 0750 /var/lib/otelcol/obs-user-sau-main-dev
[0;34m[INFO][0m Retrieving ClickHouse credentials from Secrets Manager...
[0;32m[OK][0m Retrieved ClickHouse credentials from Secrets Manager
[0;34m[INFO][0m Creating OpenTelemetry Collector configuration...
[0;34m[INFO][0m ClickHouse exporter enabled: tcp://logstore-user-sau-main-dev-clickhouse.fastorder.com:9000
[2026-01-18 23:22:52 UTC] USER=www-data EUID=0 PID=3981376 ACTION=passthru ARGS=chown otelcol:otelcol /etc/otelcol/obs-user-sau-main-dev/config.yaml
[2026-01-18 23:22:52 UTC] USER=www-data EUID=0 PID=3981385 ACTION=passthru ARGS=chmod 0640 /etc/otelcol/obs-user-sau-main-dev/config.yaml
[0;32m[OK][0m Configuration created at /etc/otelcol/obs-user-sau-main-dev/config.yaml
[0;34m[INFO][0m Setting up TLS certificate permissions...
[0;34m[INFO][0m Configuring certificate permissions for otlp_collector (user: otelcol)
[0;34m[INFO][0m Initializing certificate directory for obs-user-sau-main-dev...
[2026-01-18 23:22:52 UTC] USER=www-data EUID=0 PID=3981394 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-18 23:22:52 UTC] USER=www-data EUID=0 PID=3981422 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-18 23:22:52 UTC] USER=www-data EUID=0 PID=3981468 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-user-sau-main-dev
[0;32m[OK][0m Certificate directory initialized: /etc/fastorder/observability/certs/obs-user-sau-main-dev
[0;34m[INFO][0m Setting file permissions...
[2026-01-18 23:22:52 UTC] USER=www-data EUID=0 PID=3981479 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/otlp_collector-cert.pem
[2026-01-18 23:22:52 UTC] USER=www-data EUID=0 PID=3981494 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem
[2026-01-18 23:22:52 UTC] USER=www-data EUID=0 PID=3981511 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-user-sau-main-dev/otlp_collector-key.pem
[0;34m[INFO][0m Setting file ownership...
[2026-01-18 23:22:52 UTC] USER=www-data EUID=0 PID=3981520 ACTION=passthru ARGS=chown root:otelcol /etc/fastorder/observability/certs/obs-user-sau-main-dev/otlp_collector-key.pem
[2026-01-18 23:22:52 UTC] USER=www-data EUID=0 PID=3981530 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-user-sau-main-dev/otlp_collector-cert.pem /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem
[0;34m[INFO][0m Permission configuration completed
[0;34m[INFO][0m (Verification skipped - running via wrapper, trust chmod/chown success)
[0;32m[OK][0m ✅ Certificate permissions configured successfully for otlp_collector
[0;32m[OK][0m Certificate permissions configured
[0;34m[INFO][0m Creating systemd service: otelcol-obs-user-sau-main-dev
[0;32m[OK][0m Systemd service created at /etc/systemd/system/otelcol-obs-user-sau-main-dev.service
[0;34m[INFO][0m Adding /etc/hosts entry for telemetry-user-sau-main-dev-opentelemetry.fastorder.com -> 10.100.1.229
[2026-01-18 23:22:52 UTC] USER=www-data EUID=0 PID=3981551 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*telemetry-user-sau-main-dev-opentelemetry.fastorder.com/10.100.1.229 telemetry-user-sau-main-dev-opentelemetry.fastorder.com/ /etc/hosts
[0;32m[OK][0m Updated /etc/hosts entry to use VM_IP
[0;34m[INFO][0m Storing OTLP configuration metadata in AWS Secrets Manager (if aws CLI present)...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/user/sau/main/dev/otlp/collector-JuXPY7",
"Name": "fastorder/observability/user/sau/main/dev/otlp/collector",
"VersionId": "da74299a-9086-4486-8006-9655261e0ec9"
}
[0;32m[OK][0m Configuration metadata stored/updated in AWS Secrets Manager: fastorder/observability/user/sau/main/dev/otlp/collector
[0;34m[INFO][0m Enabling and starting OpenTelemetry Collector service...
[2026-01-18 23:22:54 UTC] USER=www-data EUID=0 PID=3981574 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:22:54 UTC] USER=www-data EUID=0 PID=3981628 ACTION=passthru ARGS=systemctl enable otelcol-obs-user-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/otelcol-obs-user-sau-main-dev.service → /etc/systemd/system/otelcol-obs-user-sau-main-dev.service.
[2026-01-18 23:22:55 UTC] USER=www-data EUID=0 PID=3981675 ACTION=passthru ARGS=systemctl restart otelcol-obs-user-sau-main-dev.service
[0;32m[OK][0m Service enabled and started
[0;34m[INFO][0m Validating deployment...
[2026-01-18 23:22:58 UTC] USER=www-data EUID=0 PID=3981769 ACTION=passthru ARGS=systemctl is-active --quiet otelcol-obs-user-sau-main-dev.service
[0;32m[OK][0m ✅ OpenTelemetry Collector is running
[0;32m[OK][0m ✅ gRPC endpoint listening on port 4317
[0;32m[OK][0m ✅ HTTP endpoint listening on port 4318
[0;32m[OK][0m ✅ Prometheus metrics endpoint listening on port 8889
[0;34m[INFO][0m Service logs (last 10 lines):
[2026-01-18 23:22:58 UTC] USER=www-data EUID=0 PID=3981784 ACTION=passthru ARGS=journalctl -u otelcol-obs-user-sau-main-dev.service -n 10 --no-pager
Jan 18 23:22:55 web-03 otelcol-obs-user-sau-main-dev[3981682]: 2026-01-18T23:22:55.660Z info internal/resourcedetection.go:125 began detecting resource information {"kind": "processor", "name": "resourcedetection", "pipeline": "logs"}
Jan 18 23:22:55 web-03 otelcol-obs-user-sau-main-dev[3981682]: 2026-01-18T23:22:55.662Z info system/system.go:201 This attribute changed from int to string. Temporarily switch back to int using the feature gate. {"kind": "processor", "name": "resourcedetection", "pipeline": "logs", "attribute": "host.cpu.family", "feature gate": "processor.resourcedetection.hostCPUModelAndFamilyAsString"}
Jan 18 23:22:55 web-03 otelcol-obs-user-sau-main-dev[3981682]: 2026-01-18T23:22:55.663Z info system/system.go:220 This attribute changed from int to string. Temporarily switch back to int using the feature gate. {"kind": "processor", "name": "resourcedetection", "pipeline": "logs", "attribute": "host.cpu.model.id", "feature gate": "processor.resourcedetection.hostCPUModelAndFamilyAsString"}
Jan 18 23:22:55 web-03 otelcol-obs-user-sau-main-dev[3981682]: 2026-01-18T23:22:55.663Z info internal/resourcedetection.go:139 detected resource information {"kind": "processor", "name": "resourcedetection", "pipeline": "logs", "resource": {"host.name":"web-03","os.type":"linux"}}
Jan 18 23:22:55 web-03 otelcol-obs-user-sau-main-dev[3981682]: 2026-01-18T23:22:55.731Z info otlpreceiver@v0.91.0/otlp.go:83 Starting GRPC server {"kind": "receiver", "name": "otlp", "data_type": "logs", "endpoint": "10.100.1.229:4317"}
Jan 18 23:22:55 web-03 otelcol-obs-user-sau-main-dev[3981682]: 2026-01-18T23:22:55.731Z info otlpreceiver@v0.91.0/otlp.go:101 Starting HTTP server {"kind": "receiver", "name": "otlp", "data_type": "logs", "endpoint": "10.100.1.229:4318"}
Jan 18 23:22:55 web-03 otelcol-obs-user-sau-main-dev[3981682]: 2026-01-18T23:22:55.734Z info prometheusreceiver@v0.91.0/metrics_receiver.go:231 Scrape job added {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "jobName": "otel-collector"}
Jan 18 23:22:55 web-03 otelcol-obs-user-sau-main-dev[3981682]: 2026-01-18T23:22:55.734Z info prometheusreceiver@v0.91.0/metrics_receiver.go:240 Starting discovery manager {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
Jan 18 23:22:55 web-03 otelcol-obs-user-sau-main-dev[3981682]: 2026-01-18T23:22:55.735Z info service@v0.91.0/service.go:171 Everything is ready. Begin running and processing data.
Jan 18 23:22:55 web-03 otelcol-obs-user-sau-main-dev[3981682]: 2026-01-18T23:22:55.735Z info prometheusreceiver@v0.91.0/metrics_receiver.go:282 Starting scrape manager {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Telemetry Collector Deployed Successfully
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: otlp
[0;34m[INFO][0m FQDN: telemetry-user-sau-main-dev-opentelemetry.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.229
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Registering OpenTelemetry Collector in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: OpenTelemetry Collector
[INFO] Identifier: user-sau-main-dev-opentelemetry
[INFO] Identifier Parent: cluster
[INFO] IP: 10.100.1.229
[INFO] Port: 4317
[INFO] FQDN: telemetry-user-sau-main-dev-opentelemetry.fastorder.com
[INFO] Status: running
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 64b466a1-2bd2-4a51-89f4-77fd07d335f0
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[0;32m[OK][0m ✅ OpenTelemetry Collector registered in dashboard
[0;34m[INFO][0m Setting up OpenTelemetry Collector metrics collection timer...
[2026-01-18 23:22:58 UTC] USER=www-data EUID=0 PID=3981854 ACTION=passthru ARGS=mv /tmp/otelcol-metrics-user-sau-main-dev.service /etc/systemd/system/
[2026-01-18 23:22:58 UTC] USER=www-data EUID=0 PID=3981864 ACTION=passthru ARGS=mv /tmp/otelcol-metrics-user-sau-main-dev.timer /etc/systemd/system/
[2026-01-18 23:22:58 UTC] USER=www-data EUID=0 PID=3981873 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:22:59 UTC] USER=www-data EUID=0 PID=3981919 ACTION=passthru ARGS=systemctl enable otelcol-metrics-user-sau-main-dev.timer
[2026-01-18 23:22:59 UTC] USER=www-data EUID=0 PID=3981971 ACTION=passthru ARGS=systemctl start otelcol-metrics-user-sau-main-dev.timer
[0;32m[OK][0m ✅ Metrics collection timer installed and started
[0;32m[OK][0m Telemetry collector (otlp) deployed successfully
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Step 7/10: METRICS BACKEND DEPLOYMENT
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Provider: prometheus
[0;34m[INFO][0m OBS Cell: obs-user-sau-main-dev
[0;34m[INFO][0m FQDN: metrics-user-sau-main-dev-prometheus.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.187
[0;34m[INFO][0m Script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/deploy-metrics.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 📊 METRICS DEPLOYMENT WRAPPER STARTED
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Script: deploy-metrics.sh
[0;34m[INFO][0m Timestamp: 2026-01-18 23:22:59 UTC
[0;34m[INFO][0m Arguments: --provider prometheus --obs-cell obs-user-sau-main-dev --fqdn metrics-user-sau-main-dev-prometheus.fastorder.com --ip 10.100.1.187
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m METRICS DEPLOYMENT
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: prometheus
[0;34m[INFO][0m Observability Cell: obs-user-sau-main-dev
[0;34m[INFO][0m FQDN: metrics-user-sau-main-dev-prometheus.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.187
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Using provider: prometheus
[0;34m[INFO][0m Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/provider/prometheus.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/provider/prometheus.sh
[0;34m[INFO][0m OBS_CELL: obs-user-sau-main-dev
[0;34m[INFO][0m FQDN: metrics-user-sau-main-dev-prometheus.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.187
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Parsed: SERVICE=user, ZONE=sau, BRANCH=main, ENV=dev
[0;34m[INFO][0m Checking and cleaning ports before installation...
[0;34m[INFO][0m Initializing certificate directory for obs-user-sau-main-dev...
[2026-01-18 23:22:59 UTC] USER=www-data EUID=0 PID=3982001 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-18 23:22:59 UTC] USER=www-data EUID=0 PID=3982013 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-18 23:22:59 UTC] USER=www-data EUID=0 PID=3982024 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-18 23:22:59 UTC] USER=www-data EUID=0 PID=3982033 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-user-sau-main-dev
[0;32m[OK][0m Certificate directory initialized: /etc/fastorder/observability/certs/obs-user-sau-main-dev
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking and cleaning ports for observability cell: obs-user-sau-main-dev
[0;34m[INFO][0m IP Address: 10.100.1.187
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking for conflicting observability services...
[0;34m[INFO][0m Service clickhouse-server-obs-user-sau-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Service clickhouse-server@obs-user-sau-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-obs-user-sau-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Found 8 observability service(s) (all belong to current cell)
[0;34m[INFO][0m Checking for remaining processes on IP 10.100.1.187...
[0;34m[INFO][0m Scanning 15 ports...
[0;32m[OK][0m ✅ All 15 ports are FREE - ready for installation
[0;32m[OK][0m Port cleanup successful on attempt 1
[0;34m[INFO][0m Deploying Prometheus for observability cell: obs-user-sau-main-dev
[0;34m[INFO][0m FQDN: metrics-user-sau-main-dev-prometheus.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.187
[0;34m[INFO][0m Prometheus Port: 9090
[0;34m[VERSION][0m Fetching latest version for prometheus from GitHub (prometheus/prometheus)...
[0;32m[VERSION][0m Latest prometheus version: 3.9.1
[0;34m[INFO][0m Resolved Prometheus version: 3.9.1
[0;34m[INFO][0m Checking if Prometheus is installed...
[0;32m[OK][0m Prometheus already installed at /usr/local/bin/prometheus
[0;34m[VERSION][0m Fetching latest version for node_exporter from GitHub (prometheus/node_exporter)...
[0;32m[VERSION][0m Latest node_exporter version: 1.10.2
[0;34m[INFO][0m Resolved Node Exporter version: 1.10.2
[0;34m[INFO][0m Checking if Node Exporter is installed...
[0;32m[OK][0m Node Exporter already installed at /usr/local/bin/node_exporter
[2026-01-18 23:23:00 UTC] USER=www-data EUID=0 PID=3982197 ACTION=fsop ARGS=mkdir -p /etc/prometheus/obs-user-sau-main-dev
[0;34m[INFO][0m Creating Node Exporter TLS web config...
[0;34m[INFO][0m Creating Node Exporter systemd service with TLS...
[2026-01-18 23:23:00 UTC] USER=www-data EUID=0 PID=3982224 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:23:01 UTC] USER=www-data EUID=0 PID=3982280 ACTION=passthru ARGS=systemctl enable node_exporter-obs-user-sau-main-dev.service
[2026-01-18 23:23:01 UTC] USER=www-data EUID=0 PID=3982326 ACTION=passthru ARGS=systemctl restart node_exporter-obs-user-sau-main-dev.service
[0;32m[OK][0m Node Exporter service configured and started
[0;34m[INFO][0m Creating configuration directory: /etc/prometheus/obs-user-sau-main-dev
[2026-01-18 23:23:01 UTC] USER=www-data EUID=0 PID=3982354 ACTION=fsop ARGS=mkdir -p /etc/prometheus/obs-user-sau-main-dev
[2026-01-18 23:23:01 UTC] USER=www-data EUID=0 PID=3982371 ACTION=fsop ARGS=mkdir -p /var/lib/prometheus/obs-user-sau-main-dev
[2026-01-18 23:23:01 UTC] USER=www-data EUID=0 PID=3982380 ACTION=fsop ARGS=mkdir -p /etc/prometheus/obs-user-sau-main-dev/rules
[0;34m[INFO][0m Creating Prometheus configuration...
[0;34m[INFO][0m Generated FQDNs:
[0;34m[INFO][0m Prometheus: metrics-user-sau-main-dev-prometheus.fastorder.com
[0;34m[INFO][0m Alertmanager: alerts-user-sau-main-dev-alertmanager.fastorder.com
[0;34m[INFO][0m Grafana: dashboards-user-sau-main-dev-grafana.fastorder.com
[0;34m[INFO][0m Otelcol: telemetry-user-sau-main-dev-opentelemetry.fastorder.com
[0;32m[OK][0m Configuration created at /etc/prometheus/obs-user-sau-main-dev/prometheus.yml
[0;34m[INFO][0m Creating Prometheus web config for HTTPS...
[0;32m[OK][0m Web config created at /etc/prometheus/obs-user-sau-main-dev/web-config.yml
[0;34m[INFO][0m Creating basic alerting rules...
[0;32m[OK][0m Alerting rules created
[2026-01-18 23:23:01 UTC] USER=www-data EUID=0 PID=3982416 ACTION=fsop ARGS=mkdir -p /etc/prometheus/obs-user-sau-main-dev/targets
[2026-01-18 23:23:01 UTC] USER=www-data EUID=0 PID=3982425 ACTION=passthru ARGS=bash -c cat > '/etc/prometheus/obs-user-sau-main-dev/targets/.placeholder.yml' << 'EOF'
# Placeholder file to prevent file_sd_configs warning
# Application targets will be added here automatically
[]
EOF
[0;34m[INFO][0m Creating systemd service: prometheus-obs-user-sau-main-dev
[0;34m[INFO][0m Binding to: 10.100.1.187:9090
[0;32m[OK][0m Systemd service created
[0;34m[INFO][0m Configuring certificate permissions...
[0;34m[INFO][0m Configuring certificate permissions for prometheus (user: root)
[0;34m[INFO][0m Initializing certificate directory for obs-user-sau-main-dev...
[2026-01-18 23:23:01 UTC] USER=www-data EUID=0 PID=3982444 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-18 23:23:01 UTC] USER=www-data EUID=0 PID=3982453 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-18 23:23:01 UTC] USER=www-data EUID=0 PID=3982462 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-18 23:23:01 UTC] USER=www-data EUID=0 PID=3982471 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-user-sau-main-dev
[0;32m[OK][0m Certificate directory initialized: /etc/fastorder/observability/certs/obs-user-sau-main-dev
[0;34m[INFO][0m Setting file permissions...
[2026-01-18 23:23:01 UTC] USER=www-data EUID=0 PID=3982481 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/prometheus-cert.pem
[2026-01-18 23:23:01 UTC] USER=www-data EUID=0 PID=3982490 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem
[2026-01-18 23:23:02 UTC] USER=www-data EUID=0 PID=3982499 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-user-sau-main-dev/prometheus-key.pem
[0;34m[INFO][0m Setting file ownership...
[2026-01-18 23:23:02 UTC] USER=www-data EUID=0 PID=3982508 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-user-sau-main-dev/prometheus-key.pem
[2026-01-18 23:23:02 UTC] USER=www-data EUID=0 PID=3982517 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-user-sau-main-dev/prometheus-cert.pem /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem
[0;34m[INFO][0m Permission configuration completed
[0;34m[INFO][0m (Verification skipped - running via wrapper, trust chmod/chown success)
[0;32m[OK][0m ✅ Certificate permissions configured successfully for prometheus
[0;32m[OK][0m Certificate permissions configured
[0;34m[INFO][0m Adding /etc/hosts entry for metrics-user-sau-main-dev-prometheus.fastorder.com -> 10.100.1.187
[2026-01-18 23:23:02 UTC] USER=www-data EUID=0 PID=3982528 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*metrics-user-sau-main-dev-prometheus.fastorder.com/10.100.1.187 metrics-user-sau-main-dev-prometheus.fastorder.com/ /etc/hosts
[0;32m[OK][0m Updated /etc/hosts entry to use VM_IP
[0;34m[INFO][0m Validating Prometheus configuration...
Checking /etc/prometheus/obs-user-sau-main-dev/prometheus.yml
SUCCESS: 1 rule files found
SUCCESS: /etc/prometheus/obs-user-sau-main-dev/prometheus.yml is valid prometheus config file syntax
Checking /etc/prometheus/obs-user-sau-main-dev/rules/basic_alerts.yml
SUCCESS: 4 rules found
[0;32m[OK][0m ✅ Configuration is valid
[0;34m[INFO][0m Storing Prometheus configuration in AWS Secrets Manager...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/user/sau/main/dev/prometheus/server-7UEoOv",
"Name": "fastorder/observability/user/sau/main/dev/prometheus/server",
"VersionId": "5af2813d-e162-4ad2-b443-3adfc7d40cfd"
}
[0;32m[OK][0m Configuration stored in AWS Secrets Manager
[0;34m[INFO][0m Enabling and starting Prometheus service...
[2026-01-18 23:23:03 UTC] USER=www-data EUID=0 PID=3982565 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:23:04 UTC] USER=www-data EUID=0 PID=3982610 ACTION=passthru ARGS=systemctl enable prometheus-obs-user-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus-obs-user-sau-main-dev.service → /etc/systemd/system/prometheus-obs-user-sau-main-dev.service.
[2026-01-18 23:23:04 UTC] USER=www-data EUID=0 PID=3982657 ACTION=passthru ARGS=systemctl restart prometheus-obs-user-sau-main-dev.service
[0;32m[OK][0m Service enabled and started
[0;34m[INFO][0m Validating deployment...
[2026-01-18 23:23:07 UTC] USER=www-data EUID=0 PID=3982734 ACTION=passthru ARGS=systemctl is-active --quiet prometheus-obs-user-sau-main-dev.service
[0;32m[OK][0m ✅ Prometheus is running
[0;32m[OK][0m ✅ Prometheus web interface listening on port 9090
[0;32m[OK][0m ✅ Prometheus health check passed (HTTPS)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m Prometheus Web UI: https://metrics-user-sau-main-dev-prometheus.fastorder.com:9090
[0;32m[OK][0m Targets: https://metrics-user-sau-main-dev-prometheus.fastorder.com:9090/targets
[0;32m[OK][0m Alerts: https://metrics-user-sau-main-dev-prometheus.fastorder.com:9090/alerts
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Service logs (last 10 lines):
[2026-01-18 23:23:09 UTC] USER=www-data EUID=0 PID=3982847 ACTION=passthru ARGS=journalctl -u prometheus-obs-user-sau-main-dev.service -n 10 --no-pager
Jan 18 23:23:04 web-03 prometheus-obs-user-sau-main-dev[3982664]: ts=2026-01-18T23:23:04.997Z caller=head.go:722 level=info component=tsdb msg="Replaying WAL, this may take a while"
Jan 18 23:23:04 web-03 prometheus-obs-user-sau-main-dev[3982664]: ts=2026-01-18T23:23:04.998Z caller=head.go:794 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
Jan 18 23:23:04 web-03 prometheus-obs-user-sau-main-dev[3982664]: ts=2026-01-18T23:23:04.998Z caller=head.go:831 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=51.278µs wal_replay_duration=580.737µs wbl_replay_duration=280ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=4.248µs total_replay_duration=1.115625ms
Jan 18 23:23:05 web-03 prometheus-obs-user-sau-main-dev[3982664]: ts=2026-01-18T23:23:05.002Z caller=main.go:1218 level=info fs_type=EXT4_SUPER_MAGIC
Jan 18 23:23:05 web-03 prometheus-obs-user-sau-main-dev[3982664]: ts=2026-01-18T23:23:05.002Z caller=main.go:1221 level=info msg="TSDB started"
Jan 18 23:23:05 web-03 prometheus-obs-user-sau-main-dev[3982664]: ts=2026-01-18T23:23:05.002Z caller=main.go:1404 level=info msg="Loading configuration file" filename=/etc/prometheus/obs-user-sau-main-dev/prometheus.yml
Jan 18 23:23:05 web-03 prometheus-obs-user-sau-main-dev[3982664]: ts=2026-01-18T23:23:05.006Z caller=main.go:1441 level=info msg="updated GOGC" old=100 new=75
Jan 18 23:23:05 web-03 prometheus-obs-user-sau-main-dev[3982664]: ts=2026-01-18T23:23:05.006Z caller=main.go:1452 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/obs-user-sau-main-dev/prometheus.yml totalDuration=4.372128ms db_storage=2.907µs remote_storage=2.895µs web_handler=1.022µs query_engine=2.054µs scrape=350.027µs scrape_sd=380.625µs notify=31.54µs notify_sd=25.399µs rules=1.688687ms tracing=9.738µs
Jan 18 23:23:05 web-03 prometheus-obs-user-sau-main-dev[3982664]: ts=2026-01-18T23:23:05.006Z caller=main.go:1182 level=info msg="Server is ready to receive web requests."
Jan 18 23:23:05 web-03 prometheus-obs-user-sau-main-dev[3982664]: ts=2026-01-18T23:23:05.007Z caller=manager.go:164 level=info component="rule manager" msg="Starting rule manager..."
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Provider script completed with exit code: 0
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Metrics Deployed Successfully
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: prometheus
[0;34m[INFO][0m FQDN: metrics-user-sau-main-dev-prometheus.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.187
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Registering Prometheus in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Prometheus
[INFO] Identifier: user-sau-main-dev-prometheus
[INFO] Identifier Parent: cluster
[INFO] IP: 10.100.1.187
[INFO] Port: 9090
[INFO] FQDN: metrics-user-sau-main-dev-prometheus.fastorder.com
[INFO] Status: running
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 9482a85e-3a1b-41fb-bdfb-942a31acfa01
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[0;32m[OK][0m Prometheus registered in dashboard
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Step 7/10: METRICS DEPLOYMENT RESULT
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Exit code: 0
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Metrics backend (prometheus) deployed successfully
[0;34m[INFO][0m Step 8/10: Deploying traces backend...
[0;34m[INFO][0m Provider: tempo (selected)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m TRACES DEPLOYMENT
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: tempo
[0;34m[INFO][0m Observability Cell: obs-user-sau-main-dev
[0;34m[INFO][0m FQDN: traces-user-sau-main-dev-tempo.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.227
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Using provider: tempo
[0;34m[INFO][0m Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Traces/provider/tempo.sh
[0;34m[INFO][0m Executing provider deployment script...
[0;34m[INFO][0m Parsed: SERVICE=user, ZONE=sau, BRANCH=main, ENV=dev
[0;34m[INFO][0m Checking and cleaning ports before installation...
[0;34m[INFO][0m Initializing certificate directory for obs-user-sau-main-dev...
[2026-01-18 23:23:10 UTC] USER=www-data EUID=0 PID=3982912 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-18 23:23:10 UTC] USER=www-data EUID=0 PID=3982921 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-18 23:23:10 UTC] USER=www-data EUID=0 PID=3982930 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-18 23:23:10 UTC] USER=www-data EUID=0 PID=3982939 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-user-sau-main-dev
[0;32m[OK][0m Certificate directory initialized: /etc/fastorder/observability/certs/obs-user-sau-main-dev
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking and cleaning ports for observability cell: obs-user-sau-main-dev
[0;34m[INFO][0m IP Address: 10.100.1.227
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Checking for conflicting observability services...
[0;34m[INFO][0m Service clickhouse-server-obs-user-sau-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Service clickhouse-server@obs-user-sau-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Service otelcol-metrics-iam-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-identity-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-sau-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-metrics-user-universe-main-dev.service not currently listening (may be stopped or starting) - skipping
[0;34m[INFO][0m Service otelcol-obs-user-sau-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Service prometheus-obs-user-sau-main-dev.service belongs to current cell (skipping)
[0;34m[INFO][0m Found 9 observability service(s) (all belong to current cell)
[0;34m[INFO][0m Checking for remaining processes on IP 10.100.1.227...
[0;34m[INFO][0m Scanning 15 ports...
[0;32m[OK][0m ✅ All 15 ports are FREE - ready for installation
[0;32m[OK][0m Port cleanup successful on attempt 1
[0;34m[INFO][0m Binding Tempo to allocated IP: 10.100.1.227
[0;34m[INFO][0m Deploying Grafana Tempo for observability cell: obs-user-sau-main-dev
[0;34m[INFO][0m FQDN: traces-user-sau-main-dev-tempo.fastorder.com
[0;34m[INFO][0m Allocated IP: 10.100.1.227
[0;34m[INFO][0m VM IP: 10.100.1.227
[0;34m[INFO][0m Ports: HTTP=3200 gRPC=9322, OTLP gRPC=4317, OTLP HTTP=4318
[0;34m[VERSION][0m Fetching latest version for tempo from GitHub (grafana/tempo)...
[0;32m[VERSION][0m Latest tempo version: 2.9.0
[0;34m[INFO][0m Resolved Tempo version: 2.9.0
[0;34m[INFO][0m Checking if Grafana Tempo is installed...
[0;32m[OK][0m Grafana Tempo already installed at /usr/local/bin/tempo
[0;34m[INFO][0m Preparing configuration and data directories...
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983114 ACTION=passthru ARGS=mkdir -p /etc/tempo/obs-user-sau-main-dev
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983123 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-user-sau-main-dev
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983132 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-user-sau-main-dev/wal
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983141 ACTION=passthru ARGS=mkdir -p /var/lib/tempo/obs-user-sau-main-dev/blocks
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983150 ACTION=passthru ARGS=chown -R tempo:tempo /etc/tempo/obs-user-sau-main-dev /var/lib/tempo/obs-user-sau-main-dev
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983159 ACTION=passthru ARGS=chmod 750 /etc/tempo/obs-user-sau-main-dev /var/lib/tempo/obs-user-sau-main-dev
[0;34m[INFO][0m Creating Grafana Tempo configuration...
[0;34m[INFO][0m TLS configuration exported for tempo
[0;34m[INFO][0m Cert: /etc/fastorder/observability/certs/obs-user-sau-main-dev/tempo-cert.pem
[0;34m[INFO][0m Key: /etc/fastorder/observability/certs/obs-user-sau-main-dev/tempo-key.pem
[0;34m[INFO][0m CA: /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem
[0;34m[INFO][0m Setting up certificate permissions for Tempo...
[0;34m[INFO][0m Configuring certificate permissions for tempo (user: tempo)
[0;34m[INFO][0m Initializing certificate directory for obs-user-sau-main-dev...
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983174 ACTION=passthru ARGS=chmod 755 /etc/fastorder
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983183 ACTION=passthru ARGS=chmod 755 /etc/fastorder/observability
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983192 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983201 ACTION=fsop ARGS=chmod 751 /etc/fastorder/observability/certs/obs-user-sau-main-dev
[0;32m[OK][0m Certificate directory initialized: /etc/fastorder/observability/certs/obs-user-sau-main-dev
[0;34m[INFO][0m Setting file permissions...
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983211 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/tempo-cert.pem
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983223 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983237 ACTION=passthru ARGS=chmod 640 /etc/fastorder/observability/certs/obs-user-sau-main-dev/tempo-key.pem
[0;34m[INFO][0m Setting file ownership...
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983255 ACTION=passthru ARGS=chown root:tempo /etc/fastorder/observability/certs/obs-user-sau-main-dev/tempo-key.pem
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983268 ACTION=passthru ARGS=chown root:root /etc/fastorder/observability/certs/obs-user-sau-main-dev/tempo-cert.pem /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem
[0;34m[INFO][0m Permission configuration completed
[0;34m[INFO][0m (Verification skipped - running via wrapper, trust chmod/chown success)
[0;32m[OK][0m ✅ Certificate permissions configured successfully for tempo
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983303 ACTION=passthru ARGS=chown tempo:tempo /etc/tempo/obs-user-sau-main-dev/config.yaml
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983314 ACTION=passthru ARGS=chmod 640 /etc/tempo/obs-user-sau-main-dev/config.yaml
[0;32m[OK][0m Configuration created at /etc/tempo/obs-user-sau-main-dev/config.yaml
[0;34m[INFO][0m Creating systemd service: tempo-obs-user-sau-main-dev
[0;32m[OK][0m Systemd service created
[0;34m[INFO][0m Adding /etc/hosts entry for traces-user-sau-main-dev-tempo.fastorder.com -> 10.100.1.227
[2026-01-18 23:23:11 UTC] USER=www-data EUID=0 PID=3983333 ACTION=passthru ARGS=sed -i s/^[0-9.]*[[:space:]]*traces-user-sau-main-dev-tempo.fastorder.com/10.100.1.227 traces-user-sau-main-dev-tempo.fastorder.com/ /etc/hosts
[0;32m[OK][0m Updated /etc/hosts entry to use VM_IP
[0;34m[INFO][0m Storing Tempo configuration in AWS Secrets Manager (if aws CLI present)...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/user/sau/main/dev/tempo/server-Cjxc6T",
"Name": "fastorder/observability/user/sau/main/dev/tempo/server",
"VersionId": "082d3896-e9e3-4935-ba32-2238edf6f550"
}
[0;32m[OK][0m Tempo configuration stored/updated in AWS Secrets Manager: fastorder/observability/user/sau/main/dev/tempo/server
[1;33m[WARN][0m Port cleanup library not found, skipping automatic cleanup
[0;34m[INFO][0m Adding iptables redirect for Tempo internal communication (required for search)...
[0;34m[INFO][0m ╔════════════════════════════════════════════════════════════════════════╗
[0;34m[INFO][0m ║ TEMPO IPTABLES DNAT CONFIGURATION (Audit Log) ║
[0;34m[INFO][0m ╠════════════════════════════════════════════════════════════════════════╣
[0;34m[INFO][0m ║ OBS_CELL: obs-user-sau-main-dev
[0;34m[INFO][0m ║ VM_IP: 10.100.1.227
[0;34m[INFO][0m ║ GRPC_PORT: 9322 (unique: 9095 + last_octet)
[0;34m[INFO][0m ║ TEMPO_UID: 989
[0;34m[INFO][0m ║ TIMESTAMP: 2026-01-18T23:23:13Z
[0;34m[INFO][0m ╚════════════════════════════════════════════════════════════════════════╝
[0;34m[INFO][0m Using --uid-owner 989 for DNAT rule (scoped to tempo user)
[0;31m[ERR][0m Could not add iptables redirect (iptables not allowed in wrapper)
[0;31m[ERR][0m ╔════════════════════════════════════════════════════════════════════════╗
[0;31m[ERR][0m ║ CRITICAL: Tempo search will NOT work without this redirect! ║
[0;31m[ERR][0m ║ ║
[0;31m[ERR][0m ║ Root cause: Tempo single-binary dials 127.0.0.1:<grpc_port> ║
[0;31m[ERR][0m ║ Each instance needs unique port + matching DNAT rule. ║
[0;31m[ERR][0m ║ ║
[0;31m[ERR][0m ║ Manually run: ║
[0;31m[ERR][0m ║ sudo iptables -t nat -A OUTPUT -p tcp -d 127.0.0.1 --dport 9322 -m owner --uid-owner 989 \ ║
[0;31m[ERR][0m ║ -j DNAT --to-destination 10.100.1.227:9322 ║
[0;31m[ERR][0m ╚════════════════════════════════════════════════════════════════════════╝
[0;34m[INFO][0m Enabling and starting Grafana Tempo service...
[2026-01-18 23:23:13 UTC] USER=www-data EUID=0 PID=3983407 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:23:13 UTC] USER=www-data EUID=0 PID=3983452 ACTION=passthru ARGS=systemctl enable tempo-obs-user-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/tempo-obs-user-sau-main-dev.service → /etc/systemd/system/tempo-obs-user-sau-main-dev.service.
[2026-01-18 23:23:14 UTC] USER=www-data EUID=0 PID=3983500 ACTION=passthru ARGS=systemctl restart tempo-obs-user-sau-main-dev.service
[0;32m[OK][0m Service enabled and started
[0;34m[INFO][0m Validating deployment...
[2026-01-18 23:23:17 UTC] USER=www-data EUID=0 PID=3983543 ACTION=passthru ARGS=systemctl is-active --quiet tempo-obs-user-sau-main-dev.service
[0;32m[OK][0m ✅ Grafana Tempo is running
[0;32m[OK][0m ✅ HTTP endpoint listening on port 3200
[0;32m[OK][0m ✅ OTLP gRPC endpoint listening on port 4317
[0;32m[OK][0m ✅ OTLP HTTP endpoint listening on port 4318
[0;34m[INFO][0m Running smoke test: Tempo search endpoint...
[1;33m[WARN][0m ⚠️ Tempo search smoke test failed - check iptables DNAT rule
[1;33m[WARN][0m Expected JSON with completedJobs/totalJobs, got: <html>
<head>
<title>Page Not Found</title>
<style>
body{
[0;34m[INFO][0m Service logs (last 10 lines):
[2026-01-18 23:23:19 UTC] USER=www-data EUID=0 PID=3983587 ACTION=passthru ARGS=journalctl -u tempo-obs-user-sau-main-dev.service -n 10 --no-pager
Jan 18 23:23:14 web-03 tempo-obs-user-sau-main-dev[3983520]: level=info ts=2026-01-18T23:23:14.653610064Z caller=lifecycler.go:687 msg="not loading tokens from file, tokens file path is empty"
Jan 18 23:23:14 web-03 tempo-obs-user-sau-main-dev[3983520]: level=info ts=2026-01-18T23:23:14.653671792Z caller=lifecycler.go:714 msg="instance not found in ring, adding with no tokens" ring=ingester
Jan 18 23:23:14 web-03 tempo-obs-user-sau-main-dev[3983520]: level=info ts=2026-01-18T23:23:14.653765952Z caller=lifecycler.go:556 msg="auto-joining cluster after timeout" ring=ingester
Jan 18 23:23:14 web-03 tempo-obs-user-sau-main-dev[3983520]: ts=2026-01-18T23:23:14Z level=info msg="Starting GRPC server" component=tempo endpoint=10.100.1.227:4317
Jan 18 23:23:14 web-03 tempo-obs-user-sau-main-dev[3983520]: ts=2026-01-18T23:23:14Z level=info msg="Starting HTTP server" component=tempo endpoint=10.100.1.227:4318
Jan 18 23:23:14 web-03 tempo-obs-user-sau-main-dev[3983520]: level=info ts=2026-01-18T23:23:14.655564919Z caller=worker.go:250 msg="total worker concurrency updated" totalConcurrency=20
Jan 18 23:23:14 web-03 tempo-obs-user-sau-main-dev[3983520]: ts=2026-01-18T23:23:14Z level=info msg="Starting UDP server for Binary Thrift" component=tempo endpoint=10.100.1.227:7059
Jan 18 23:23:14 web-03 tempo-obs-user-sau-main-dev[3983520]: ts=2026-01-18T23:23:14Z level=info msg="Starting UDP server for Compact Thrift" component=tempo endpoint=10.100.1.227:7058
Jan 18 23:23:14 web-03 tempo-obs-user-sau-main-dev[3983520]: ts=2026-01-18T23:23:14Z level=info msg="Starting HTTP server for Jaeger Thrift" component=tempo endpoint=10.100.1.227:14495
Jan 18 23:23:14 web-03 tempo-obs-user-sau-main-dev[3983520]: ts=2026-01-18T23:23:14Z level=info msg="Starting gRPC server for Jaeger Protobuf" component=tempo endpoint=10.100.1.227:14477
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Traces Deployed Successfully
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: tempo
[0;34m[INFO][0m FQDN: traces-user-sau-main-dev-tempo.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.227
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Registering Tempo in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Tempo
[INFO] Identifier: user-sau-main-dev-tempo
[INFO] Identifier Parent: cluster
[INFO] IP: 10.100.1.227
[INFO] Port: 3200
[INFO] FQDN: traces-user-sau-main-dev-tempo.fastorder.com
[INFO] Status: running
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 0982502e-e167-4e9f-b61a-8f9042a31099
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[0;32m[OK][0m ✅ Tempo registered in dashboard
[0;32m[OK][0m Traces backend (tempo) deployed successfully
[0;34m[INFO][0m Step 9/10: Deploying dashboards...
[0;34m[INFO][0m Provider: grafana (selected)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m DASHBOARDS DEPLOYMENT
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: grafana
[0;34m[INFO][0m Observability Cell: obs-user-sau-main-dev
[0;34m[INFO][0m FQDN: dashboards-user-sau-main-dev-grafana.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.188
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Using provider: grafana
[0;34m[INFO][0m Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Dashboards/provider/grafana.sh
[0;34m[INFO][0m Executing provider deployment script...
[0;34m[INFO][0m Parsed: SERVICE=user, ZONE=sau, BRANCH=main, ENV=dev
[0;34m[INFO][0m Binding to allocated IP: 10.100.1.188
[0;34m[INFO][0m Deploying Grafana for observability cell: obs-user-sau-main-dev
[0;34m[INFO][0m FQDN: dashboards-user-sau-main-dev-grafana.fastorder.com
[0;34m[INFO][0m Allocated IP: 10.100.1.188
[0;34m[INFO][0m VM IP: 10.100.1.188
[0;34m[INFO][0m HTTP Port: 3000
[0;34m[INFO][0m Checking if Grafana is installed...
[0;32m[OK][0m Grafana already installed
[0;34m[INFO][0m Installing Grafana plugins...
[0;34m[INFO][0m Installing ClickHouse datasource plugin...
[1;33m[WARN][0m Failed to install ClickHouse plugin (may need internet access)
[0;34m[INFO][0m Validating TLS certificate and key...
[0;34m[INFO][0m Setting certificate permissions...
[0;32m[OK][0m TLS cert/key found and permissions set
[0;34m[INFO][0m Creating configuration and data directories...
[2026-01-18 23:23:20 UTC] USER=www-data EUID=0 PID=3983712 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-user-sau-main-dev
[2026-01-18 23:23:20 UTC] USER=www-data EUID=0 PID=3983721 ACTION=passthru ARGS=mkdir -p /var/lib/grafana/obs-user-sau-main-dev
[2026-01-18 23:23:20 UTC] USER=www-data EUID=0 PID=3983730 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-user-sau-main-dev/provisioning/datasources
[2026-01-18 23:23:20 UTC] USER=www-data EUID=0 PID=3983739 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-user-sau-main-dev/provisioning/dashboards
[2026-01-18 23:23:20 UTC] USER=www-data EUID=0 PID=3983748 ACTION=passthru ARGS=mkdir -p /etc/grafana/obs-user-sau-main-dev/provisioning/notifiers
[0;34m[INFO][0m Creating Grafana configuration at /etc/grafana/obs-user-sau-main-dev/grafana.ini...
[0;32m[OK][0m Configuration created
[0;34m[INFO][0m Creating Prometheus datasource provisioning...
[0;32m[OK][0m Prometheus datasource provisioned
[0;34m[INFO][0m Creating Tempo datasource provisioning...
[0;32m[OK][0m Tempo datasource provisioned
[0;34m[INFO][0m Creating Loki datasource provisioning...
[0;32m[OK][0m Loki datasource provisioned
[0;34m[INFO][0m Creating ClickHouse datasource provisioning...
[0;32m[OK][0m Retrieved ClickHouse credentials from Secrets Manager
[0;32m[OK][0m ClickHouse datasource provisioned
[0;34m[INFO][0m Creating systemd service: grafana-obs-user-sau-main-dev
[0;32m[OK][0m Systemd service created
[2026-01-18 23:23:22 UTC] USER=www-data EUID=0 PID=3983867 ACTION=passthru ARGS=chown -R grafana:grafana /etc/grafana/obs-user-sau-main-dev
[2026-01-18 23:23:22 UTC] USER=www-data EUID=0 PID=3983876 ACTION=passthru ARGS=chown -R grafana:grafana /var/lib/grafana/obs-user-sau-main-dev
[2026-01-18 23:23:22 UTC] USER=www-data EUID=0 PID=3983885 ACTION=passthru ARGS=chmod 750 /etc/grafana/obs-user-sau-main-dev /var/lib/grafana/obs-user-sau-main-dev
[0;34m[INFO][0m Adding /etc/hosts entry for dashboards-user-sau-main-dev-grafana.fastorder.com -> 10.100.1.188
[1;33m[WARN][0m /etc/hosts entry already exists
[0;34m[INFO][0m Storing Grafana credentials in AWS Secrets Manager (if aws CLI present)...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/user/sau/main/dev/grafana/admin-RAx77H",
"Name": "fastorder/observability/user/sau/main/dev/grafana/admin",
"VersionId": "53082dda-0525-4c2d-8f8c-9c38af480e81"
}
[0;32m[OK][0m Credentials stored in AWS Secrets Manager: fastorder/observability/user/sau/main/dev/grafana/admin
[0;34m[INFO][0m Enabling and starting Grafana service...
[2026-01-18 23:23:23 UTC] USER=www-data EUID=0 PID=3983966 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:23:24 UTC] USER=www-data EUID=0 PID=3984015 ACTION=passthru ARGS=systemctl enable grafana-obs-user-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/grafana-obs-user-sau-main-dev.service → /etc/systemd/system/grafana-obs-user-sau-main-dev.service.
[2026-01-18 23:23:24 UTC] USER=www-data EUID=0 PID=3984061 ACTION=passthru ARGS=systemctl restart grafana-obs-user-sau-main-dev.service
[0;32m[OK][0m Service enabled and started
[0;34m[INFO][0m Validating deployment...
[2026-01-18 23:23:29 UTC] USER=www-data EUID=0 PID=3984117 ACTION=passthru ARGS=systemctl is-active --quiet grafana-obs-user-sau-main-dev.service
[0;32m[OK][0m ✅ Grafana is running
[1;33m[WARN][0m ⚠️ Grafana web interface not yet listening on port 3000
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m Grafana Dashboard URL: https://dashboards-user-sau-main-dev-grafana.fastorder.com:3000
[0;32m[OK][0m Username: admin
[0;32m[OK][0m Password is stored in AWS Secrets Manager at: fastorder/observability/user/sau/main/dev/grafana/admin
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Service logs (last 10 lines):
[2026-01-18 23:23:29 UTC] USER=www-data EUID=0 PID=3984128 ACTION=passthru ARGS=journalctl -u grafana-obs-user-sau-main-dev.service -n 10 --no-pager
Jan 18 23:23:29 web-03 grafana-obs-user-sau-main-dev[3984070]: logger=migrator t=2026-01-18T23:23:29.644164475Z level=info msg="Migration successfully executed" id="add snapshot local_directory column" duration=9.132338ms
Jan 18 23:23:29 web-03 grafana-obs-user-sau-main-dev[3984070]: logger=migrator t=2026-01-18T23:23:29.648105893Z level=info msg="Executing migration" id="add snapshot gms_snapshot_uid column"
Jan 18 23:23:29 web-03 grafana-obs-user-sau-main-dev[3984070]: logger=migrator t=2026-01-18T23:23:29.668139346Z level=info msg="Migration successfully executed" id="add snapshot gms_snapshot_uid column" duration=20.029424ms
Jan 18 23:23:29 web-03 grafana-obs-user-sau-main-dev[3984070]: logger=migrator t=2026-01-18T23:23:29.672281787Z level=info msg="Executing migration" id="add snapshot encryption_key column"
Jan 18 23:23:29 web-03 grafana-obs-user-sau-main-dev[3984070]: logger=migrator t=2026-01-18T23:23:29.681485739Z level=info msg="Migration successfully executed" id="add snapshot encryption_key column" duration=9.203973ms
Jan 18 23:23:29 web-03 grafana-obs-user-sau-main-dev[3984070]: logger=migrator t=2026-01-18T23:23:29.684329567Z level=info msg="Executing migration" id="add snapshot error_string column"
Jan 18 23:23:29 web-03 grafana-obs-user-sau-main-dev[3984070]: logger=migrator t=2026-01-18T23:23:29.694205429Z level=info msg="Migration successfully executed" id="add snapshot error_string column" duration=9.870283ms
Jan 18 23:23:29 web-03 grafana-obs-user-sau-main-dev[3984070]: logger=migrator t=2026-01-18T23:23:29.698103645Z level=info msg="Executing migration" id="create cloud_migration_resource table v1"
Jan 18 23:23:29 web-03 grafana-obs-user-sau-main-dev[3984070]: logger=migrator t=2026-01-18T23:23:29.699033798Z level=info msg="Migration successfully executed" id="create cloud_migration_resource table v1" duration=927.406µs
Jan 18 23:23:29 web-03 grafana-obs-user-sau-main-dev[3984070]: logger=migrator t=2026-01-18T23:23:29.70281486Z level=info msg="Executing migration" id="delete cloud_migration_snapshot.result column"
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Dashboards Deployed Successfully
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: grafana
[0;34m[INFO][0m FQDN: dashboards-user-sau-main-dev-grafana.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.188
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Registering Grafana in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Grafana
[INFO] Identifier: user-sau-main-dev-grafana
[INFO] Identifier Parent: cluster
[INFO] IP: 10.100.1.188
[INFO] Port: 3000
[INFO] FQDN: dashboards-user-sau-main-dev-grafana.fastorder.com
[INFO] Status: running
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: e9ec9a6f-752d-4601-9ab8-de5406638023
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[0;32m[OK][0m ✅ Grafana registered in dashboard
[0;32m[OK][0m Dashboards (grafana) deployed successfully
[0;34m[INFO][0m Step 10/10: Deploying alerting...
[0;34m[INFO][0m Provider: alertmanager (selected)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m ALERTING DEPLOYMENT
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: alertmanager
[0;34m[INFO][0m Observability Cell: obs-user-sau-main-dev
[0;34m[INFO][0m FQDN: alerts-user-sau-main-dev-alertmanager.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.228
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Ports: Web=9093 Cluster=9094 (bound to IP: 10.100.1.228)
[0;34m[INFO][0m Using provider: alertmanager
[0;34m[INFO][0m Provider script: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Alerting/provider/alertmanager.sh
[0;34m[INFO][0m Executing provider deployment script...
[0;34m[INFO][0m Parsed: SERVICE=user, ZONE=sau, BRANCH=main, ENV=dev
[0;34m[INFO][0m Binding to allocated IP: 10.100.1.228
[0;34m[INFO][0m Deploying Alertmanager for observability cell: obs-user-sau-main-dev
[0;34m[INFO][0m FQDN: alerts-user-sau-main-dev-alertmanager.fastorder.com
[0;34m[INFO][0m Allocated IP: 10.100.1.228
[0;34m[INFO][0m VM IP: 10.100.1.228
[0;34m[INFO][0m Ports: Web=9093 Cluster=9094
[0;34m[VERSION][0m Fetching latest version for alertmanager from GitHub (prometheus/alertmanager)...
[0;32m[VERSION][0m Latest alertmanager version: 0.30.1
[0;34m[INFO][0m Resolved Alertmanager version: 0.30.1
[0;34m[INFO][0m Checking if Alertmanager is installed...
[0;32m[OK][0m Alertmanager already installed at /usr/local/bin/alertmanager
[0;34m[INFO][0m Validating TLS certificate and key...
[0;32m[OK][0m TLS cert/key found in /etc/fastorder/observability/certs/obs-user-sau-main-dev
[0;34m[INFO][0m Creating configuration and data directories...
[2026-01-18 23:23:30 UTC] USER=www-data EUID=0 PID=3984235 ACTION=passthru ARGS=mkdir -p /etc/alertmanager/obs-user-sau-main-dev
[2026-01-18 23:23:31 UTC] USER=www-data EUID=0 PID=3984245 ACTION=passthru ARGS=mkdir -p /var/lib/alertmanager/obs-user-sau-main-dev
[2026-01-18 23:23:31 UTC] USER=www-data EUID=0 PID=3984254 ACTION=passthru ARGS=mkdir -p /etc/alertmanager/obs-user-sau-main-dev/templates
[0;34m[INFO][0m Creating Alertmanager configuration...
[0;32m[OK][0m Alertmanager configuration created at /etc/alertmanager/obs-user-sau-main-dev/alertmanager.yml
[0;34m[INFO][0m Creating notification templates...
[0;32m[OK][0m Notification templates created
[0;34m[INFO][0m Creating Alertmanager web TLS configuration with mTLS...
[0;32m[OK][0m Web mTLS configuration created at /etc/alertmanager/obs-user-sau-main-dev/web-config.yml
[0;34m[INFO][0m Validating Alertmanager configuration...
[2026-01-18 23:23:31 UTC] USER=www-data EUID=0 PID=3984292 ACTION=passthru ARGS=chmod 755 /etc/alertmanager/obs-user-sau-main-dev
[2026-01-18 23:23:31 UTC] USER=www-data EUID=0 PID=3984302 ACTION=passthru ARGS=chmod 644 /etc/alertmanager/obs-user-sau-main-dev/alertmanager.yml
Checking '/etc/alertmanager/obs-user-sau-main-dev/alertmanager.yml' SUCCESS
Found:
- global config
- route
- 6 inhibit rules
- 5 receivers
- 1 templates
SUCCESS
[0;32m[OK][0m ✅ Configuration is valid
[0;34m[INFO][0m Creating systemd service: alertmanager-obs-user-sau-main-dev
[0;32m[OK][0m Systemd service created
[2026-01-18 23:23:31 UTC] USER=www-data EUID=0 PID=3984336 ACTION=passthru ARGS=chown alertmanager:alertmanager /etc/fastorder/observability/certs/obs-user-sau-main-dev/alertmanager-key.pem
[2026-01-18 23:23:31 UTC] USER=www-data EUID=0 PID=3984345 ACTION=passthru ARGS=chown alertmanager:alertmanager /etc/fastorder/observability/certs/obs-user-sau-main-dev/alertmanager-cert.pem
[2026-01-18 23:23:31 UTC] USER=www-data EUID=0 PID=3984354 ACTION=passthru ARGS=chmod 644 /etc/fastorder/observability/certs/obs-user-sau-main-dev/ca-cert.pem
[2026-01-18 23:23:31 UTC] USER=www-data EUID=0 PID=3984363 ACTION=passthru ARGS=chown -R alertmanager:alertmanager /etc/alertmanager/obs-user-sau-main-dev
[2026-01-18 23:23:31 UTC] USER=www-data EUID=0 PID=3984372 ACTION=passthru ARGS=chown -R alertmanager:alertmanager /var/lib/alertmanager/obs-user-sau-main-dev
[2026-01-18 23:23:31 UTC] USER=www-data EUID=0 PID=3984381 ACTION=passthru ARGS=chmod 750 /etc/alertmanager/obs-user-sau-main-dev /var/lib/alertmanager/obs-user-sau-main-dev
[0;34m[INFO][0m Adding /etc/hosts entry for alerts-user-sau-main-dev-alertmanager.fastorder.com -> 10.100.1.228
[1;33m[WARN][0m /etc/hosts entry already exists
[0;34m[INFO][0m Storing Alertmanager configuration in AWS Secrets Manager (if aws CLI present)...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/observability/user/sau/main/dev/alertmanager/server-MyizBc",
"Name": "fastorder/observability/user/sau/main/dev/alertmanager/server",
"VersionId": "99e0a7ef-200a-4aad-a9d0-995e71431f73"
}
[0;32m[OK][0m Configuration stored in AWS Secrets Manager: fastorder/observability/user/sau/main/dev/alertmanager/server
[0;34m[INFO][0m Enabling and starting Alertmanager service...
[2026-01-18 23:23:32 UTC] USER=www-data EUID=0 PID=3984409 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:23:33 UTC] USER=www-data EUID=0 PID=3984457 ACTION=passthru ARGS=systemctl enable alertmanager-obs-user-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/alertmanager-obs-user-sau-main-dev.service → /etc/systemd/system/alertmanager-obs-user-sau-main-dev.service.
[2026-01-18 23:23:33 UTC] USER=www-data EUID=0 PID=3984511 ACTION=passthru ARGS=systemctl restart alertmanager-obs-user-sau-main-dev.service
[0;32m[OK][0m Service enabled and started
[0;34m[INFO][0m Validating deployment...
[2026-01-18 23:23:36 UTC] USER=www-data EUID=0 PID=3984645 ACTION=passthru ARGS=systemctl is-active --quiet alertmanager-obs-user-sau-main-dev.service
[0;32m[OK][0m ✅ Alertmanager is running
[0;32m[OK][0m ✅ Alertmanager HTTPS web interface listening on port 9093
[0;32m[OK][0m ✅ Alertmanager cluster port listening on port 9094
[1;33m[WARN][0m ⚠️ Alertmanager health check not responding yet (HTTPS)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m Alertmanager Web UI: https://alerts-user-sau-main-dev-alertmanager.fastorder.com:9093
[0;32m[OK][0m API Endpoint: https://alerts-user-sau-main-dev-alertmanager.fastorder.com:9093/api/v2
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Service logs (last 10 lines):
[2026-01-18 23:23:36 UTC] USER=www-data EUID=0 PID=3984669 ACTION=passthru ARGS=journalctl -u alertmanager-obs-user-sau-main-dev.service -n 10 --no-pager
Jan 18 23:23:33 web-03 systemd[1]: Started Alertmanager - obs-user-sau-main-dev.
Jan 18 23:23:33 web-03 alertmanager-obs-user-sau-main-dev[3984532]: ts=2026-01-18T23:23:33.959Z caller=main.go:245 level=info msg="Starting Alertmanager" version="(version=0.26.0, branch=HEAD, revision=d7b4f0c7322e7151d6e3b1e31cbc15361e295d8d)"
Jan 18 23:23:33 web-03 alertmanager-obs-user-sau-main-dev[3984532]: ts=2026-01-18T23:23:33.959Z caller=main.go:246 level=info build_context="(go=go1.20.7, platform=linux/amd64, user=root@df8d7debeef4, date=20230824-11:11:58, tags=netgo)"
Jan 18 23:23:33 web-03 alertmanager-obs-user-sau-main-dev[3984532]: ts=2026-01-18T23:23:33.961Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s
Jan 18 23:23:34 web-03 alertmanager-obs-user-sau-main-dev[3984532]: ts=2026-01-18T23:23:34.015Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/obs-user-sau-main-dev/alertmanager.yml
Jan 18 23:23:34 web-03 alertmanager-obs-user-sau-main-dev[3984532]: ts=2026-01-18T23:23:34.017Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/obs-user-sau-main-dev/alertmanager.yml
Jan 18 23:23:34 web-03 alertmanager-obs-user-sau-main-dev[3984532]: ts=2026-01-18T23:23:34.021Z caller=tls_config.go:274 level=info msg="Listening on" address=10.100.1.228:9093
Jan 18 23:23:34 web-03 alertmanager-obs-user-sau-main-dev[3984532]: ts=2026-01-18T23:23:34.023Z caller=tls_config.go:310 level=info msg="TLS is enabled." http2=true address=10.100.1.228:9093
Jan 18 23:23:35 web-03 alertmanager-obs-user-sau-main-dev[3984532]: ts=2026-01-18T23:23:35.962Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000506909s
Jan 18 23:23:36 web-03 alertmanager-obs-user-sau-main-dev[3984532]: 2026/01/18 23:23:36 http: TLS handshake error from 10.100.1.228:55440: tls: client didn't provide a certificate
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Alerting Deployed Successfully
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Provider: alertmanager
[0;34m[INFO][0m FQDN: alerts-user-sau-main-dev-alertmanager.fastorder.com
[0;34m[INFO][0m IP: 10.100.1.228
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Setting up HTTPS reverse proxy...
[0;34m[INFO][0m Backend port: 9093
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Alertmanager HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OBS Cell: obs-user-sau-main-dev
FQDN: alerts-user-sau-main-dev-alertmanager.fastorder.com
Backend: https://alerts-user-sau-main-dev-alertmanager.fastorder.com:9093/ (resolved via /etc/hosts)
Backend IP: 10.100.1.228
Email: admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verifying prerequisites...
[0;31m[ERROR][0m This script must be run as root or with sudo
[1;33m[WARN][0m ⚠️ HTTPS setup failed (Alertmanager is still running on HTTP)
[0;34m[INFO][0m Registering Alertmanager in monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Alertmanager
[INFO] Identifier: user-sau-main-dev-alertmanager
[INFO] Identifier Parent: cluster
[INFO] IP: 10.100.1.228
[INFO] Port: 9093
[INFO] FQDN: alerts-user-sau-main-dev-alertmanager.fastorder.com
[INFO] Status: running
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: c9d78146-3d61-4b9b-a231-f0a9de4fc1ba
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[0;32m[OK][0m ✅ Alertmanager registered in dashboard
[0;32m[OK][0m Alerting (alertmanager) deployed successfully
[0;34m[INFO][0m Step 10.5: Deploying Blackbox Exporter for synthetic monitoring...
[0;34m[VERSION][0m Fetching latest version for blackbox_exporter from GitHub (prometheus/blackbox_exporter)...
[0;32m[VERSION][0m Latest blackbox_exporter version: 0.28.0
[0;32m[BLACKBOX][0m Resolved Blackbox Exporter version: 0.28.0
[0;32m[BLACKBOX][0m Starting Blackbox Exporter deployment for obs-user-sau-main-dev
[0;32m[BLACKBOX][0m VM IP: 10.100.1.187
[0;32m[BLACKBOX][0m Version: 0.28.0
[0;32m[BLACKBOX][0m Checking prerequisites...
[0;32m[BLACKBOX][0m Creating directories...
[0;32m[BLACKBOX][0m Downloading Blackbox Exporter v0.28.0...
Sorry, user www-data is not allowed to execute '/usr/bin/mv /tmp/tmp.8oJlEPtSK5/blackbox_exporter-0.28.0.linux-amd64/blackbox_exporter /usr/local/bin/' as root on web-03.
[1;33m[WARN][0m Blackbox Exporter deployment failed (non-fatal, synthetic monitoring disabled)
[0;34m[INFO][0m Step 11/13: Configuring HTTPS reverse proxies...
[0;34m[INFO][0m Setting up Prometheus HTTPS proxy...
[2026-01-18 23:23:38 UTC] USER=www-data EUID=0 PID=3984779 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Metrics/https/setup-prometheus-https.sh --obs-cell obs-user-sau-main-dev --backend-ip 10.100.1.187
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Prometheus HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OBS Cell: obs-user-sau-main-dev
FQDN: metrics-user-sau-main-dev-prometheus.fastorder.com
Backend: https://metrics-user-sau-main-dev-prometheus.fastorder.com:9090/ (resolved via /etc/hosts)
Backend IP: 10.100.1.187
Email: admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verifying prerequisites...
[0;34m[INFO][0m Enabling Apache modules...
[0;34m[INFO][0m Testing backend connectivity (will retry up to 60s)...
[0;32m[OK][0m Backend is accessible
[0;34m[INFO][0m Creating HTTP VirtualHost for ACME challenge...
[0;32m[OK][0m HTTP VirtualHost created
[0;34m[INFO][0m Obtaining Let's Encrypt certificate...
[0;32m[OK][0m Certificate exists and is valid for 83 more days
[0;34m[INFO][0m Creating HTTPS VirtualHost with reverse proxy...
[0;32m[OK][0m HTTPS VirtualHost created and Apache reloaded
[0;34m[INFO][0m Setting up certificate auto-renewal...
[0;32m[OK][0m Auto-renewal configured
[0;34m[INFO][0m Updating /etc/hosts...
[0;32m[OK][0m /etc/hosts updated
[0;34m[INFO][0m Verifying HTTPS setup...
[0;32m[OK][0m HTTPS endpoint is working
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✅ Prometheus HTTPS Setup Complete[0m
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
HTTPS Endpoints:
Health: https://metrics-user-sau-main-dev-prometheus.fastorder.com/-/healthy
Ready: https://metrics-user-sau-main-dev-prometheus.fastorder.com/-/ready
Graph: https://metrics-user-sau-main-dev-prometheus.fastorder.com/graph
Targets: https://metrics-user-sau-main-dev-prometheus.fastorder.com/targets
Alerts: https://metrics-user-sau-main-dev-prometheus.fastorder.com/alerts
API: https://metrics-user-sau-main-dev-prometheus.fastorder.com/api/v1/...
Apache VirtualHosts:
HTTP: /etc/apache2/sites-available/metrics-user-sau-main-dev-prometheus.fastorder.com.conf
HTTPS: /etc/apache2/sites-available/metrics-user-sau-main-dev-prometheus.fastorder.com-ssl.conf
Certificate:
Path: /etc/letsencrypt/live/metrics-user-sau-main-dev-prometheus.fastorder.com/
Renewal: certbot renew --cert-name metrics-user-sau-main-dev-prometheus.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m Prometheus HTTPS proxy configured
[0;34m[INFO][0m Setting up Grafana HTTPS proxy...
[2026-01-18 23:23:41 UTC] USER=www-data EUID=0 PID=3984957 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Dashboards/https/setup-grafana-https.sh --obs-cell obs-user-sau-main-dev --backend-ip 10.100.1.188
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OBS Cell: obs-user-sau-main-dev
FQDN: dashboards-user-sau-main-dev-grafana.fastorder.com
Backend: https://dashboards-user-sau-main-dev-grafana.fastorder.com:3000/ (resolved via /etc/hosts)
Backend IP: 10.100.1.188
Email: admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verifying prerequisites...
[0;34m[INFO][0m Enabling Apache modules...
[0;34m[INFO][0m Testing backend connectivity...
[0;34m[INFO][0m Creating HTTP VirtualHost for ACME challenge...
[0;32m[OK][0m HTTP VirtualHost created
[0;34m[INFO][0m Obtaining Let's Encrypt certificate...
[0;32m[OK][0m Certificate already exists
[0;34m[INFO][0m Creating HTTPS VirtualHost...
[0;32m[OK][0m HTTPS VirtualHost created and Apache reloaded
[0;34m[INFO][0m Updating /etc/hosts...
[0;32m[OK][0m /etc/hosts updated
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana HTTPS Setup Complete!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana URL: https://dashboards-user-sau-main-dev-grafana.fastorder.com/
Metrics: https://dashboards-user-sau-main-dev-grafana.fastorder.com/metrics
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m Grafana HTTPS proxy configured
[0;34m[INFO][0m Setting up OpenTelemetry Collector HTTPS proxy...
[2026-01-18 23:23:42 UTC] USER=www-data EUID=0 PID=3985025 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Telemetry/https/setup-otelcol-https.sh --obs-cell obs-user-sau-main-dev --backend-ip 10.100.1.229
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OpenTelemetry Collector HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OBS Cell: obs-user-sau-main-dev
FQDN: telemetry-user-sau-main-dev-opentelemetry.fastorder.com
Backend: http://telemetry-user-sau-main-dev-opentelemetry.fastorder.com:8888/ (resolved via /etc/hosts)
Backend IP: 10.100.1.229
Email: admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verifying prerequisites...
[0;34m[INFO][0m Enabling Apache modules...
[0;34m[INFO][0m Testing backend connectivity...
[0;32m[OK][0m Backend is accessible and returning metrics via HTTPS
[0;34m[INFO][0m Creating HTTP VirtualHost for ACME challenge...
[0;32m[OK][0m HTTP VirtualHost created
[0;34m[INFO][0m Obtaining Let's Encrypt certificate...
[0;32m[OK][0m Certificate exists and is valid for 83 more days
[0;34m[INFO][0m Creating HTTPS VirtualHost with reverse proxy...
[0;32m[OK][0m HTTPS VirtualHost created and Apache reloaded
[0;34m[INFO][0m Setting up certificate auto-renewal...
[0;32m[OK][0m Auto-renewal configured
[0;34m[INFO][0m Updating /etc/hosts...
[0;32m[OK][0m /etc/hosts updated
[0;34m[INFO][0m Verifying HTTPS setup...
[0;32m[OK][0m HTTPS endpoint is working and returning metrics
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32mOpenTelemetry Collector HTTPS Setup Complete[0m
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
HTTPS Endpoints:
Metrics: https://telemetry-user-sau-main-dev-opentelemetry.fastorder.com/metrics
Apache VirtualHosts:
HTTP: /etc/apache2/sites-available/telemetry-user-sau-main-dev-opentelemetry.fastorder.com.conf
HTTPS: /etc/apache2/sites-available/telemetry-user-sau-main-dev-opentelemetry.fastorder.com-ssl.conf
Certificate:
Path: /etc/letsencrypt/live/telemetry-user-sau-main-dev-opentelemetry.fastorder.com/
Renewal: certbot renew --cert-name telemetry-user-sau-main-dev-opentelemetry.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m OpenTelemetry Collector HTTPS proxy configured
[0;34m[INFO][0m Setting up ClickHouse HTTPS proxy...
[2026-01-18 23:23:44 UTC] USER=www-data EUID=0 PID=3985191 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/LogStorageBackend/https/setup-clickhouse-https.sh --obs-cell obs-user-sau-main-dev --backend-ip 10.100.1.217
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ClickHouse HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OBS Cell: obs-user-sau-main-dev
FQDN: logstore-user-sau-main-dev.fastorder.com
Backend: http://logstore-user-sau-main-dev.fastorder.com:8123/ (resolved via /etc/hosts)
Backend IP: 10.100.1.217
Email: admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verifying prerequisites...
[0;34m[INFO][0m Enabling Apache modules...
[0;34m[INFO][0m Testing backend connectivity (will retry up to 60s)...
[0;32m[OK][0m Backend is accessible
[0;34m[INFO][0m Creating HTTP VirtualHost for ACME challenge...
[0;32m[OK][0m HTTP VirtualHost created
[0;34m[INFO][0m Obtaining Let's Encrypt certificate...
[0;32m[OK][0m Certificate exists and is valid for 83 more days
[0;34m[INFO][0m Creating HTTPS VirtualHost with reverse proxy...
[0;32m[OK][0m HTTPS VirtualHost created and Apache reloaded
[0;34m[INFO][0m Setting up certificate auto-renewal...
[0;32m[OK][0m Auto-renewal configured
[0;34m[INFO][0m Updating /etc/hosts...
[0;32m[OK][0m /etc/hosts updated
[0;34m[INFO][0m Verifying HTTPS setup...
[0;32m[OK][0m HTTPS endpoint is working
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✅ ClickHouse HTTPS Setup Complete[0m
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
HTTPS Endpoints:
Health: https://logstore-user-sau-main-dev.fastorder.com/
Dashboard: https://logstore-user-sau-main-dev.fastorder.com/dashboard
Playground: https://logstore-user-sau-main-dev.fastorder.com/play
Metrics: https://logstore-user-sau-main-dev.fastorder.com/metrics
Login Instructions:
1. Get credentials from skeleton: POST /api/monitoring/clickhouse/credentials
2. Use auto-login URL: https://logstore-user-sau-main-dev.fastorder.com/dashboard#user=<USER>&password=<PASS>
3. Or use skeleton monitoring dashboard for one-click access
Apache VirtualHosts:
HTTP: /etc/apache2/sites-available/logstore-user-sau-main-dev.fastorder.com.conf
HTTPS: /etc/apache2/sites-available/logstore-user-sau-main-dev.fastorder.com-ssl.conf
Certificate:
Path: /etc/letsencrypt/live/logstore-user-sau-main-dev.fastorder.com/
Auto-renewal: Enabled via certbot.timer
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ClickHouse HTTPS proxy configured
[0;34m[INFO][0m Setting up Tempo HTTPS proxy...
[2026-01-18 23:23:55 UTC] USER=www-data EUID=0 PID=3985754 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Traces/https/setup-tempo-https.sh --obs-cell obs-user-sau-main-dev --backend-ip 10.100.1.227
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Grafana Tempo HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OBS Cell: obs-user-sau-main-dev
FQDN: traces-user-sau-main-dev-tempo.fastorder.com
Backend: https://10.100.1.227:3200/
Backend IP: 10.100.1.227
Email: admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verifying prerequisites...
[0;34m[INFO][0m Enabling Apache modules...
[0;34m[INFO][0m Testing backend connectivity...
[1;33m[WARN][0m Cannot verify Tempo health endpoint (it may not be running yet), continuing anyway...
[0;34m[INFO][0m Creating HTTP VirtualHost for ACME challenge...
[0;32m[OK][0m HTTP VirtualHost created
[0;34m[INFO][0m Obtaining Let's Encrypt certificate...
[0;32m[OK][0m Certificate already exists
[0;34m[INFO][0m Generating Apache client certificate for mTLS backend connection...
[0;32m[OK][0m Apache client certificate already exists
[0;34m[INFO][0m Creating HTTPS VirtualHost with mTLS backend...
[0;32m[OK][0m HTTPS VirtualHost created and Apache reloaded
[0;34m[INFO][0m Updating /etc/hosts...
[0;32m[OK][0m Tempo HTTPS proxy configured
[0;34m[INFO][0m Setting up Alertmanager HTTPS proxy...
[2026-01-18 23:23:55 UTC] USER=www-data EUID=0 PID=3985830 ACTION=passthru ARGS=bash /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/Alerting/https/setup-alertmanager-https.sh --obs-cell obs-user-sau-main-dev --backend-ip 10.100.1.228
[0;34m[INFO][0m Backend port: 9093
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Alertmanager HTTPS Reverse Proxy Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
OBS Cell: obs-user-sau-main-dev
FQDN: alerts-user-sau-main-dev-alertmanager.fastorder.com
Backend: https://alerts-user-sau-main-dev-alertmanager.fastorder.com:9093/ (resolved via /etc/hosts)
Backend IP: 10.100.1.228
Email: admin@fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verifying prerequisites...
[0;34m[INFO][0m Enabling Apache modules...
[0;34m[INFO][0m Testing backend connectivity...
[1;33m[WARN][0m Backend health check inconclusive - proceeding anyway
[0;34m[INFO][0m Creating HTTP VirtualHost for ACME challenge...
[0;32m[OK][0m HTTP VirtualHost created
[0;34m[INFO][0m Obtaining Let's Encrypt certificate...
[0;32m[OK][0m Certificate exists and is valid for 83 more days
[0;34m[INFO][0m Creating HTTPS VirtualHost with reverse proxy...
[0;32m[OK][0m HTTPS VirtualHost created and Apache reloaded
[0;34m[INFO][0m Setting up certificate auto-renewal...
[0;32m[OK][0m Auto-renewal configured
[0;34m[INFO][0m Updating /etc/hosts...
[0;32m[OK][0m /etc/hosts updated
[0;34m[INFO][0m Verifying HTTPS setup...
OK[0;32m[OK][0m HTTPS endpoint is working
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✅ Alertmanager HTTPS Setup Complete[0m
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
HTTPS Endpoints:
Health: https://alerts-user-sau-main-dev-alertmanager.fastorder.com/-/healthy
Ready: https://alerts-user-sau-main-dev-alertmanager.fastorder.com/-/ready
Web UI: https://alerts-user-sau-main-dev-alertmanager.fastorder.com/
API: https://alerts-user-sau-main-dev-alertmanager.fastorder.com/api/v2/...
Apache VirtualHosts:
HTTP: /etc/apache2/sites-available/alerts-user-sau-main-dev-alertmanager.fastorder.com.conf
HTTPS: /etc/apache2/sites-available/alerts-user-sau-main-dev-alertmanager.fastorder.com-ssl.conf
Certificate:
Path: /etc/letsencrypt/live/alerts-user-sau-main-dev-alertmanager.fastorder.com/
Renewal: certbot renew --cert-name alerts-user-sau-main-dev-alertmanager.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m Alertmanager HTTPS proxy configured
[0;32m[OK][0m HTTPS reverse proxies configured
[0;34m[INFO][0m Step 12/13: Configuring firewall rules (network segmentation)...
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING FIREWALL RULES FOR OBSERVABILITY CELL
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Cell ID: obs-user-sau-main-dev
[0;34m[INFO][0m Internal Network: 10.0.0.0/8
[0;34m[INFO][0m Discovering dashboard/skeleton VM IPs...
[0;34m[INFO][0m Discovered skeleton IP: 142.93.238.16 (skeleton.fastorder.com)
[0;34m[INFO][0m Authorized dashboard IPs:
[0;34m[INFO][0m - 10.100.60.2
[0;34m[INFO][0m - 142.93.238.16
[0;34m[INFO][0m Configuring UFW firewall rules...
[0;34m[INFO][0m Allowing prometheus (port 9090) from internal network...
[2026-01-18 23:23:58 UTC] USER=www-data EUID=0 PID=3986031 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 9090 proto tcp comment Obs: prometheus from internal
ERROR: passthru not allowed: ufw
[0;34m[INFO][0m Allowing alertmanager (port 9093) from internal network...
[2026-01-18 23:23:58 UTC] USER=www-data EUID=0 PID=3986039 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 9093 proto tcp comment Obs: alertmanager from internal
ERROR: passthru not allowed: ufw
[0;34m[INFO][0m Allowing clickhouse (port 8123) from internal network...
[2026-01-18 23:23:59 UTC] USER=www-data EUID=0 PID=3986047 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 8123 proto tcp comment Obs: clickhouse from internal
ERROR: passthru not allowed: ufw
[0;34m[INFO][0m Allowing grafana (port 3000) from internal network...
[2026-01-18 23:23:59 UTC] USER=www-data EUID=0 PID=3986055 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 3000 proto tcp comment Obs: grafana from internal
ERROR: passthru not allowed: ufw
[0;34m[INFO][0m Allowing otelcol (port 4318) from internal network...
[2026-01-18 23:23:59 UTC] USER=www-data EUID=0 PID=3986064 ACTION=passthru ARGS=ufw allow from 10.0.0.0/8 to any port 4318 proto tcp comment Obs: otelcol from internal
ERROR: passthru not allowed: ufw
[0;34m[INFO][0m Allowing loki (port 3100) from internal network...
[0;34m[INFO][0m Allowing tempo (port 3200) from internal network...
[0;34m[INFO][0m Allowing dashboard access from 10.100.60.2...
[2026-01-18 23:23:59 UTC] USER=www-data EUID=0 PID=3986088 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 9090 proto tcp comment Dashboard: prometheus
ERROR: passthru not allowed: ufw
[2026-01-18 23:23:59 UTC] USER=www-data EUID=0 PID=3986112 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3000 proto tcp comment Dashboard: grafana
ERROR: passthru not allowed: ufw
[2026-01-18 23:23:59 UTC] USER=www-data EUID=0 PID=3986128 ACTION=passthru ARGS=ufw allow from 10.100.60.2 to any port 3100 proto tcp comment Dashboard: loki
ERROR: passthru not allowed: ufw
[0;34m[INFO][0m Allowing dashboard access from 142.93.238.16...
[2026-01-18 23:23:59 UTC] USER=www-data EUID=0 PID=3986152 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 9093 proto tcp comment Dashboard: alertmanager
[2026-01-18 23:23:59 UTC] USER=www-data EUID=0 PID=3986186 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 4318 proto tcp comment Dashboard: otelcol
ERROR: passthru not allowed: ufw
[2026-01-18 23:23:59 UTC] USER=www-data EUID=0 PID=3986194 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 3100 proto tcp comment Dashboard: loki
ERROR: passthru not allowed: ufw
[2026-01-18 23:23:59 UTC] USER=www-data EUID=0 PID=3986202 ACTION=passthru ARGS=ufw allow from 142.93.238.16 to any port 3200 proto tcp comment Dashboard: tempo
ERROR: passthru not allowed: ufw
[2026-01-18 23:23:59 UTC] USER=www-data EUID=0 PID=3986210 ACTION=passthru ARGS=ufw allow 443/tcp comment HTTPS obs-proxy
ERROR: passthru not allowed: ufw
[0;32m[OK][0m UFW firewall rules configured
[0;32m[OK][0m ═══════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Firewall configuration completed
[0;32m[OK][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Current firewall status:
[0;32m[OK][0m Firewall rules configured
[0;34m[INFO][0m Step 13/13: Configuring OAuth/SSO...
[0;34m[INFO][0m OAuth/SSO configuration script not found, skipping...
[0;34m[INFO][0m Running validation checks...
[0;34m[INFO][0m Validation script not found, skipping...
[0;34m[INFO][0m Registering observability components to dashboard...
[0;34m[INFO][0m Components to register: metrics alerts dashboards traces telemetry logstore proxy
[0;34m[INFO][0m Skipping metrics - registered by deploy script
[0;34m[INFO][0m Skipping alerts - registered by deploy script
[0;34m[INFO][0m Skipping dashboards - registered by deploy script
[0;34m[INFO][0m Skipping traces - registered by deploy script
[0;34m[INFO][0m Skipping telemetry - registered by deploy script
[0;34m[INFO][0m Skipping logstore - registered by deploy script
[0;34m[INFO][0m Processing component: proxy
[0;34m[INFO][0m Registering: proxy (obs-user-sau-main-dev-proxy)
[INFO] Detected observability component, parsing: user-sau-main-dev-proxy
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Observability Proxy
[INFO] Identifier: obs-user-sau-main-dev-proxy
[INFO] Identifier Parent: observability-cell
[INFO] IP: 10.100.1.166
[INFO] Port: 443
[INFO] FQDN: observe-user-sau-main-dev.fastorder.com
[INFO] Status: running
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 4254de5b-5b4a-4d10-8491-e279dc7070fd
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[0;32m[OK][0m ✓ Registered: proxy
[0;34m[INFO][0m Registering short DNS aliases...
[0;32m[OK][0m ✓ Observability components registration completed
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying all observability services are running...
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m[OK][0m ✓ prometheus-obs-user-sau-main-dev.service is running
[0;32m[OK][0m ✓ alertmanager-obs-user-sau-main-dev.service is running
[0;32m[OK][0m ✓ tempo-obs-user-sau-main-dev.service is running
[0;32m[OK][0m ✓ otelcol-obs-user-sau-main-dev.service is running
[0;32m[OK][0m ✓ All observability services verified running
═══════════════════════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ OBSERVABILITY CELL PROVISIONED: obs-user-sau-main-dev
═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m DNS Entries:
metrics-user-sau-main-dev-prometheus.fastorder.com (10.100.1.187)
alerts-user-sau-main-dev-alertmanager.fastorder.com (10.100.1.228)
dashboards-user-sau-main-dev-grafana.fastorder.com (10.100.1.188)
traces-user-sau-main-dev-tempo.fastorder.com (10.100.1.227)
telemetry-user-sau-main-dev-opentelemetry.fastorder.com (10.100.1.229)
logstore-user-sau-main-dev-clickhouse.fastorder.com (10.100.1.217)
observe-user-sau-main-dev.fastorder.com (10.100.1.166)
[0;34m[INFO][0m Secrets Path: fastorder/observability/user/sau/dev/*
[0;34m[INFO][0m Access (Purpose-Oriented URLs):
Dashboards: https://dashboards-user-sau-main-dev-grafana.fastorder.com (SSO enabled)
Metrics: https://metrics-user-sau-main-dev-prometheus.fastorder.com (internal only)
Alerts: https://alerts-user-sau-main-dev-alertmanager.fastorder.com
Log Storage: https://logstore-user-sau-main-dev-clickhouse.fastorder.com
[0;34m[INFO][0m Backend Implementation (Internal - Not Exposed to Clients):
Telemetry: otlp
Metrics: prometheus
Traces: tempo
Dashboards: grafana
Alerting: alertmanager
Log Storage: clickhouse
[0;34m[INFO][0m For applications in user-sau-main-dev:
- Metrics: Push to telemetry-user-sau-main-dev-opentelemetry.fastorder.com:4318 (OTLP/HTTP)
- Logs: Push to telemetry-user-sau-main-dev-opentelemetry.fastorder.com:4318 (OTLP/HTTP)
- Traces: Push to telemetry-user-sau-main-dev-opentelemetry.fastorder.com:4317 (OTLP/gRPC)
- Query Metrics: https://metrics-user-sau-main-dev-prometheus.fastorder.com
- Query Logs: https://logstore-user-sau-main-dev-clickhouse.fastorder.com
- Query Traces: https://traces-user-sau-main-dev-tempo.fastorder.com
[0;34m[INFO][0m Runbook: /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/RUNBOOK.md
═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m Using search engine from SEARCH_ENGINE environment variable: elasticsearch
[0;34m[INFO][0m Cleaning up any existing locks...
[0;32m[1mStarting search engine: elasticsearch[0m
[1;33m═══════════════════════════════════════════════[0m
[0;36m[1m════════════════════════════════════════════════════════════════[0m
[0;36m[1m Elasticsearch Deployment Runner [0m
[0;36m[1m════════════════════════════════════════════════════════════════[0m
[0;34m[INFO][0m Cleaning up any existing locks (without triggering package configurations)...
[1;33m[WARNING][0m Lock cleanup skipped (wrapper not available or insufficient permissions)
[0;32m[1m🚀 Auto mode enabled - running automatic installation[0m
[0;32m[1mStarting Automatic Installation...[0m
[1;33m═══════════════════════════════════════════════[0m
[0;34mWill execute all deployment tasks in sequence:[0m
[0;32m[1m[1][0m Install Elasticsearch Http [0;35m(01-install-elasticsearch-http)[0m
[0;32m[1m[2][0m Make Https [0;35m(02-make-https)[0m
[0;32m[1m[3][0m Create Index Llm [0;35m(03-create-index-llm)[0m
[0;32m[1m[4][0m Monitoring Setup [0;35m(10-monitoring-setup)[0m
[1;33m═══════════════════════════════════════════════[0m
[0;32m🚀 Auto mode - proceeding automatically...[0m
[0;32m[1mRunning automatic installation...[0m
[0;36m[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;32m[1mStep 1: Executing Install Elasticsearch Http[0m
[0;35mFolder: 01-install-elasticsearch-http[0m
[0;36m[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
=== Elasticsearch HTTP Setup ===
Install and configure Elasticsearch with HTTP access
Architecture: Per-node VM IPs with default port (9200)
[INFO] Using web-provided environment: user-sau-main-dev
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment:
Nodes: 1
Port: 9200 (default Elasticsearch port)
Coordinator endpoint: http://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
Checking if Elasticsearch is already installed for environment: ...
Validating Elasticsearch installation...
./run.sh: line 132: /var/www/html/skeleton.dev.fastorder.com/fixing/scripts/lib/elasticsearch_validator.sh: No such file or directory
⚠️ Elasticsearch installation issues detected. Attempting automatic repair...
./run.sh: line 134: /var/www/html/skeleton.dev.fastorder.com/fixing/scripts/lib/elasticsearch_validator.sh: No such file or directory
Executing: steps/01-setup-directories.sh
+ 01-setup-directories.sh:4:main: echo '=== Step 1: Creating directory structure ==='
=== Step 1: Creating directory structure ===
+++ 01-setup-directories.sh:4:main: dirname steps/01-setup-directories.sh
++ 01-setup-directories.sh:4:main: cd steps
++ 01-setup-directories.sh:4:main: pwd
+ 01-setup-directories.sh:4:main: SCRIPT_DIR=/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps
+ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/lib/provisioning-init.sh
++ 01-setup-directories.sh:4:main: RED='\033[0;31m'
++ 01-setup-directories.sh:4:main: GREEN='\033[0;32m'
++ 01-setup-directories.sh:4:main: YELLOW='\033[1;33m'
++ 01-setup-directories.sh:4:main: BLUE='\033[0;34m'
++ 01-setup-directories.sh:4:main: NC='\033[0m'
++ 01-setup-directories.sh:4:main: export TERM=dumb
++ 01-setup-directories.sh:4:main: TERM=dumb
++ 01-setup-directories.sh:4:main: export DEBIAN_FRONTEND=noninteractive
++ 01-setup-directories.sh:4:main: DEBIAN_FRONTEND=noninteractive
++ 01-setup-directories.sh:4:main: export NEEDRESTART_MODE=a
++ 01-setup-directories.sh:4:main: NEEDRESTART_MODE=a
++ 01-setup-directories.sh:4:main: export NEEDRESTART_SUSPEND=1
++ 01-setup-directories.sh:4:main: NEEDRESTART_SUSPEND=1
++ 01-setup-directories.sh:4:main: export DEBIAN_PRIORITY=critical
++ 01-setup-directories.sh:4:main: DEBIAN_PRIORITY=critical
++ 01-setup-directories.sh:4:main: export UCF_FORCE_CONFFOLD=1
++ 01-setup-directories.sh:4:main: UCF_FORCE_CONFFOLD=1
++ 01-setup-directories.sh:4:main: export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
++ 01-setup-directories.sh:4:main: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
++ 01-setup-directories.sh:4:main: [[ -n '' ]]
++ 01-setup-directories.sh:4:main: [[ -n /opt/fastorder/bash/scripts/env_app_setup/state ]]
++ 01-setup-directories.sh:4:main: [[ -d /opt/fastorder/bash/scripts/env_app_setup/state ]]
++ 01-setup-directories.sh:4:main: STATE_DIR=/opt/fastorder/bash/scripts/env_app_setup/state
++ 01-setup-directories.sh:4:main: export STATE_DIR
++ 01-setup-directories.sh:4:main: [[ -f /opt/fastorder/bash/scripts/env_app_setup/setup/setup.json ]]
++ 01-setup-directories.sh:4:main: SETUP_JSON=/opt/fastorder/bash/scripts/env_app_setup/setup/setup.json
++ 01-setup-directories.sh:4:main: FO_WRAPPER=/usr/local/bin/fastorder-provisioning-wrapper.sh
++ 01-setup-directories.sh:4:main: HTTP_PORT_BASE=9200
++ 01-setup-directories.sh:4:main: TRANSPORT_PORT_BASE=9300
++ 01-setup-directories.sh:4:main: PG_PORT_BASE=5432
++ 01-setup-directories.sh:4:main: APP_IP_SUBNETS=(['observability']='10.100.5' ['obs']='10.100.5' ['prometheus']='10.100.5' ['grafana']='10.100.5' ['loki']='10.100.5' ['tempo']='10.100.5' ['postgresql']='10.100.10' ['postgres']='10.100.10' ['pg']='10.100.10' ['elasticsearch']='10.100.20' ['es']='10.100.20' ['kafka']='10.100.30' ['redis']='10.100.40' ['mongodb']='10.100.50' ['mongo']='10.100.50' ['iam']='10.100.60' ['keycloak']='10.100.60' ['general']='10.100.1')
++ 01-setup-directories.sh:4:main: declare -A APP_IP_SUBNETS
++ 01-setup-directories.sh:4:main: APP_IP_RESERVED_START=(['observability']='2' ['postgresql']='2' ['elasticsearch']='2' ['kafka']='2' ['redis']='2' ['mongodb']='2' ['iam']='2' ['general']='50')
++ 01-setup-directories.sh:4:main: declare -A APP_IP_RESERVED_START
++ 01-setup-directories.sh:4:main: APP_IP_RESERVED_END=(['observability']='49' ['postgresql']='254' ['elasticsearch']='254' ['kafka']='254' ['redis']='254' ['mongodb']='254' ['iam']='254' ['general']='250')
++ 01-setup-directories.sh:4:main: declare -A APP_IP_RESERVED_END
+++ 01-setup-directories.sh:4:main: dirname /opt/fastorder/bash/scripts/env_app_setup/lib/provisioning-init.sh
++ 01-setup-directories.sh:4:main: _CONFIG_MGMT_LIB=/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
++ 01-setup-directories.sh:4:main: [[ -f /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh ]]
++ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
+++ 01-setup-directories.sh:4:main: set -Eeuo pipefail
+++ 01-setup-directories.sh:4:main: : /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
+++ 01-setup-directories.sh:4:main: STATE_DIR=/opt/fastorder/bash/scripts/env_app_setup/state
+++ 01-setup-directories.sh:4:main: VERSION_FETCHER_LIB=/opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/lib/version_fetcher.sh
+++ 01-setup-directories.sh:4:main: [[ -f /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/lib/version_fetcher.sh ]]
+++ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/setup/02-observability-cell/lib/version_fetcher.sh
++ 01-setup-directories.sh:4:main: [[ /opt/fastorder/bash/scripts/env_app_setup/lib/provisioning-init.sh == \s\t\e\p\s\/\0\1\-\s\e\t\u\p\-\d\i\r\e\c\t\o\r\i\e\s\.\s\h ]]
++ 01-setup-directories.sh:4:main: set +e
++ 01-setup-directories.sh:4:main: set +u
++ 01-setup-directories.sh:4:main: set +o pipefail
++ 01-setup-directories.sh:4:main: set +E
+ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh
++ 01-setup-directories.sh:4:main: [[ /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh == \s\t\e\p\s\/\0\1\-\s\e\t\u\p\-\d\i\r\e\c\t\o\r\i\e\s\.\s\h ]]
+ 01-setup-directories.sh:4:main: init_environment
+ 01-setup-directories.sh:4:main: require_bin jq
+ 01-setup-directories.sh:4:main: for b in "$@"
+ 01-setup-directories.sh:4:main: command -v jq
+ 01-setup-directories.sh:4:main: local app_type=general
+ 01-setup-directories.sh:4:main: ENV_ID=user-sau-main-dev
+ 01-setup-directories.sh:4:main: [[ -z user-sau-main-dev ]]
+ 01-setup-directories.sh:4:main: [[ -z user-sau-main-dev ]]
+ 01-setup-directories.sh:4:main: ENV_ID=user-sau-main-dev
+ 01-setup-directories.sh:4:main: [[ -z user-sau-main-dev ]]
+ 01-setup-directories.sh:4:main: [[ -z user-sau-main-dev ]]
+ 01-setup-directories.sh:4:main: [[ -z user-sau-main-dev ]]
++ 01-setup-directories.sh:4:main: env_dir_for user-sau-main-dev
++ 01-setup-directories.sh:4:main: echo /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev
+ 01-setup-directories.sh:4:main: ENV_DIR=/opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev
++ 01-setup-directories.sh:4:main: topo_path_for user-sau-main-dev
++ 01-setup-directories.sh:4:main: echo /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: TOPOLOGY_JSON=/opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: [[ ! -f /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json ]]
+ 01-setup-directories.sh:4:main: validate_topology_json /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: local topo=/opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: [[ -r /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json ]]
+ 01-setup-directories.sh:4:main: jq -e '
.schema_version == 1
and (.general.id | type=="string")
and (.general.shared_ip | type=="string")
and (.general.service | type=="string")
and (.general.zone | type=="string")
and (.general.env | type=="string")
' /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
++ 01-setup-directories.sh:4:main: jq -r .general.service /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: SERVICE=user
++ 01-setup-directories.sh:4:main: jq -r .general.zone /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: ZONE=sau
++ 01-setup-directories.sh:4:main: jq -r .general.branch /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: BRANCH=main
++ 01-setup-directories.sh:4:main: jq -r .general.env /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: ENV=dev
++ 01-setup-directories.sh:4:main: jq -r '.general.es_nodes_num // 3' /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: ES_NODES_NUM=1
++ 01-setup-directories.sh:4:main: jq -r '.general.pg_workers_num // 3' /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: PG_WORKERS_NUM=1
++ 01-setup-directories.sh:4:main: jq -r '.general.pg_WORKERS_STANDBY_NUM // 3' /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: PG_WORKERS_STANDBY_NUM=3
++ 01-setup-directories.sh:4:main: jq -r '.general.pg_citus_enabled // "yes"' /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: PG_CITUS_ENABLED=yes
+ 01-setup-directories.sh:4:main: [[ general != \g\e\n\e\r\a\l ]]
++ 01-setup-directories.sh:4:main: jq -r .general.shared_ip /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: VM_IP=142.93.238.16
+ 01-setup-directories.sh:4:main: [[ general != \g\e\n\e\r\a\l ]]
++ 01-setup-directories.sh:4:main: jq -r '.general.shared_iface // empty' /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: IFACE=eth0:16
+ 01-setup-directories.sh:4:main: local FINAL_VM_IP=142.93.238.16
+ 01-setup-directories.sh:4:main: set -a
+ 01-setup-directories.sh:4:main: [[ -r /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/generated/general.env ]]
+ 01-setup-directories.sh:4:main: source /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/generated/general.env
++ 01-setup-directories.sh:4:main: ENV_ID=user-sau-main-dev
++ 01-setup-directories.sh:4:main: SERVICE=user
++ 01-setup-directories.sh:4:main: zone=sau
++ 01-setup-directories.sh:4:main: BRANCH=main
++ 01-setup-directories.sh:4:main: ENV=dev
++ 01-setup-directories.sh:4:main: VM_IP=142.93.238.16
++ 01-setup-directories.sh:4:main: IFACE=eth0:16
++ 01-setup-directories.sh:4:main: ROOT_DIR=/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
++ 01-setup-directories.sh:4:main: ENV_DIR=/opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev
++ 01-setup-directories.sh:4:main: TOPOLOGY_JSON=/opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
++ 01-setup-directories.sh:4:main: LOG_LEVEL=info
++ 01-setup-directories.sh:4:main: DEBUG_MODE=false
+ 01-setup-directories.sh:4:main: set +a
+ 01-setup-directories.sh:4:main: VM_IP=142.93.238.16
+ 01-setup-directories.sh:4:main: export ENV_ID SERVICE ZONE BRANCH ENV VM_IP IFACE ENV_DIR TOPOLOGY_JSON
+ 01-setup-directories.sh:4:main: export ES_NODES_NUM PG_WORKERS_NUM PG_WORKERS_STANDBY_NUM PG_CITUS_ENABLED
+ 01-setup-directories.sh:4:main: [[ general != \g\e\n\e\r\a\l ]]
+ 01-setup-directories.sh:4:main: info 'Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)'
+ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)'
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
+ 01-setup-directories.sh:4:main: return 0
+ 01-setup-directories.sh:4:main: SERVICE=user
+ 01-setup-directories.sh:4:main: ZONE=sau
+ 01-setup-directories.sh:4:main: BRANCH=main
+ 01-setup-directories.sh:4:main: ENV=dev
++ 01-setup-directories.sh:4:main: env_id
++ 01-setup-directories.sh:4:main: '[' user = auth ']'
++ 01-setup-directories.sh:4:main: '[' user = item ']'
++ 01-setup-directories.sh:4:main: echo user-sau-main-dev
+ 01-setup-directories.sh:4:main: ENV_ID=user-sau-main-dev
+ 01-setup-directories.sh:4:main: env=user-sau-main-dev
+ 01-setup-directories.sh:4:main: nodes=1
+ 01-setup-directories.sh:4:main: [[ 1 =~ ^[1-9][0-9]*$ ]]
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /etc/elasticsearch
[2026-01-18 23:24:13 UTC] USER=www-data EUID=0 PID=3986595 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /data/elasticsearch
[2026-01-18 23:24:13 UTC] USER=www-data EUID=0 PID=3986605 ACTION=fsop ARGS=mkdir -p /data/elasticsearch
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /var/log/elasticsearch
[2026-01-18 23:24:13 UTC] USER=www-data EUID=0 PID=3986614 ACTION=fsop ARGS=mkdir -p /var/log/elasticsearch
+ 01-setup-directories.sh:4:main: APP_NAME=search
+ 01-setup-directories.sh:4:main: TOPOLOGY_FILE=/opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: command -v jq
+ 01-setup-directories.sh:4:main: [[ -f /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json ]]
++ 01-setup-directories.sh:4:main: jq -r --arg app search '.applications[$app].vm_ip // empty' /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: COORD_IP=10.100.1.89
+ 01-setup-directories.sh:4:main: [[ -z 10.100.1.89 ]]
+ 01-setup-directories.sh:4:main: [[ 10.100.1.89 == \n\u\l\l ]]
++ 01-setup-directories.sh:4:main: get_application_domain search
++ 01-setup-directories.sh:4:main: local app_type=search
++ 01-setup-directories.sh:4:main: [[ search == \g\e\n\e\r\a\l ]]
++ 01-setup-directories.sh:4:main: jq -r --arg app search '.applications[$app].domain // empty' /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
+ 01-setup-directories.sh:4:main: COORD_DOMAIN=search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com
+ 01-setup-directories.sh:4:main: info 'Coordinator exists: search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.89)'
+ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Coordinator exists: search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.89)'
[INFO] Coordinator exists: search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.89)
+ 01-setup-directories.sh:4:main: (( i=1 ))
+ 01-setup-directories.sh:4:main: (( i<=nodes ))
++ 01-setup-directories.sh:4:main: printf %02d 1
+ 01-setup-directories.sh:4:main: node_num=01
+ 01-setup-directories.sh:4:main: IDENTIFIER=node-01
+ 01-setup-directories.sh:4:main: APP_NAME=search-node-01
+ 01-setup-directories.sh:4:main: read -r NODE_IP NODE_DOMAIN
++ 01-setup-directories.sh:4:main: setup_directories_per_node node-01 search-node-01
++ 01-setup-directories.sh:4:main: local IDENTIFIER=node-01
++ 01-setup-directories.sh:4:main: local APP_NAME=search-node-01
++ 01-setup-directories.sh:4:main: local env
+++ 01-setup-directories.sh:4:main: env_id
+++ 01-setup-directories.sh:4:main: '[' user = auth ']'
+++ 01-setup-directories.sh:4:main: '[' user = item ']'
+++ 01-setup-directories.sh:4:main: echo user-sau-main-dev
++ 01-setup-directories.sh:4:main: env=user-sau-main-dev
++ 01-setup-directories.sh:4:main: local TOPOLOGY_FILE=/opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
++ 01-setup-directories.sh:4:main: info 'Setting up Elasticsearch node: node-01'
++ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Setting up Elasticsearch node: node-01'
++ 01-setup-directories.sh:4:main: local NODE_IP NODE_DOMAIN
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /etc/elasticsearch/user-sau-main-dev/node-01 /etc/elasticsearch/user-sau-main-dev-node-01
+++ 01-setup-directories.sh:4:main: jq -r --arg app search-node-01 '.applications[$app].vm_ip // empty' /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
[2026-01-18 23:24:13 UTC] USER=www-data EUID=0 PID=3986635 ACTION=fsop ARGS=ln -sfn /etc/elasticsearch/user-sau-main-dev/node-01 /etc/elasticsearch/user-sau-main-dev-node-01
+ 01-setup-directories.sh:4:main: [[ 1 -eq 1 ]]
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /etc/elasticsearch/user-sau-main-dev/node-01 /etc/elasticsearch/current
++ 01-setup-directories.sh:4:main: NODE_IP=10.100.1.152
++ 01-setup-directories.sh:4:main: [[ -z 10.100.1.152 ]]
++ 01-setup-directories.sh:4:main: [[ 10.100.1.152 == \n\u\l\l ]]
+++ 01-setup-directories.sh:4:main: get_application_domain search-node-01
+++ 01-setup-directories.sh:4:main: local app_type=search-node-01
+++ 01-setup-directories.sh:4:main: [[ search-node-01 == \g\e\n\e\r\a\l ]]
+++ 01-setup-directories.sh:4:main: jq -r --arg app search-node-01 '.applications[$app].domain // empty' /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/topology.json
[2026-01-18 23:24:13 UTC] USER=www-data EUID=0 PID=3986648 ACTION=fsop ARGS=ln -sfn /etc/elasticsearch/user-sau-main-dev/node-01 /etc/elasticsearch/current
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /data/elasticsearch/user-sau-main-dev/node-01 /data/elasticsearch/current
++ 01-setup-directories.sh:4:main: NODE_DOMAIN=search-user-sau-main-dev-elasticsearch-node-01.fastorder.com
++ 01-setup-directories.sh:4:main: info 'Using existing node-01: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.152)'
++ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Using existing node-01: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.152)'
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh: line 13: printf: write error: Broken pipe
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /etc/elasticsearch/user-sau-main-dev/node-01/certs
[2026-01-18 23:24:13 UTC] USER=www-data EUID=0 PID=3986657 ACTION=fsop ARGS=ln -sfn /data/elasticsearch/user-sau-main-dev/node-01 /data/elasticsearch/current
+ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop ln -sfn /var/log/elasticsearch/user-sau-main-dev/node-01 /var/log/elasticsearch/current
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /data/elasticsearch/user-sau-main-dev/node-01/tmp
+ 01-setup-directories.sh:4:main: (( i++ ))
+ 01-setup-directories.sh:4:main: (( i<=nodes ))
+ 01-setup-directories.sh:4:main: success 'Directory structure created for '\''user-sau-main-dev'\'' with 1 node(s).'
+ 01-setup-directories.sh:4:main: printf '[ OK ] %s\n' 'Directory structure created for '\''user-sau-main-dev'\'' with 1 node(s).'
[ OK ] Directory structure created for 'user-sau-main-dev' with 1 node(s).
Executing: steps/02-install-dependencies.sh
=== Step 2: Installing/Validating Elasticsearch (latest) ===
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop mkdir -p /var/log/elasticsearch/user-sau-main-dev/node-01
++ 01-setup-directories.sh:4:main: id -u elasticsearch
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-sau-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chown -R elasticsearch:elasticsearch /data/elasticsearch/user-sau-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chown -R elasticsearch:elasticsearch /var/log/elasticsearch/user-sau-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chmod 0750 /etc/elasticsearch/user-sau-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chmod 0750 /data/elasticsearch/user-sau-main-dev/node-01
++ 01-setup-directories.sh:4:main: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop chmod 0750 /var/log/elasticsearch/user-sau-main-dev/node-01
++ 01-setup-directories.sh:4:main: info 'Created dirs for user-sau-main-dev/node-01 @ 10.100.1.152'
++ 01-setup-directories.sh:4:main: printf '[INFO] %s\n' 'Created dirs for user-sau-main-dev/node-01 @ 10.100.1.152'
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh: line 13: printf: write error: Broken pipe
++ 01-setup-directories.sh:4:main: printf '%s\n' 10.100.1.152
/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh: line 59: printf: write error: Broken pipe
++ 01-setup-directories.sh:4:main: printf '%s\n' search-user-sau-main-dev-elasticsearch-node-01.fastorder.com
/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/01-install-elasticsearch-http/steps/lib/setup_directories_per_node.sh: line 60: printf: write error: Broken pipe
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] Cleaning dpkg/apt locks...
[2026-01-18 23:24:14 UTC] USER=www-data EUID=0 PID=3986775 ACTION=cleanup-dpkg-locks ARGS=
steps/02-install-dependencies.sh: line 16: 3986773 Killed command sudo -n "$WRAP" cleanup-dpkg-locks
[2026-01-18 23:24:14 UTC] USER=www-data EUID=0 PID=3986787 ACTION=fsop ARGS=mkdir -p /etc/apt/keyrings
[2026-01-18 23:24:14 UTC] USER=www-data EUID=0 PID=3986801 ACTION=fsop ARGS=chmod 0755 /etc/apt/keyrings
[INFO] apt-get update…
[2026-01-18 23:24:14 UTC] USER=www-data EUID=0 PID=3986815 ACTION=pkg ARGS=update
Hit:1 http://apt.postgresql.org/pub/repos/apt jammy-pgdg InRelease
Hit:2 https://artifacts.elastic.co/packages/8.x/apt stable InRelease
Hit:3 https://apt.grafana.com stable InRelease
Hit:4 https://packages.confluent.io/deb/7.6 stable InRelease
Hit:5 https://packages.microsoft.com/repos/azure-cli jammy InRelease
Hit:6 https://deb.nodesource.com/node_22.x nodistro InRelease
Hit:7 https://mirrors.edge.kernel.org/ubuntu jammy InRelease
Get:8 https://mirrors.edge.kernel.org/ubuntu jammy-backports InRelease [127 kB]
Get:9 https://mirrors.edge.kernel.org/ubuntu jammy-security InRelease [129 kB]
Get:10 https://mirrors.edge.kernel.org/ubuntu jammy-updates InRelease [128 kB]
Hit:11 https://packages.clickhouse.com/deb stable InRelease
Hit:12 https://ppa.launchpadcontent.net/ondrej/php/ubuntu jammy InRelease
Hit:13 https://repos.citusdata.com/community/ubuntu jammy InRelease
Fetched 384 kB in 1s (295 kB/s)
Reading package lists...
[INFO] Installed version : 8.19.10
[INFO] Candidate version : 8.19.10
✅ Elasticsearch already at latest (or only) available version.
✅ Elasticsearch installation validated.
🎉 Dependencies installed and up-to-date.
Executing: steps/03-create-env-configs.sh
=== Step 3: Creating environment configurations (master + nodes, TLS, units) ===
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Configuring env: user-sau-main-dev (nodes: 1, http: 9200, transport: 9300)
Using heap size: 1024m per node
[2026-01-18 23:24:27 UTC] USER=www-data EUID=0 PID=3988103 ACTION=fsop ARGS=chown root:root /etc/default/elasticsearch
[2026-01-18 23:24:27 UTC] USER=www-data EUID=0 PID=3988112 ACTION=fsop ARGS=chmod 0644 /etc/default/elasticsearch
[2026-01-18 23:24:28 UTC] USER=www-data EUID=0 PID=3988141 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/jvm.options
[2026-01-18 23:24:28 UTC] USER=www-data EUID=0 PID=3988150 ACTION=fsop ARGS=chmod 0644 /etc/elasticsearch/jvm.options
[2026-01-18 23:24:28 UTC] USER=www-data EUID=0 PID=3988168 ACTION=fsop ARGS=mkdir -p /etc/systemd/system.conf.d /etc/systemd/user.conf.d
[2026-01-18 23:24:28 UTC] USER=www-data EUID=0 PID=3988195 ACTION=passthru ARGS=systemctl daemon-reload
Current max_map_count: 262144
Current swappiness: 1
[2026-01-18 23:24:28 UTC] USER=www-data EUID=0 PID=3988272 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/log4j2.properties
[2026-01-18 23:24:28 UTC] USER=www-data EUID=0 PID=3988281 ACTION=fsop ARGS=chmod 0644 /etc/elasticsearch/log4j2.properties
[2026-01-18 23:24:28 UTC] USER=www-data EUID=0 PID=3988290 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-sau-main-dev/template
[2026-01-18 23:24:28 UTC] USER=www-data EUID=0 PID=3988299 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/user-sau-main-dev /etc/elasticsearch/user-sau-main-dev/template
[2026-01-18 23:24:28 UTC] USER=www-data EUID=0 PID=3988309 ACTION=fsop ARGS=chmod 0755 /etc/elasticsearch/user-sau-main-dev
[2026-01-18 23:24:28 UTC] USER=www-data EUID=0 PID=3988318 ACTION=fsop ARGS=cp /etc/elasticsearch/jvm.options /etc/elasticsearch/user-sau-main-dev/template/jvm.options
[INFO] 🌐 Registering general environment domain: user-sau-main-dev.fastorder.com
[INFO] Allocated VM IP: 10.100.1.51 for general environment
[INFO] Configuring VM IP 10.100.1.51 on network interface...
[1;33m[WARNING][0m VM IP may already be configured or need manual setup
[1;33m[WARNING][0m Warning: VM IP 10.100.1.51 not found on network interfaces
[ OK ] ✅ Registered general domain user-sau-main-dev.fastorder.com -> 10.100.1.51
[ OK ] ✅ DNS resolution verified for user-sau-main-dev.fastorder.com
[INFO] → Configuring user-sau-main-dev-node-01 (10.100.1.152) roles=[ master, data, data_hot, data_content, ingest ]
[2026-01-18 23:24:29 UTC] USER=www-data EUID=0 PID=3988421 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-sau-main-dev/node-01/certs /data/elasticsearch/user-sau-main-dev/node-01/tmp /var/log/elasticsearch/user-sau-main-dev/node-01
[2026-01-18 23:24:29 UTC] USER=www-data EUID=0 PID=3988430 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-sau-main-dev/node-01
[2026-01-18 23:24:29 UTC] USER=www-data EUID=0 PID=3988439 ACTION=fsop ARGS=chmod 0750 /etc/elasticsearch/user-sau-main-dev/node-01 /data/elasticsearch/user-sau-main-dev/node-01 /var/log/elasticsearch/user-sau-main-dev/node-01
[2026-01-18 23:24:29 UTC] USER=www-data EUID=0 PID=3988448 ACTION=fsop ARGS=cp /etc/elasticsearch/user-sau-main-dev/template/jvm.options /etc/elasticsearch/user-sau-main-dev/node-01/jvm.options
[2026-01-18 23:24:29 UTC] USER=www-data EUID=0 PID=3988457 ACTION=fsop ARGS=sed -i s/^-Xms.*/-Xms1024m/ /etc/elasticsearch/user-sau-main-dev/node-01/jvm.options
[2026-01-18 23:24:29 UTC] USER=www-data EUID=0 PID=3988466 ACTION=fsop ARGS=sed -i s/^-Xmx.*/-Xmx1024m/ /etc/elasticsearch/user-sau-main-dev/node-01/jvm.options
[2026-01-18 23:24:29 UTC] USER=www-data EUID=0 PID=3988487 ACTION=fsop ARGS=cp /etc/elasticsearch/log4j2.properties /etc/elasticsearch/user-sau-main-dev/node-01/log4j2.properties
[2026-01-18 23:24:29 UTC] USER=www-data EUID=0 PID=3988557 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-sau-main-dev/node-01
[2026-01-18 23:24:29 UTC] USER=www-data EUID=0 PID=3988572 ACTION=fsop ARGS=chmod 0644 /etc/elasticsearch/user-sau-main-dev/node-01/elasticsearch.yml
[2026-01-18 23:24:30 UTC] USER=www-data EUID=0 PID=3988599 ACTION=fsop ARGS=chmod 0644 /etc/default/elasticsearch-user-sau-main-dev-node-01
[2026-01-18 23:24:30 UTC] USER=www-data EUID=0 PID=3988624 ACTION=passthru ARGS=ip addr add 10.100.1.152/32 dev eth0 label eth0:152
[2026-01-18 23:24:30 UTC] USER=www-data EUID=0 PID=3988633 ACTION=fsop ARGS=sed -i /[[:space:]]search-user-sau-main-dev-elasticsearch-node-01.fastorder.com\([[:space:]]\|$\)/d /etc/hosts
[INFO] → Also added short domain: search-user-sau-main-dev.fastorder.com
[INFO] ✔ Created configuration for user-sau-main-dev/node-01 (roles=single-node)
[2026-01-18 23:24:30 UTC] USER=www-data EUID=0 PID=3988663 ACTION=fsop ARGS=sed -i /[[:space:]]search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com\([[:space:]]\|$\)/d /etc/hosts
[INFO] ✔ Registered master domain search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com -> 10.100.1.152 (points to node-01)
[INFO] Cleaning up legacy non-templated elasticsearch-*.service units (if any)...
[INFO] No legacy units found.
[INFO] Base template exists: elasticsearch@.service
[ OK ] Created unit: elasticsearch@user-sau-main-dev-node-01.service
[2026-01-18 23:24:30 UTC] USER=www-data EUID=0 PID=3988714 ACTION=passthru ARGS=systemctl daemon-reload
[ OK ] Environment configurations (master + nodes with TLS) created successfully!
[INFO] Environment: user-sau-main-dev
[INFO] Nodes: 1
[INFO] HTTP Port: 9200
[INFO] Transport Port: 9300
[INFO] Heap Size: 1024m per node
[INFO] Master: search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com (10.100.1.89)
[INFO] node-01: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.152)
[INFO] Systemd units prepared (not started). Start sequence runs in Step 7.
Executing: steps/04-start-clusters.sh
=== Step 7: Starting Elasticsearch clusters (with waits) ===
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Starting Elasticsearch cluster for environment: user-sau-main-dev (1 nodes)
[INFO] === Ensuring VM IP services are started ===
[1;33m[WARNING][0m VM IP service vm-ip-10-100-1-152.service not found - IP might not persist
[INFO] Manually configuring IP: 10.100.1.152
[2026-01-18 23:24:33 UTC] USER=www-data EUID=0 PID=3988815 ACTION=configure-network-interface ARGS=lo:search01 10.100.1.152
[INFO] Cleaning up any existing Elasticsearch processes and lock files...
[2026-01-18 23:24:33 UTC] USER=www-data EUID=0 PID=3988849 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@user-sau-main-dev-node-01.service
[INFO] Stopping Elasticsearch services for environment: user-sau-main-dev ...
[INFO] No active Elasticsearch services found for environment: user-sau-main-dev
[INFO] Removing lock files from: /data/elasticsearch/user-sau-main-dev/node-01
[2026-01-18 23:24:33 UTC] USER=www-data EUID=0 PID=3988869 ACTION=fsop ARGS=find /data/elasticsearch/user-sau-main-dev/node-01 -name *.lock -delete
[2026-01-18 23:24:33 UTC] USER=www-data EUID=0 PID=3988878 ACTION=fsop ARGS=find /data/elasticsearch/user-sau-main-dev/node-01 -name node.lock -delete
[2026-01-18 23:24:33 UTC] USER=www-data EUID=0 PID=3988887 ACTION=fsop ARGS=find /data/elasticsearch/user-sau-main-dev/node-01 -name _state -type d -exec rm -rf {} +
[2026-01-18 23:24:33 UTC] USER=www-data EUID=0 PID=3988897 ACTION=fsop ARGS=find /tmp -name *elasticsearch*user-sau-main-dev-node-01* -delete
[ OK ] Cleanup completed for environment: user-sau-main-dev
[INFO] Checking for port conflicts before starting Elasticsearch...
[INFO] Checking for port conflicts on 10.100.1.89:9200 and 10.100.1.89:9300...
[ OK ] ✓ Ports 9200 and 9300 are available on 10.100.1.89
[INFO] Ensuring correct ownership of Elasticsearch directories...
[2026-01-18 23:24:35 UTC] USER=www-data EUID=0 PID=3988965 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch
[2026-01-18 23:24:36 UTC] USER=www-data EUID=0 PID=3988982 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /data/elasticsearch
[2026-01-18 23:24:36 UTC] USER=www-data EUID=0 PID=3988991 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /var/log/elasticsearch
[ OK ] Directory ownership fixed
[INFO] === Starting Elasticsearch Nodes ===
[INFO] Starting 1 node(s) for cluster
▶ Starting elasticsearch@user-sau-main-dev-node-01.service (search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200)
[2026-01-18 23:24:36 UTC] USER=www-data EUID=0 PID=3989006 ACTION=passthru ARGS=systemctl is-enabled --quiet elasticsearch@user-sau-main-dev-node-01.service
[2026-01-18 23:24:36 UTC] USER=www-data EUID=0 PID=3989062 ACTION=passthru ARGS=systemctl start elasticsearch@user-sau-main-dev-node-01.service
⏳ Waiting for TCP search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200 to be accessible (timeout 360s)...
✅ Port 9200 is accessible on search-user-sau-main-dev-elasticsearch-node-01.fastorder.com.
⏳ Waiting for ES HTTP readiness on http://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200 (timeout 300s)...
[ OK ] ES HTTP ready on search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[ OK ] elasticsearch@user-sau-main-dev-node-01.service is up and answering HTTP on search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Node 1 started successfully
[INFO] Cluster with 1 node(s) started successfully
⏳ Waiting for the cluster to elect master and settle...
⏳ Waiting for cluster health=green via search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200 (timeout 300s)...
[ OK ] Cluster is GREEN (nodes="number_of_nodes") on search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
[ OK ] Cluster user-sau-main-dev is healthy and green!
[INFO] === Final Status Check ===
[2026-01-18 23:25:17 UTC] USER=www-data EUID=0 PID=3990302 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@user-sau-main-dev-node-01.service
[ OK ] elasticsearch@user-sau-main-dev-node-01.service is ACTIVE (search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200)
└── HTTP responding on search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200 ✓
[ OK ] All 1 node(s) in environment 'user-sau-main-dev' are running successfully!
[INFO] Node endpoints:
- http://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[ OK ] Elasticsearch cluster started successfully!
[INFO] Environment: user-sau-main-dev
[INFO] Nodes: 1
[INFO] Cluster endpoints:
- http://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] === Quick Cluster Information ===
Cluster Name: fastorder-user-sau-main-dev
Node Name: user-sau-main-dev-node-01
Version: 8.19.10
Architecture: 1 node(s), each on default port 9200
Cluster with 1 node(s) started successfully (each on port 9200)
Executing: steps/05-verify-setup.sh
=== Step 8: Verifying setup (with retries) ===
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Verifying environment: user-sau-main-dev (1 nodes, Single-node)
Main HTTP endpoint: http://search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
Testing network connectivity to search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200...
✓ Domain connection available
Testing HTTP response...
[ OK ] ✓ user-sau-main-dev is responding on search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
[INFO] === Cluster Health ===
{
"cluster_name" : "fastorder-user-sau-main-dev",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"unassigned_primary_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
[ OK ] Cluster status: GREEN ("number_of_nodes" nodes)
[INFO] === Cluster Nodes ===
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.100.1.152 54 99 85 5.53 4.65 5.03 dhims * user-sau-main-dev-node-01
[INFO] === Single-Node Service Verification ===
Testing coordinator service (search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200)...
✓ Coordinator HTTP responding on search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
Name: user-sau-main-dev-node-01, Version: 8.19.10
[INFO] === Cluster State Summary ===
Using jq for formatted output:
jq parsing failed
[ OK ] === Verification Summary ===
[INFO] Environment: user-sau-main-dev
[INFO] Nodes configured: 1
[INFO] Main endpoint: http://search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
[INFO] Service endpoint: http://search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
[INFO] === Final Connectivity Test ===
✓ Coordinator: search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
[ OK ] Single-node cluster is responding successfully!
[ OK ] Elasticsearch cluster 'user-sau-main-dev' verification completed successfully!
Executing: steps/06-confirm-working.sh
=== Step 9: Comprehensive Cluster Verification (gated) ===
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
========================================
🔍 Verifying Environment: user-sau-main-dev (1 nodes)
========================================
Domain: search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com
Environment: user-sau-main-dev
Nodes: 1
[INFO] Testing network connectivity...
Setup type: Single-node
Testing endpoint: search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
[ OK ] ✓ Using domain: search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com
📡 Coordinator Service (elasticsearch@user-sau-main-dev-node-01.service)
Endpoint: search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
--------------------------------
[2026-01-18 23:25:18 UTC] USER=www-data EUID=0 PID=3990472 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@user-sau-main-dev-node-01.service
✅ Service: ACTIVE
⏳ Waiting for TCP search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200 to be accessible (timeout 5s)...
✅ Port 9200 is accessible on search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com.
✅ Port: LISTENING on search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
✅ HTTP: RESPONDING on search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200
Node name: user-sau-main-dev-node-01
========================================
🏥 Cluster Health Check
========================================
Cluster Name: fastorder-user-sau-main-dev
Nodes Count: "number_of_nodes"
Status: green
[ OK ] ✅ Cluster status: GREEN (healthy)
Full cluster health:
{
"cluster_name" : "fastorder-user-sau-main-dev",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"unassigned_primary_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
========================================
📊 Final Verification Results
========================================
[ OK ] ✅ Comprehensive verification PASSED!
[ OK ] Environment 'user-sau-main-dev' with 1 nodes is fully operational
📋 QUICK DIAGNOSTIC COMMANDS:
----------------------------------------
# Test cluster endpoints:
curl http://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
# Check cluster health:
curl http://search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200/_cluster/health?pretty
# Check nodes info:
curl http://search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com:9200/_cat/nodes?v
# Check all Elasticsearch ports:
sudo ss -tlnp | grep java
# Check systemd service status:
sudo /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status elasticsearch@user-sau-main-dev-node-01.service
# View recent logs:
sudo journalctl -u elasticsearch@user-sau-main-dev-node-01.service -f
[INFO] Environment: user-sau-main-dev
[INFO] Nodes: 1
[INFO] Port: 9200 (default Elasticsearch port)
[INFO] Coordinator endpoint: http://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
=== Elasticsearch HTTP Setup completed successfully! ===
Environment: (1 nodes)
Port: 9200 (default Elasticsearch port)
✅ Coordinator endpoint: http://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
Quick test commands:
curl http://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
curl http://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty
[0;32m[1m✓ Step 1 completed successfully![0m
[0;36m[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;32m[1mStep 2: Executing Make Https[0m
[0;35mFolder: 02-make-https[0m
[0;36m[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
=== Elasticsearch HTTPS Setup ===
Configure HTTPS/SSL for Elasticsearch cluster
[INFO] Using web-provided environment: user-sau-main-dev
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment:
Nodes: 1
Node: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.152)
Port: 9200 (default port)
Executing: steps/01-generate-ssl-certificates.sh
==================================================================
STEP 1: Generate SSL certificates for Elasticsearch transport
==================================================================
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Domain: user-sau-main-dev.fastorder.com
Environment: user-sau-main-dev
Nodes: 1
Per-node VM IPs and domains:
Node 1: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.152)
Port: 9200 (default port for all nodes)
=== Generating SSL certificates for ES transport ===
[INFO] Generating certificates for environment: user-sau-main-dev (1 nodes)
[INFO] Configuring certificates for 1 node(s)
[INFO] Certificate storage: /etc/fastorder/elasticsearch/certs/user-sau-main-dev
[2026-01-18 23:25:19 UTC] USER=www-data EUID=0 PID=3990582 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/temp-3990539
[2026-01-18 23:25:19 UTC] USER=www-data EUID=0 PID=3990591 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/temp-3990539
[2026-01-18 23:25:19 UTC] USER=www-data EUID=0 PID=3990600 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/temp-3990539
[INFO] Creating certificate instances configuration...
Adding node: user-sau-main-dev-node-01 (search-user-sau-main-dev-elasticsearch-node-01.fastorder.com, 10.100.1.152)
[INFO] Certificate instances configuration:
instances:
- name: user-sau-main-dev-node-01
dns: [ "user-sau-main-dev-node-01", "localhost", "search-user-sau-main-dev-elasticsearch-node-01.fastorder.com" ]
ip: [ "10.100.1.152", "127.0.0.1" ]
[INFO] Creating Certificate Authority for user-sau-main-dev...
[2026-01-18 23:25:20 UTC] USER=www-data EUID=0 PID=3990644 ACTION=fsop ARGS=mkdir -p /etc/fastorder/elasticsearch/certs/user-sau-main-dev/node-01/certs
[2026-01-18 23:25:20 UTC] USER=www-data EUID=0 PID=3990653 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/fastorder/elasticsearch/certs/user-sau-main-dev
[2026-01-18 23:25:20 UTC] USER=www-data EUID=0 PID=3990672 ACTION=fsop ARGS=rm -f /etc/fastorder/elasticsearch/certs/user-sau-main-dev/node-01/certs/user-sau-main-dev-ca.zip
yes: standard output: Broken pipe
[ OK ] ✓ CA certificate created
[INFO] Creating node certificates for user-sau-main-dev...
yes: standard output: Broken pipe
[ OK ] ✓ Node certificates created
[INFO] Distributing certificates...
Configuring certificates for node 1 (user-sau-main-dev-node-01)...
[2026-01-18 23:25:26 UTC] USER=www-data EUID=0 PID=3990890 ACTION=fsop ARGS=chmod 644 /etc/fastorder/elasticsearch/certs/user-sau-main-dev/node-01/user-sau-main-dev-node-01.crt
[2026-01-18 23:25:26 UTC] USER=www-data EUID=0 PID=3990901 ACTION=fsop ARGS=chmod 600 /etc/fastorder/elasticsearch/certs/user-sau-main-dev/node-01/user-sau-main-dev-node-01.key
[ OK ] ✓ Certificates copied for user-sau-main-dev-node-01
[2026-01-18 23:25:26 UTC] USER=www-data EUID=0 PID=3990910 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/fastorder/elasticsearch/certs/user-sau-main-dev/node-01
[2026-01-18 23:25:26 UTC] USER=www-data EUID=0 PID=3990919 ACTION=fsop ARGS=find /etc/fastorder/elasticsearch/certs/user-sau-main-dev/node-01/certs -type f -name *.key -exec chmod 600 {} ;
[2026-01-18 23:25:26 UTC] USER=www-data EUID=0 PID=3990930 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/fastorder/elasticsearch/certs/user-sau-main-dev/node-01/certs
[2026-01-18 23:25:26 UTC] USER=www-data EUID=0 PID=3990939 ACTION=fsop ARGS=rm -rf /etc/elasticsearch/temp-3990539
[ OK ] ✓ Certificates ready for environment: user-sau-main-dev
[ OK ] ✓ SSL certificate generation completed successfully!
[INFO] Environment: user-sau-main-dev
[INFO] Nodes configured: 1
[INFO] Per-node VM IPs and domains (each with default port 9200):
Node 1: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.152)
[INFO] Certificate directory: /etc/fastorder/elasticsearch/certs/user-sau-main-dev/node-01/certs
[INFO] === Certificate Summary ===
CA Certificate: /etc/fastorder/elasticsearch/certs/user-sau-main-dev/node-01/certs/ca/ca.crt
Node Certificates:
- user-sau-main-dev-node-01: /etc/fastorder/elasticsearch/certs/user-sau-main-dev/node-01/
[INFO] === Verification Commands ===
# Verify CA certificate:
openssl x509 -in /etc/fastorder/elasticsearch/certs/user-sau-main-dev/node-01/certs/ca/ca.crt -text -noout
# Verify node certificates:
openssl x509 -in /etc/fastorder/elasticsearch/certs/user-sau-main-dev/node-01/user-sau-main-dev-node-01.crt -text -noout
[INFO] Next: Configure transport SSL in Elasticsearch configuration files
Executing: steps/02-enable-security-transport.sh
==================================================================
STEP 2: Enable security with transport SSL
==================================================================
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Nodes: 1
Node: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.152)
[INFO] === Single-Node Setup ===
[INFO] Enabling security (xpack.security.enabled: true)
[2026-01-18 23:25:26 UTC] USER=www-data EUID=0 PID=3990976 ACTION=fsop ARGS=sed -i /^xpack.security.enabled:/d /etc/elasticsearch/user-sau-main-dev/node-01/elasticsearch.yml
[INFO] Disabling transport SSL (not needed for single-node)
[2026-01-18 23:25:26 UTC] USER=www-data EUID=0 PID=3990995 ACTION=fsop ARGS=sed -i /^xpack.security.transport.ssl.enabled:/d /etc/elasticsearch/user-sau-main-dev/node-01/elasticsearch.yml
[ OK ] ==================================================================
[ OK ] Security and Transport SSL Configuration Complete
[ OK ] ==================================================================
[INFO] Environment: user-sau-main-dev
[INFO] Nodes: 1
[INFO] Security enabled: true
[INFO] Transport SSL enabled: false (not required for single-node)
[INFO] === Next Step ===
Restart services to apply security configuration (step 04)
Executing: steps/03-http-ssl.sh
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
==================================================================
Direct HTTPS Configuration (native TLS, PEM)
==================================================================
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Nodes: 1
Node: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.152)
Port: 9200 (default port)
Domain: user-sau-main-dev.fastorder.com
[INFO] === Single-Node Direct HTTPS Setup ===
🔐 Generated PKCS12 password: sxFWyRml... (32 chars)
[INFO] Using first node configuration: /etc/elasticsearch/user-sau-main-dev/node-01
[2026-01-18 23:25:27 UTC] USER=www-data EUID=0 PID=3991072 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-sau-main-dev/node-01
[2026-01-18 23:25:27 UTC] USER=www-data EUID=0 PID=3991083 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-sau-main-dev/node-01
[INFO] Checking prerequisites...
[ OK ] ✓ Prerequisites verified
[INFO] Setting up temporary directories...
[2026-01-18 23:25:27 UTC] USER=www-data EUID=0 PID=3991102 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out /etc/elasticsearch/user-sau-main-dev/node-01/http-certs
[2026-01-18 23:25:27 UTC] USER=www-data EUID=0 PID=3991111 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out /etc/elasticsearch/user-sau-main-dev/node-01/http-certs
[2026-01-18 23:25:27 UTC] USER=www-data EUID=0 PID=3991120 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-sau-main-dev
[2026-01-18 23:25:27 UTC] USER=www-data EUID=0 PID=3991129 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-sau-main-dev/node-01
[ OK ] ✓ Directories created
[INFO] Building certificate instances configuration...
[INFO] Certificate instances configuration:
instances:
- name: "user-sau-main-dev-http"
dns: [ "localhost", "web-03", "search-user-sau-main-dev-elasticsearch-node-01.fastorder.com", "user-sau-main-dev-node-01.fastorder.com", "search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com", "search-user-sau-main-dev.fastorder.com", "user-sau-main-dev-node-01.local" ]
ip: [ "10.100.1.152", "127.0.0.1", "::1" ]
[ OK ] ✓ Instances configuration created
[INFO] Generating HTTP Certificate Authority...
[2026-01-18 23:25:27 UTC] USER=www-data EUID=0 PID=3991174 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-sau-main-dev/node-01/http-certs
[2026-01-18 23:25:27 UTC] USER=www-data EUID=0 PID=3991183 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-sau-main-dev/node-01/http-certs
[2026-01-18 23:25:31 UTC] USER=www-data EUID=0 PID=3991299 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/http-ca.zip
Archive: /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/http-ca.zip
inflating: /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/ca/ca.crt
inflating: /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/ca/ca.key
[ OK ] ✓ HTTP CA generated successfully
[INFO] Generating per-node HTTP certificates...
[2026-01-18 23:25:31 UTC] USER=www-data EUID=0 PID=3991313 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out
[2026-01-18 23:25:31 UTC] USER=www-data EUID=0 PID=3991322 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out
[2026-01-18 23:25:34 UTC] USER=www-data EUID=0 PID=3991405 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out/http-certs.zip
[2026-01-18 23:25:34 UTC] USER=www-data EUID=0 PID=3991414 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out/http
Archive: /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out/http-certs.zip
inflating: /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out/http/user-sau-main-dev-http/user-sau-main-dev-http.crt
inflating: /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out/http/user-sau-main-dev-http/user-sau-main-dev-http.key
[ OK ] ✓ HTTP certificates generated successfully
[INFO] Installing certificates and configuring services...
[INFO] Configuring main service for single-node HTTPS...
[2026-01-18 23:25:34 UTC] USER=www-data EUID=0 PID=3991426 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-sau-main-dev/node-01/certs
[2026-01-18 23:25:34 UTC] USER=www-data EUID=0 PID=3991444 ACTION=fsop ARGS=cp /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out/http/user-sau-main-dev-http/user-sau-main-dev-http.crt /etc/elasticsearch/user-sau-main-dev/node-01/certs/http.crt
[2026-01-18 23:25:34 UTC] USER=www-data EUID=0 PID=3991453 ACTION=fsop ARGS=cp /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out/http/user-sau-main-dev-http/user-sau-main-dev-http.key /etc/elasticsearch/user-sau-main-dev/node-01/certs/http.key
[2026-01-18 23:25:34 UTC] USER=www-data EUID=0 PID=3991462 ACTION=fsop ARGS=cp /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/ca/ca.crt /etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt
[2026-01-18 23:25:34 UTC] USER=www-data EUID=0 PID=3991471 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/user-sau-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-01-18 23:25:34 UTC] USER=www-data EUID=0 PID=3991480 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-sau-main-dev/node-01/certs
[2026-01-18 23:25:34 UTC] USER=www-data EUID=0 PID=3991489 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/user-sau-main-dev/node-01/certs/http.key
[2026-01-18 23:25:34 UTC] USER=www-data EUID=0 PID=3991498 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/user-sau-main-dev/node-01/certs/http.crt /etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt /etc/elasticsearch/user-sau-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-01-18 23:25:34 UTC] USER=www-data EUID=0 PID=3991507 ACTION=fsop ARGS=sed -i -E -e /^\s*xpack\.security\.http\.ssl(\..*)?\s*:/d /etc/elasticsearch/user-sau-main-dev/node-01/elasticsearch.yml
[2026-01-18 23:25:34 UTC] USER=www-data EUID=0 PID=3991516 ACTION=fsop ARGS=sed -i /^# --- BEGIN direct HTTPS (managed, PEM) ---$/,/^# --- END direct HTTPS (managed, PEM) ---$/d /etc/elasticsearch/user-sau-main-dev/node-01/elasticsearch.yml
[2026-01-18 23:25:34 UTC] USER=www-data EUID=0 PID=3991526 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/user-sau-main-dev/node-01/certs/fastorder_ra_root.crt
YAML: /etc/elasticsearch/user-sau-main-dev/node-01/elasticsearch.yml
[ OK ] ✓ Main service configured with HTTPS
[2026-01-18 23:25:34 UTC] USER=www-data EUID=0 PID=3991562 ACTION=fsop ARGS=rm -rf /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients
[2026-01-18 23:25:34 UTC] USER=www-data EUID=0 PID=3991571 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients
[2026-01-18 23:25:34 UTC] USER=www-data EUID=0 PID=3991580 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients
Archive: /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client.zip
creating: /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/
inflating: /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.crt
inflating: /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.key
[INFO] Creating P12 keystore for es-client...
[2026-01-18 23:25:36 UTC] USER=www-data EUID=0 PID=3991649 ACTION=fsop ARGS=mv /tmp/es-client-3991032.p12 /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.p12
[ OK ] ✓ Created P12 keystore: /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-01-18 23:25:36 UTC] USER=www-data EUID=0 PID=3991658 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients
[2026-01-18 23:25:36 UTC] USER=www-data EUID=0 PID=3991668 ACTION=fsop ARGS=chmod 640 /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.key
[2026-01-18 23:25:36 UTC] USER=www-data EUID=0 PID=3991677 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-01-18 23:25:36 UTC] USER=www-data EUID=0 PID=3991686 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.crt /etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt
[INFO] Saving keystore passwords to secrets vault...
🔑 Configuring AWS credentials for secrets vault...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[0;34m[INFO][0m 🔐 Vaulting search passwords to remote backend...
[0;32m✅ Passwords vaulted to remote backend[0m
[0;32m✓ Keystore passwords saved to secrets vault: search/user-sau-main-dev/keystore-passwords[0m
[0;34m[INFO][0m === Installing CA Certificate for Users ===
[0;34m[INFO][0m HOME not set, skipping user CA installation
[0;32m✓ Direct HTTPS configuration completed for environment: user-sau-main-dev[0m
[0;34m[INFO][0m All services now serve HTTPS using PEM certificates
[0;34m[INFO][0m Network binding: 10.100.1.152
[0;34m[INFO][0m HTTPS endpoint: https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[0;34m[INFO][0m === Certificate Summary ===
CA Certificate: /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/ca/ca.crt
Certificate Directory: /etc/elasticsearch/user-sau-main-dev/node-01/http-certs
Main service certificates: /etc/elasticsearch/user-sau-main-dev/node-01/certs/
[0;34m[INFO][0m === Next Steps ===
1. Restart Elasticsearch services to apply HTTPS configuration
2. Test HTTPS connectivity: curl https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
3. Verify certificates: openssl s_client -connect search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[0;34m[INFO][0m === SSL Certificate Information ===
• CA certificate installed for user - curl will work without -k flag
• Each certificate includes node-specific domain, VM IP, and localhost in Subject Alternative Names
• Certificates are valid for 3 years from generation date
[1;33m[WARNING][0m Important: You'll need to restart Elasticsearch services for HTTPS to take effect
Executing: steps/04-restart-systemd-services.sh
==================================================================
STEP 4 (STRICT): Restart systemd services and verify secure health
==================================================================
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
Environment: user-sau-main-dev
Nodes: 1
Per-node endpoints (all use default port 9200):
Node 1: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.152)
[INFO] Building service list for environment: user-sau-main-dev (1 nodes)
- elasticsearch@user-sau-main-dev-node-01.service (port 9200)
[INFO] Will restart 1 service(s) for environment: user-sau-main-dev
[2026-01-18 23:25:40 UTC] USER=www-data EUID=0 PID=3991835 ACTION=passthru ARGS=systemctl daemon-reload
[INFO] === Ensuring VM IPs are configured correctly ===
[INFO] ✓ 10.100.1.152 already configured on eth0 for node-01
[INFO] === Ensuring transport SSL certificates for all nodes ===
[INFO] ✓ Transport certificates already exist for node-01
[INFO] === Restarting Services ===
↻ Restarting elasticsearch@user-sau-main-dev-node-01.service ...
[2026-01-18 23:25:40 UTC] USER=www-data EUID=0 PID=3991893 ACTION=passthru ARGS=systemctl restart elasticsearch@user-sau-main-dev-node-01.service
[2026-01-18 23:25:45 UTC] USER=www-data EUID=0 PID=3992089 ACTION=passthru ARGS=systemctl is-active --quiet elasticsearch@user-sau-main-dev-node-01.service
[ OK ] elasticsearch@user-sau-main-dev-node-01.service is active
[INFO] Waiting 10s for Elasticsearch to start listening on ports...
[INFO] === Waiting for STRICT Secure Cluster Health ===
[INFO] Waiting for port 9200 on 10.100.1.152 (timeout 120s)...
[INFO] Waiting for cluster to form and be ready for write operations...
✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
[INFO] Cluster stable and ready for operations (20 consecutive healthy responses over 40s)
✓ Retrieved password from AWS Secrets Manager
[INFO] Testing cluster at: https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Using SSL CA certificate: /etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt
[INFO] Using client cert/key for mTLS
[INFO] Using client cert/key for mTLS: /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.key
[INFO] ⏳ waiting for secure cluster health (require 200) at https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200 (timeout 30s)...
[1;33m[WARNING][0m 401 (auth required) — creds OK at TLS, waiting for health 200…
[1;33m[WARNING][0m 401 (auth required) — creds OK at TLS, waiting for health 200…
[1;33m[WARNING][0m 401 (auth required) — creds OK at TLS, waiting for health 200…
[1;33m[WARNING][0m 401 (auth required) — creds OK at TLS, waiting for health 200…
[1;33m[WARNING][0m 401 (auth required) — creds OK at TLS, waiting for health 200…
[1;33m[WARNING][0m 401 (auth required) — creds OK at TLS, waiting for health 200…
[1;33m[WARNING][0m 401 (auth required) — creds OK at TLS, waiting for health 200…
[1;33m[WARNING][0m 401 (auth required) — creds OK at TLS, waiting for health 200…
[1;33m[WARNING][0m 401 (auth required) — creds OK at TLS, waiting for health 200…
[1;33m[WARNING][0m 401 (auth required) — creds OK at TLS, waiting for health 200…
[1;33m[WARNING][0m 401 (auth required) — creds OK at TLS, waiting for health 200…
[1;33m[WARNING][0m 401 (auth required) — creds OK at TLS, waiting for health 200…
[1;33m[WARNING][0m 401 (auth required) — creds OK at TLS, waiting for health 200…
[1;33m[WARNING][0m 401 (auth required) — creds OK at TLS, waiting for health 200…
[1;33m[WARNING][0m Cluster did not become healthy (secure 200) within 30s
[1;33m[WARNING][0m Initial authentication failed - password may not be set in Elasticsearch yet
[1;33m[WARNING][0m Running password setup to set/reset Elasticsearch password...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ Elasticsearch Password Management via AWS Secrets MGR ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment: user-sau-main-dev[0m
[0;34mUser: elastic[0m
[0;34mIdentifier: node-01[0m
[0;34mAWS Secret: fastorder/search/user/sau/main/dev/elasticsearch/node-01[0m
Using configuration path: /etc/elasticsearch/user-sau-main-dev/node-01 (IDENTIFIER: node-01)
Node domain: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com
HTTP port: 9200 (default Elasticsearch port)
[INFO] xpack.security.enabled already true → no restart.
[INFO] No restart needed.
[2026-01-18 23:27:31 UTC] USER=www-data EUID=0 PID=3994856 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-sau-main-dev/node-01
[2026-01-18 23:27:31 UTC] USER=www-data EUID=0 PID=3994885 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/user-sau-main-dev/node-01/users /etc/elasticsearch/user-sau-main-dev/node-01/users_roles
[2026-01-18 23:27:31 UTC] USER=www-data EUID=0 PID=3994894 ACTION=fsop ARGS=chmod 660 /etc/elasticsearch/user-sau-main-dev/node-01/users /etc/elasticsearch/user-sau-main-dev/node-01/users_roles
[0;32m✓ users/users_roles present and writable[0m
[2026-01-18 23:27:31 UTC] USER=www-data EUID=0 PID=3994903 ACTION=fsop ARGS=chown elasticsearch:elasticsearch /etc/elasticsearch/user-sau-main-dev/node-01/elasticsearch.keystore
[2026-01-18 23:27:31 UTC] USER=www-data EUID=0 PID=3994912 ACTION=fsop ARGS=chmod 660 /etc/elasticsearch/user-sau-main-dev/node-01/elasticsearch.keystore
[0;32m✓ Keystore exists: /etc/elasticsearch/user-sau-main-dev/node-01/elasticsearch.keystore[0m
[0;34mHTTPS is enabled in configuration[0m
[0;32m✓ Found HTTP CA certificate: /etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt[0m
[0;32m✓ Using client certificates for mTLS[0m
[0;34mWaiting for Elasticsearch to be reachable at https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200...[0m
[0;32m✓ Elasticsearch is reachable (HTTP 401)[0m
[0;34mES_PATH_CONF: /etc/elasticsearch/user-sau-main-dev/node-01[0m
[0;34mHTTP URL: https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200[0m
[0;34mRunning HTTP reset (online, --batch)…[0m
[0;34mNote: Using HTTPS - tools will read SSL config from elasticsearch.yml[0m
Command output:
Password for the [elastic] user successfully reset.
New value: silIukM1=kC+UVuB0SHB
Exit status: 0
[0;32m✓ HTTP reset succeeded for elastic[0m
[0;34mStoring credentials in AWS Secrets Manager: fastorder/search/user/sau/main/dev/elasticsearch/node-01[0m
ℹ️ Setting Elasticsearch credentials in vault: fastorder/search/user/sau/main/dev/elasticsearch/node-01
ℹ️ Setting secret in AWS Secrets Manager: fastorder/search/user/sau/main/dev/elasticsearch/node-01
✅ Secret updated: fastorder/search/user/sau/main/dev/elasticsearch/node-01
✅ Elasticsearch credentials set in vault: fastorder/search/user/sau/main/dev/elasticsearch/node-01
[0;32m✓ Password stored in AWS Secrets Manager: fastorder/search/user/sau/main/dev/elasticsearch/node-01[0m
[0;32m✓ Cache cleared for: fastorder/search/user/sau/main/dev/elasticsearch/node-01[0m
[0;32m✓ Done. Password stored in AWS Secrets Manager: fastorder/search/user/sau/main/dev/elasticsearch/node-01[0m
Usage Examples:
# Retrieve password using AWS CLI
aws secretsmanager get-secret-value --secret-id fastorder/search/user/sau/main/dev/elasticsearch/node-01 --region ${AWS_REGION:-me-central-1}
# Using fastctl
fastctl secrets get fastorder/search/user/sau/main/dev/elasticsearch/node-01
# Test connection
curl -u elastic:$(fastctl secrets get fastorder/search/user/sau/main/dev/elasticsearch/node-01 --field password) https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health
✓ Retrieved password from AWS Secrets Manager
[INFO] Retrying authentication with new password...
[INFO] Using client cert/key for mTLS: /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.key
[INFO] ⏳ waiting for secure cluster health (require 200) at https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200 (timeout 300s)...
[ OK ] Cluster health OK: green
==================================================================
[ OK ] All services restarted successfully!
[ OK ] Cluster is healthy, HTTPS-secure, and responding with 200
[INFO] Environment: user-sau-main-dev
[INFO] Services: 1
[INFO] Endpoint: https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] === Manual verification (copy/paste) ===
curl -u 'elastic:silIukM1=kC+UVuB0SHB' \
--cacert '/etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt' \
--cert '/etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.crt' \
--key '/etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.key' \
'https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty'
[INFO] === Quick checks ===
curl -u 'elastic:silIukM1=kC+UVuB0SHB' --cacert '/etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt' --cert '/etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.crt' --key '/etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.key' https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cat/nodes?v
curl -u 'elastic:silIukM1=kC+UVuB0SHB' --cacert '/etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt' --cert '/etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.crt' --key '/etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.key' https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/
Executing: steps/05-test-elastic.sh
==================================================================
STEP 5: Test Elasticsearch Cluster
==================================================================
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
Environment: user-sau-main-dev
Nodes: 1
Node: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.152)
Port: 9200 (default port)
[INFO] Using centralized test suite: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/02-make-https/lib/elasticsearch-test-suite.sh
[INFO] Executing centralized test suite with args: -v -t all --env user-sau-main-dev -u elastic
[0;34m[INFO][0m Using environment from web interface: user-sau-main-dev
[0;32m[2026-01-18 23:27:39][0m Using web-provided environment: user-sau-main-dev
[0;32m[2026-01-18 23:27:39][0m Service: user, Zone: sau, Branch: main, Env: dev
[0;32m✓[0m Environment initialized successfully (mode: general)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[0;34mℹ[0m Using environment from web interface: user-sau-main-dev
[0;32m[2026-01-18 23:27:39][0m Using web-provided environment: user-sau-main-dev
[0;32m[2026-01-18 23:27:39][0m Service: user, Zone: sau, Branch: main, Env: dev
[0;32m✓[0m Environment initialized successfully (mode: general)
[0;34mℹ[0m Project root: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch
[0;34mℹ[0m Environment: user-sau-main-dev
[0;34mℹ[0m Nodes count: 1
[0;34mℹ[0m Endpoint: https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[0;34mℹ[0m Using CA: /etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt
[0;34mℹ[0m Using mTLS: /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.key
[0;34m╔════════════════════════════════════════════╗[0m
[0;34m║ Elasticsearch Centralized Test Suite ║[0m
[0;34m╚════════════════════════════════════════════╝[0m
[0;34m=== Authentication Test ===[0m
[0;32m✓[0m Loaded credentials for user elastic from AWS Secrets Manager
[0;34mCurl (local):[0m curl --cacert /etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt --cert /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.crt --key /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.key -u 'elastic:********' 'https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty'
[0;32m✓[0m Local authentication successful (HTTP 200).
{
"cluster_name" : "fastorder-user-sau-main-dev",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 3,
"active_shards" : 3,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"unassigned_primary_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
Executing: steps/06-final-testing.sh
==================================================================
STEP 6: Final Testing and Verification
==================================================================
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
Environment: user-sau-main-dev
Nodes: 1
Node: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.152)
Port: 9200 (default port)
[INFO] Using centralized test suite: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/02-make-https/lib/elasticsearch-test-suite.sh
[0;34m[INFO][0m Using environment from web interface: user-sau-main-dev
[0;32m[2026-01-18 23:27:40][0m Using web-provided environment: user-sau-main-dev
[0;32m[2026-01-18 23:27:40][0m Service: user, Zone: sau, Branch: main, Env: dev
[0;32m✓[0m Environment initialized successfully (mode: general)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[0;34mℹ[0m Using environment from web interface: user-sau-main-dev
[0;32m[2026-01-18 23:27:40][0m Using web-provided environment: user-sau-main-dev
[0;32m[2026-01-18 23:27:40][0m Service: user, Zone: sau, Branch: main, Env: dev
[0;32m✓[0m Environment initialized successfully (mode: general)
[0;34mℹ[0m Project root: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch
[0;34mℹ[0m Environment: user-sau-main-dev
[0;34mℹ[0m Nodes count: 1
[0;34mℹ[0m Endpoint: https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[0;34mℹ[0m Using CA: /etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt
[0;34mℹ[0m Using mTLS: /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.crt / /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.key
[0;34m╔════════════════════════════════════════════╗[0m
[0;34m║ Elasticsearch Centralized Test Suite ║[0m
[0;34m╚════════════════════════════════════════════╝[0m
[0;34m=== Authentication Test ===[0m
[0;32m✓[0m Loaded credentials for user elastic from AWS Secrets Manager
[0;34mCurl (local):[0m curl --cacert /etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt --cert /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.crt --key /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.key -u 'elastic:********' 'https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health?pretty'
[0;32m✓[0m Local authentication successful (HTTP 200).
{
"cluster_name" : "fastorder-user-sau-main-dev",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 3,
"active_shards" : 3,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"unassigned_primary_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
Executing: steps/07-set-passwords.sh
==================================================================
STEP 7: Setting cluster passwords (bootstrap via alias)
==================================================================
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] Using HTTPS with CA: /etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt (host: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com)
[INFO] Using centralized password setter: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/02-make-https/steps/../lib/elasticsearch-set-password.sh
[ OK ] Elastic password already valid (HTTP 200) via search-user-sau-main-dev-elasticsearch-node-01.fastorder.com; nothing to do.
Executing: steps/08-create-app-user.sh
==================================================================
STEP 8: Create Application User and Roles (cluster-scoped)
==================================================================
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
Environment: user-sau-main-dev
Nodes: 1
[INFO] Using HTTPS with CA: /etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt (host: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com)
[ OK ] Retrieved elastic password from Vault (cluster scope).
[INFO] Configuration:
[INFO] App User : app_user
[INFO] Read-only Role : app_ro
[INFO] Read-write Role : app_rw
[INFO] Index Patterns : app-*,cdc-*,user_sau_*,*_account_router
[INFO] Endpoint : https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Creating read-only role: app_ro
[ OK ] ✓ Role app_ro ensured
[INFO] Creating read-write role: app_rw
[ OK ] ✓ Role app_rw ensured
[INFO] Creating/Updating application user: app_user
[ OK ] ✓ User app_user ensured
ℹ️ Setting Elasticsearch credentials in vault: fastorder/search/user/sau/main/dev/elasticsearch/node-01/app_user
ℹ️ Setting secret in AWS Secrets Manager: fastorder/search/user/sau/main/dev/elasticsearch/node-01/app_user
✅ Secret updated: fastorder/search/user/sau/main/dev/elasticsearch/node-01/app_user
✅ Elasticsearch credentials set in vault: fastorder/search/user/sau/main/dev/elasticsearch/node-01/app_user
[ OK ] ✓ Stored app_user password under 'node-01/app_user'
ℹ️ Setting Elasticsearch credentials in vault: fastorder/search/user/sau/main/dev/elasticsearch/cluster/app_user
ℹ️ Setting secret in AWS Secrets Manager: fastorder/search/user/sau/main/dev/elasticsearch/cluster/app_user
✅ Secret updated: fastorder/search/user/sau/main/dev/elasticsearch/cluster/app_user
✅ Elasticsearch credentials set in vault: fastorder/search/user/sau/main/dev/elasticsearch/cluster/app_user
[ OK ] ✓ Stored app_user password under 'cluster/app_user'
[INFO] Testing authentication for app_user...
[ OK ] ✓ Authentication test passed for app_user
[ OK ] ✓ Application user and roles created successfully!
[INFO] User : app_user
[INFO] Roles : app_ro, app_rw
[INFO] Patterns: app-*,cdc-*,user_sau_*,*_account_router
[INFO] Endpoint: https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
Executing: steps/09-config.sh
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Nodes: 1
Node: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.152)
Port: 9200
[0;32m✓[0m Auto mode: Cloud IMDS detected → MODE=role
[0;34m[INFO][0m Mode: role
[0;34m[INFO][0m AWS Region: me-central-1
[0;34m[INFO][0m MODE=role → will purge any static S3 keys from each node keystore
[2026-01-18 23:28:53 UTC] USER=www-data EUID=0 PID=3997588 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-sau-main-dev/node-01
[0;34m[INFO][0m • node-01 keystore cleared (role-based auth)
[2026-01-18 23:28:59 UTC] USER=www-data EUID=0 PID=3997900 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:28:59 UTC] USER=www-data EUID=0 PID=3997968 ACTION=passthru ARGS=systemctl restart elasticsearch@user-sau-main-dev-node-01.service
[0;32m✓[0m ✓ restarted elasticsearch@user-sau-main-dev-node-01.service
⏳ Waiting for HTTPS readiness on https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Waiting HTTP readiness at https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/ (200/401/302)…
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
Waiting ...
[OK] Ready: 401
⏳ Waiting for cluster health (green|yellow)
[INFO] Waiting health (green|yellow) at https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health…
[OK] 401 pre-auth received; security enabled.
[0;32m✓[0m ✓ user-sau-main-dev is responding via search-user-sau-main-dev-elasticsearch-node-01.fastorder.com
[0;32m✓[0m ✓ AWS S3 configuration completed for environment: user-sau-main-dev (1 nodes)
[0;34m[INFO][0m Mode: role
[0;34m[INFO][0m Region: me-central-1
Executing: steps/0ld-03-http-ssl.sh
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
==================================================================
STEP 5: HTTP SSL Configuration (Optional)
==================================================================
Environment: user-sau-main-dev
Nodes: 1
Node: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.152)
Port: 9200 (default port)
[ OK ] 🚀 Auto mode/Default installation: Selecting Direct HTTPS configuration (option 1)
[ OK ] Configuring Direct HTTPS (Elasticsearch native SSL)...
──────────────────────────────────────────────────────────
[INFO] Environment: user-sau-main-dev (1 nodes)
[INFO] Node: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.152)
[INFO] Port: 9200 (default port)
==================================================================
Direct HTTPS Configuration (native TLS, PEM)
==================================================================
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Nodes: 1
Node: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com (10.100.1.152)
Port: 9200 (default port)
Domain: user-sau-main-dev.fastorder.com
[INFO] === Single-Node Direct HTTPS Setup ===
🔐 Generated PKCS12 password: omd12v5U... (32 chars)
[INFO] Using first node configuration: /etc/elasticsearch/user-sau-main-dev/node-01
[2026-01-18 23:29:39 UTC] USER=www-data EUID=0 PID=3998949 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-sau-main-dev/node-01
[2026-01-18 23:29:40 UTC] USER=www-data EUID=0 PID=3998958 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-sau-main-dev/node-01
[INFO] Checking prerequisites...
[ OK ] ✓ Prerequisites verified
[INFO] Setting up temporary directories...
[2026-01-18 23:29:40 UTC] USER=www-data EUID=0 PID=3998969 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out /etc/elasticsearch/user-sau-main-dev/node-01/http-certs
[2026-01-18 23:29:40 UTC] USER=www-data EUID=0 PID=3998981 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out /etc/elasticsearch/user-sau-main-dev/node-01/http-certs
[2026-01-18 23:29:40 UTC] USER=www-data EUID=0 PID=3999001 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-sau-main-dev
[2026-01-18 23:29:40 UTC] USER=www-data EUID=0 PID=3999021 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-sau-main-dev/node-01
[2026-01-18 23:29:40 UTC] USER=www-data EUID=0 PID=3999031 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-sau-main-dev/node-01/http-certs
[2026-01-18 23:29:40 UTC] USER=www-data EUID=0 PID=3999054 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out
[ OK ] ✓ Directories created
[INFO] Building certificate instances configuration...
[INFO] Certificate instances configuration:
instances:
- name: "user-sau-main-dev-http"
dns: [ "localhost", "web-03", "search-user-sau-main-dev-elasticsearch-node-01.fastorder.com", "user-sau-main-dev-node-01.fastorder.com", "search-user-sau-main-dev-elasticsearch-coordinator.fastorder.com", "search-user-sau-main-dev.fastorder.com", "user-sau-main-dev-node-01.local" ]
ip: [ "10.100.1.152", "127.0.0.1", "::1" ]
[ OK ] ✓ Instances configuration created
[INFO] Generating HTTP Certificate Authority...
[2026-01-18 23:29:40 UTC] USER=www-data EUID=0 PID=3999075 ACTION=fsop ARGS=rm -f /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/http-ca.zip /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out/http-certs.zip
[2026-01-18 23:29:40 UTC] USER=www-data EUID=0 PID=3999084 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-sau-main-dev/node-01/http-certs
[2026-01-18 23:29:40 UTC] USER=www-data EUID=0 PID=3999093 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-sau-main-dev/node-01/http-certs
[2026-01-18 23:29:43 UTC] USER=www-data EUID=0 PID=3999185 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/http-ca.zip
Archive: /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/http-ca.zip
inflating: /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/ca/ca.crt
inflating: /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/ca/ca.key
[ OK ] ✓ HTTP CA generated successfully
[INFO] Generating per-node HTTP certificates...
[2026-01-18 23:29:43 UTC] USER=www-data EUID=0 PID=3999197 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out
[2026-01-18 23:29:43 UTC] USER=www-data EUID=0 PID=3999206 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out
[2026-01-18 23:29:45 UTC] USER=www-data EUID=0 PID=3999278 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out/http-certs.zip
[2026-01-18 23:29:45 UTC] USER=www-data EUID=0 PID=3999287 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out/http
Archive: /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out/http-certs.zip
inflating: /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out/http/user-sau-main-dev-http/user-sau-main-dev-http.crt
inflating: /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out/http/user-sau-main-dev-http/user-sau-main-dev-http.key
[ OK ] ✓ HTTP certificates generated successfully
[INFO] Installing certificates and configuring services...
[INFO] Configuring main service for single-node HTTPS...
[2026-01-18 23:29:45 UTC] USER=www-data EUID=0 PID=3999299 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-sau-main-dev/node-01/certs
[2026-01-18 23:29:45 UTC] USER=www-data EUID=0 PID=3999308 ACTION=fsop ARGS=chmod 755 /etc/elasticsearch/user-sau-main-dev/node-01/certs
[2026-01-18 23:29:45 UTC] USER=www-data EUID=0 PID=3999317 ACTION=fsop ARGS=cp /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out/http/user-sau-main-dev-http/user-sau-main-dev-http.crt /etc/elasticsearch/user-sau-main-dev/node-01/certs/http.crt
[2026-01-18 23:29:45 UTC] USER=www-data EUID=0 PID=3999326 ACTION=fsop ARGS=cp /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/out/http/user-sau-main-dev-http/user-sau-main-dev-http.key /etc/elasticsearch/user-sau-main-dev/node-01/certs/http.key
[2026-01-18 23:29:45 UTC] USER=www-data EUID=0 PID=3999335 ACTION=fsop ARGS=cp /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/ca/ca.crt /etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt
[2026-01-18 23:29:45 UTC] USER=www-data EUID=0 PID=3999344 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/user-sau-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-01-18 23:29:45 UTC] USER=www-data EUID=0 PID=3999353 ACTION=fsop ARGS=chown -R elasticsearch:elasticsearch /etc/elasticsearch/user-sau-main-dev/node-01/certs
[2026-01-18 23:29:45 UTC] USER=www-data EUID=0 PID=3999362 ACTION=fsop ARGS=chmod 600 /etc/elasticsearch/user-sau-main-dev/node-01/certs/http.key
[2026-01-18 23:29:45 UTC] USER=www-data EUID=0 PID=3999371 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/user-sau-main-dev/node-01/certs/http.crt /etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt /etc/elasticsearch/user-sau-main-dev/node-01/certs/fastorder_ra_root.crt
[2026-01-18 23:29:45 UTC] USER=www-data EUID=0 PID=3999380 ACTION=fsop ARGS=sed -i -E -e /^\s*xpack\.security\.http\.ssl(\..*)?\s*:/d /etc/elasticsearch/user-sau-main-dev/node-01/elasticsearch.yml
[2026-01-18 23:29:45 UTC] USER=www-data EUID=0 PID=3999389 ACTION=fsop ARGS=sed -i /^# --- BEGIN direct HTTPS (managed, PEM) ---$/,/^# --- END direct HTTPS (managed, PEM) ---$/d /etc/elasticsearch/user-sau-main-dev/node-01/elasticsearch.yml
[2026-01-18 23:29:45 UTC] USER=www-data EUID=0 PID=3999398 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /etc/elasticsearch/user-sau-main-dev/node-01/certs/fastorder_ra_root.crt
YAML: /etc/elasticsearch/user-sau-main-dev/node-01/elasticsearch.yml
[ OK ] ✓ Main service configured with HTTPS
[2026-01-18 23:29:45 UTC] USER=www-data EUID=0 PID=3999428 ACTION=fsop ARGS=rm -rf /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients
[2026-01-18 23:29:45 UTC] USER=www-data EUID=0 PID=3999449 ACTION=fsop ARGS=mkdir -p /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients
[2026-01-18 23:29:45 UTC] USER=www-data EUID=0 PID=3999462 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients
Archive: /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client.zip
creating: /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/
inflating: /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.crt
inflating: /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.key
[INFO] Creating P12 keystore for es-client...
[2026-01-18 23:29:48 UTC] USER=www-data EUID=0 PID=3999597 ACTION=fsop ARGS=mv /tmp/es-client-3998895.p12 /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.p12
[ OK ] ✓ Created P12 keystore: /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.p12
[2026-01-18 23:29:48 UTC] USER=www-data EUID=0 PID=3999607 ACTION=fsop ARGS=chown -R elasticsearch:sslusers /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients
[2026-01-18 23:29:48 UTC] USER=www-data EUID=0 PID=3999616 ACTION=fsop ARGS=chmod 640 /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.key
[2026-01-18 23:29:48 UTC] USER=www-data EUID=0 PID=3999634 ACTION=fsop ARGS=chmod 644 /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.crt /etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt
[INFO] Saving keystore passwords to secrets vault...
🔑 Configuring AWS credentials for secrets vault...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[0;34m[INFO][0m 🔐 Vaulting search passwords to remote backend...
[0;32m✅ Passwords vaulted to remote backend[0m
[0;32m✓ Keystore passwords saved to secrets vault: search/user-sau-main-dev/keystore-passwords[0m
[0;34m[INFO][0m === Installing CA Certificate for Users ===
[0;34m[INFO][0m HOME not set, skipping user CA installation
[0;32m✓ Direct HTTPS configuration completed for environment: user-sau-main-dev[0m
[0;34m[INFO][0m All services now serve HTTPS using PEM certificates
[0;34m[INFO][0m Network binding: 10.100.1.152
[0;34m[INFO][0m HTTPS endpoint: https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[0;34m[INFO][0m === Certificate Summary ===
CA Certificate: /etc/elasticsearch/user-sau-main-dev/node-01/http-certs/ca/ca.crt
Certificate Directory: /etc/elasticsearch/user-sau-main-dev/node-01/http-certs
Main service certificates: /etc/elasticsearch/user-sau-main-dev/node-01/certs/
[0;34m[INFO][0m === Next Steps ===
1. Restart Elasticsearch services to apply HTTPS configuration
2. Test HTTPS connectivity: curl https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
3. Verify certificates: openssl s_client -connect search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[0;34m[INFO][0m === SSL Certificate Information ===
• CA certificate installed for user - curl will work without -k flag
• Each certificate includes node-specific domain, VM IP, and localhost in Subject Alternative Names
• Certificates are valid for 3 years from generation date
[1;33m[WARNING][0m Important: You'll need to restart Elasticsearch services for HTTPS to take effect
[ OK ] ✓ Direct HTTPS configuration completed successfully
[ OK ] ==================================================================
[ OK ] HTTP SSL Configuration Complete
[ OK ] ==================================================================
[INFO] Environment: user-sau-main-dev
[INFO] Nodes: 1
[INFO] Configuration applied to port: 9200 (default port for all nodes)
[INFO] === Next Steps ===
1. Verify Elasticsearch is running: systemctl status elasticsearch@user-sau-main-dev-node-01.service
2. Test cluster health: curl https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_cluster/health
3. Check SSL certificate: openssl s_client -connect search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
=== HTTPS Setup completed successfully! ===
Environment: (1 nodes)
Domain: .fastorder.com
HTTPS endpoint: https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
Node IP: 10.100.1.152
[0;32m[1m✓ Step 2 completed successfully![0m
[0;36m[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;32m[1mStep 3: Executing Create Index Llm[0m
[0;35mFolder: 03-create-index-llm[0m
[0;36m[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
==================================================================
Elasticsearch LLM/Semantic Search Setup
==================================================================
[INFO] Using web-provided environment: user-sau-main-dev
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
Environment: user-sau-main-dev
Service : user
🔍 Checking Elasticsearch availability…
✅ Elasticsearch is accessible at https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
=== Phase 1: Common steps under /data/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps ===
(no numbered steps in: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps)
=== Phase 2: Service-scoped steps for 'user' under /data/opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/user ===
📚 Detected features: contracts
── Feature: contracts
▶️ Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/user/contracts/00-create-ingest-pipeline.sh
==================================================================
STEP 0: Create Ingest Pipeline (User Contracts)
==================================================================
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] Creating ingest pipeline: user_sau_main_dev_user_contracts_pipeline
[1;32m[OK][0m Ingest pipeline created: user_sau_main_dev_user_contracts_pipeline
[INFO] Testing pipeline with sample document...
[1;32m[OK][0m Pipeline simulation completed
==================================================================
Ingest Pipeline Configuration Complete
==================================================================
Pipeline: user_sau_main_dev_user_contracts_pipeline
ES URL: https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/_ingest/pipeline/user_sau_main_dev_user_contracts_pipeline
OPERATIONS:
- Default values for status fields
- Lowercase normalization (username)
- Uppercase normalization (country_code)
- Timestamp parsing (created_at, updated_at, dates)
- Safety net removal of raw PII fields
NOT DONE (by design):
- Email/phone hashing (done at PostgreSQL level)
- PII transformation (should never reach this pipeline)
==================================================================
✅ 00-create-ingest-pipeline.sh completed
▶️ Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/user/contracts/01-create-model-and-pipeline.sh
==================================================================
STEP 1: Create Model and Ingest Pipeline (User Contracts)
==================================================================
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] ES URL: https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200
[INFO] Endpoint ID (ES): user-contracts-embedding-001
[INFO] Provider model: text-embedding-3-large
[INFO] Pipeline ID: user-contracts-embed-pipeline-001
[INFO] Checking authentication identity…
{
"username":"elastic","roles":["superuser"],"full_name":null,"email":null,"metadata":{"_reserved":true},"enabled":true,"authentication_realm":{"name":"reserved","type":"reserved"},"lookup_realm":{"name":"reserved","type":"reserved"},"authentication_type":"realm"
}
[INFO] Checking Elasticsearch license…
[INFO] License type: unknown
[WARN] Inference API requires Enterprise/Platinum license (found: unknown)
[WARN] Skipping inference endpoint and pipeline creation
[1;32m[OK][0m Setup completed (inference features skipped due to license)
✅ 01-create-model-and-pipeline.sh completed
▶️ Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/user/contracts/02-create-index.sh
==================================================================
STEP 2: Create Minimal User Contracts Index (ILM bootstrap)
==================================================================
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] [check] Verifying embedding pipeline exists: user-contracts-embed-pipeline-001
[WARN] Embedding pipeline 'user-contracts-embed-pipeline-001' does not exist (HTTP 404)
[WARN] Index will be created WITHOUT default_pipeline.
[WARN] Run 01-create-model-and-pipeline.sh first if you need ML embeddings.
[WARN] CDC data flow will work normally without embeddings.
[INFO] [cluster] Ensure disk watermarks permit allocation
[1;32m[OK][0m Cluster watermarks set/confirmed.
[INFO] [create] Create/Update ILM policy: user-user-contracts-ilm
[1;32m[OK][0m ILM policy ready.
[INFO] [create] Create/Update index template: user_sau_main_dev_user_contracts_template
[1;32m[OK][0m Index template ready.
[INFO] [check] Concrete index: user_sau_main_dev_user_contracts-000001
[INFO] [create] Create first index + attach write alias: user_sau_main_dev_user_contracts-000001
[1;32m[OK][0m Created user_sau_main_dev_user_contracts-000001 with write alias user_sau_main_dev_user_contracts.
[INFO] [verify] Wait for index to be at least YELLOW
[1;32m[OK][0m Cluster health OK for user_sau_main_dev_user_contracts-000001.
[INFO] [verify] Alias points to a concrete write index
[1;32m[OK][0m Alias verification passed.
[INFO] [explain] ILM status
{
"indices" : {
"user_sau_main_dev_user_contracts-000001" : {
"index" : "user_sau_main_dev_user_contracts-000001",
"managed" : true,
"policy" : "user-user-contracts-ilm",
"index_creation_date_millis" : 1768778996385,
"time_since_index_creation" : "580ms",
"lifecycle_date_millis" : 1768778996385,
"age" : "580ms",
"phase" : "hot",
"phase_time_millis" : 1768778996630,
"action" : "rollover",
"action_time_millis" : 1768778996630,
"step" : "check-rollover-ready",
"step_time_millis" : 1768778996630,
"phase_execution" : {
"policy" : "user-user-contracts-ilm",
"phase_definition" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_age" : "30d",
"max_primary_shard_docs" : 200000000,
"min_docs" : 1,
"max_primary_shard_size" : "5gb"
}
}
},
"version" : 1,
"modified_date_in_millis" : 1768778995754
},
"skip" : false
}
}
}
[1;32m[OK][0m Minimal User Contracts Index bootstrap complete.
Index (concrete): user_sau_main_dev_user_contracts-000001
Alias (stable) : user_sau_main_dev_user_contracts (is_write_index=true)
ILM policy : user-user-contracts-ilm
Default pipeline: user-contracts-embed-pipeline-001
Vectors : dense_vector dims=1536 (KNN cosine)
==================================================================
PRIVACY-BY-DESIGN: Minimal User Index Compliance
==================================================================
INDEXED: user_id, tenant_id, home_region, status, username, display_name
email_hash, phone_hash, country_code, region_code, tags, segments
contract_id, contract_type, contract_status, dates
EXCLUDED: gender, age, DOB, national_id, exact_address, payment_data
PRINCIPLE: Route to region + safe display snippet, load full details from Postgres
==================================================================
✅ 02-create-index.sh completed
▶️ Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/user/contracts/03-llm.sh
==================================================================
STEP 3: LLM Semantic Search Configuration (User Contracts)
==================================================================
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] Checking Elasticsearch license for semantic search...
[INFO] License type: unknown
[WARN] Semantic search features require Enterprise/Platinum license (found: unknown)
[WARN] Text-based search will still work, but KNN/vector search is unavailable
[INFO] [verify] Checking inference endpoint: user-contracts-embedding-001
[WARN] Inference endpoint user-contracts-embedding-001 not found (HTTP 404)
[WARN] Run 01-create-model-and-pipeline.sh first to create the endpoint
[WARN] Semantic search will fall back to text-based search
[INFO] [create] Creating search template for user contracts...
[WARN] Search template creation returned HTTP 400. Response:
HTTP/1.1 400 Bad Request
X-elastic-product: Elasticsearch
content-type: application/json
content-length: 1847
{"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[40:13] [stored script source] failed to parse field [source]"}],"type":"x_content_parse_exception","reason":"[40:13] [stored script source] failed to parse field [source]","caused_by":{"type":"unchecked_i_o_exception","reason":"com.fasterxml.jackson.core.JsonParseException: Unexpected character ('{' (code 123)): was expecting double-quote to start field name\n at [Source: (byte[])\"{\n \"script\": {\n \"lang\": \"mustache\",\n \"source\": {\n \"size\": \"{{size}}{{^size}}10{{/size}}\",\n \"query\": {\n \"bool\": {\n \"should\": [\n {\n \"multi_match\": {\n \"query\": \"{{query}}\",\n \"fields\": [\n \"username^3\",\n \"display_name^2\",\n \"contract_summary_en\",\n \"contract_summary_ar\"\n ],\n \"type\": \"best_fields\",\n \"fuzzin\"[truncated 1411 bytes]; line: 40, column: 14]","caused_by":{"type":"json_parse_exception","reason":"Unexpected character ('{' (code 123)): was expecting double-quote to start field name\n at [Source: (byte[])\"{\n \"script\": {\n \"lang\": \"mustache\",\n \"source\": {\n \"size\": \"{{size}}{{^size}}10{{/size}}\",\n \"query\": {\n \"bool\": {\n \"should\": [\n {\n \"multi_match\": {\n \"query\": \"{{query}}\",\n \"fields\": [\n \"username^3\",\n \"display_name^2\",\n \"contract_summary_en\",\n \"contract_summary_ar\"\n ],\n \"type\": \"best_fields\",\n \"fuzzin\"[truncated 1411 bytes]; line: 40, column: 14]"}}},"status":400}
[INFO] [create] Creating KNN search template for semantic similarity...
[WARN] KNN template creation returned HTTP 400. Response:
HTTP/1.1 400 Bad Request
X-elastic-product: Elasticsearch
content-type: application/json
content-length: 1831
{"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[18:9] [stored script source] failed to parse field [source]"}],"type":"x_content_parse_exception","reason":"[18:9] [stored script source] failed to parse field [source]","caused_by":{"type":"unchecked_i_o_exception","reason":"com.fasterxml.jackson.core.JsonParseException: Unexpected character ('{' (code 123)): was expecting double-quote to start field name\n at [Source: (byte[])\"{\n \"script\": {\n \"lang\": \"mustache\",\n \"source\": {\n \"size\": \"{{size}}{{^size}}10{{/size}}\",\n \"knn\": {\n \"field\": \"{{vector_field}}{{^vector_field}}embedding_en{{/vector_field}}\",\n \"query_vector_builder\": {\n \"text_embedding\": {\n \"model_id\": \"user-contracts-embedding-001\",\n \"model_text\": \"{{query}}\"\n }\n },\n \"k\": \"{{k}}{{^k}}10{{/k}}\",\n \"num_candidates\": \"{{num_candidates}}{{^num_candidates}}100{{/num_candidate\"[truncated 662 bytes]; line: 18, column: 10]","caused_by":{"type":"json_parse_exception","reason":"Unexpected character ('{' (code 123)): was expecting double-quote to start field name\n at [Source: (byte[])\"{\n \"script\": {\n \"lang\": \"mustache\",\n \"source\": {\n \"size\": \"{{size}}{{^size}}10{{/size}}\",\n \"knn\": {\n \"field\": \"{{vector_field}}{{^vector_field}}embedding_en{{/vector_field}}\",\n \"query_vector_builder\": {\n \"text_embedding\": {\n \"model_id\": \"user-contracts-embedding-001\",\n \"model_text\": \"{{query}}\"\n }\n },\n \"k\": \"{{k}}{{^k}}10{{/k}}\",\n \"num_candidates\": \"{{num_candidates}}{{^num_candidates}}100{{/num_candidate\"[truncated 662 bytes]; line: 18, column: 10]"}}},"status":400}
[INFO] [create] Creating user lookup template for exact matches...
[WARN] Lookup template creation returned HTTP 400. Response:
HTTP/1.1 400 Bad Request
X-elastic-product: Elasticsearch
content-type: application/json
content-length: 1831
{"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[9:13] [stored script source] failed to parse field [source]"}],"type":"x_content_parse_exception","reason":"[9:13] [stored script source] failed to parse field [source]","caused_by":{"type":"unchecked_i_o_exception","reason":"com.fasterxml.jackson.core.JsonParseException: Unexpected character ('{' (code 123)): was expecting double-quote to start field name\n at [Source: (byte[])\"{\n \"script\": {\n \"lang\": \"mustache\",\n \"source\": {\n \"size\": 1,\n \"query\": {\n \"bool\": {\n \"must\": [\n {{#email_hash}}\n { \"term\": { \"email_hash\": \"{{email_hash}}\" } }\n {{/email_hash}}\n {{#phone_hash}}\n { \"term\": { \"phone_hash\": \"{{phone_hash}}\" } }\n {{/phone_hash}}\n {{#user_id}}\n { \"term\": { \"user_id\": \"{{user_id}}\" } }\n {{/user_id}}\n {{#username}}\n {\"[truncated 530 bytes]; line: 9, column: 14]","caused_by":{"type":"json_parse_exception","reason":"Unexpected character ('{' (code 123)): was expecting double-quote to start field name\n at [Source: (byte[])\"{\n \"script\": {\n \"lang\": \"mustache\",\n \"source\": {\n \"size\": 1,\n \"query\": {\n \"bool\": {\n \"must\": [\n {{#email_hash}}\n { \"term\": { \"email_hash\": \"{{email_hash}}\" } }\n {{/email_hash}}\n {{#phone_hash}}\n { \"term\": { \"phone_hash\": \"{{phone_hash}}\" } }\n {{/phone_hash}}\n {{#user_id}}\n { \"term\": { \"user_id\": \"{{user_id}}\" } }\n {{/user_id}}\n {{#username}}\n {\"[truncated 530 bytes]; line: 9, column: 14]"}}},"status":400}
[1;32m[OK][0m LLM configuration complete for User Contracts index.
Available search templates:
1. user_sau_main_dev_user_contracts_search - Text + filter search
2. user_sau_main_dev_user_contracts_knn_search - KNN vector similarity search
3. user_sau_main_dev_user_contracts_lookup - Exact match lookup (by hash)
Usage examples:
Text search:
POST https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/user_sau_main_dev_user_contracts/_search/template
{ "id": "user_sau_main_dev_user_contracts_search", "params": { "query": "service agreement", "tenant_id": "t1" } }
KNN semantic search:
POST https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/user_sau_main_dev_user_contracts/_search/template
{ "id": "user_sau_main_dev_user_contracts_knn_search", "params": { "query": "find contracts about data privacy" } }
User lookup (by hashed email):
POST https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/user_sau_main_dev_user_contracts/_search/template
{ "id": "user_sau_main_dev_user_contracts_lookup", "params": { "email_hash": "<sha256_hash>" } }
✅ 03-llm.sh completed
▶️ Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/user/contracts/04-index-sample-data.sh
==================================================================
STEP 4: Index Sample Data (User Contracts - Minimal Index)
==================================================================
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[WARN] Pipeline 'user-contracts-embed-pipeline-001' not found (HTTP 404); proceeding without it.
[INFO] [bulk] Index seed documents -> user_sau_main_dev_user_contracts
[INFO] PRIVACY: Using hashed email/phone, coarse location, no sensitive PII
[1;32m[OK][0m Seeded and refreshed.
[summary] items=5 errors=0
[INFO] [verify] Search a sample term: 'service agreement'
{
"took" : 126,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 0.87546873,
"hits" : [
{
"_index" : "user_sau_main_dev_user_contracts-000001",
"_id" : "U05x05sBnWgIQnI0oEEX",
"_score" : 0.87546873,
"_source" : {
"user_id" : "u_003_uae_dev",
"tenant_id" : "t_uae_dev",
"home_region" : "are",
"home_env" : "dev",
"status" : "active",
"username" : "omar.hassan",
"username_lc" : "omar.hassan",
"display_name" : "Omar Hassan",
"email_hash" : "c0d2e4f6a8b0c2d4e6f8a0b2c4d6e8f0a2b4c6d8e0f2a4b6c8d0e2f4a6b8c0d2",
"phone_hash" : "d6c5b4a3928170f8e7d6c5b4a3928170f8e7d6c5b4a3928170f8e7d6c5b4a392",
"country_code" : "AE",
"region_code" : "DXB",
"tags" : [
"premium",
"early_adopter"
],
"segments" : [
"enterprise"
],
[1;32m[OK][0m Sample data indexing step completed.
==================================================================
PRIVACY COMPLIANCE: Sample Data Structure
==================================================================
INDEXED (minimal):
- Identity: user_id, tenant_id, home_region, home_env, status
- Search: username, display_name, email_hash, phone_hash
- Location: country_code, region_code (coarse only)
- Metadata: tags, segments, flags, external_refs
- Contract: contract_id, type, status, dates, summary
EXCLUDED (by design):
- Gender, Age, Date of Birth
- National ID, Passport Number
- Exact Address, Precise GPS Coordinates
- Payment/Financial Data
PRINCIPLE: Universal index for routing + lookup only.
Full user details loaded from zonal Postgres.
==================================================================
✅ 04-index-sample-data.sh completed
▶️ Running /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/user/contracts/05-create-cdc-index.sh
==================================================================
STEP 5: Create CDC User Contracts Index (for dashboard visibility)
==================================================================
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] Creating CDC index: user_sau_main_dev_user_contracts_cdc
[INFO] Index does not exist (status 404), creating...
[1;32m[OK][0m CDC index created successfully
Index: user_sau_main_dev_user_contracts_cdc
URL: https://search-user-sau-main-dev-elasticsearch-node-01.fastorder.com:9200/user_sau_main_dev_user_contracts_cdc
==================================================================
PRIVACY-BY-DESIGN: CDC Index Structure
==================================================================
INDEXED (in 'after' object):
- Identity: user_id, tenant_id, home_region, home_env, status
- Search: username, display_name (email_hash/phone_hash stored only)
- Location: country_code, region_code (coarse)
- Metadata: tags, segments, flags, external_refs
- Contract: contract_id, type, status, dates
STORED BUT NOT INDEXED (in 'before' object):
- user_id, tenant_id, status, contract_status (for audit trail)
DISABLED (not stored):
- source (Debezium metadata, not needed for search)
EXCLUDED BY DESIGN:
- Gender, Age, DOB, National ID
- Exact Address, Precise GPS
- Payment/Financial Data
==================================================================
--- Dashboard can now list this index before CDC pipeline writes data
--- ES Sink connector will write CDC data to this index automatically
✅ 05-create-cdc-index.sh completed
=== Phase 3: Optional search smoke tests ===
(semantic search test script not found: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/search-semantic.sh)
(hybrid search test script not found: /opt/fastorder/bash/scripts/env_app_setup/setup/03-search/engine/elasticsearch/03-create-index-llm/steps/hybrid-search.sh)
==================================================================
🎉 LLM/Semantic Search setup completed successfully!
==================================================================
Available commands:
• Test semantic search:
bash steps/search-semantic.sh en "password policy"
bash steps/search-semantic.sh ar "كلمة المرور"
• Test hybrid search:
bash steps/hybrid-search.sh en "user authentication"
bash steps/hybrid-search.sh ar "مصادقة المستخدم"
Alias : user_sau_main_dev_account_router
Index : user_sau_main_dev_account_router-000001
ILM : user-account-router-ilm
Model : user-text-embedding-001
Pipeline: user-embed-pipeline-001
==================================================================
[0;32m[1m✓ Step 3 completed successfully![0m
[0;36m[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;32m[1mStep 4: Executing Monitoring Setup[0m
[0;35mFolder: 10-monitoring-setup[0m
[0;36m[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[INFO] Using web-provided environment: user-sau-main-dev
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Elasticsearch Monitoring Integration for user-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-user-sau-main-dev
[1;32m[OK][0m Observability cell endpoints registered for user-sau-main-dev
[1;32m[OK][0m ✓ Observability cell is ready
[INFO] 2️⃣ Discovering Elasticsearch configuration...
[1;32m[OK][0m ✓ Found Elasticsearch at 10.100.1.152:9200
[INFO] 3️⃣ Setting up elasticsearch_exporter integration...
[INFO] Using elasticsearch_exporter port: 9114
[INFO] SSL certificates configured for elasticsearch_exporter:
[INFO] CA cert: /etc/elasticsearch/user-sau-main-dev/node-01/certs/http_ca.crt
[INFO] Client cert: /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.crt
[INFO] Client key: /etc/elasticsearch/user-sau-main-dev/node-01/certs/clients/es-client/es-client.key
[INFO] Checking observability cell readiness: obs-user-sau-main-dev
[1;32m[OK][0m Observability cell endpoints registered for user-sau-main-dev
[INFO] Setting up elasticsearch_exporter for user-sau-main-dev
[INFO] Elasticsearch exporter will bind to: 10.100.1.152:9114
[2026-01-18 23:30:00 UTC] USER=www-data EUID=0 PID=4000776 ACTION=passthru ARGS=mv /tmp/elasticsearch_exporter-user-sau-main-dev.service /etc/systemd/system/elasticsearch_exporter-user-sau-main-dev.service
[2026-01-18 23:30:00 UTC] USER=www-data EUID=0 PID=4000785 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:30:01 UTC] USER=www-data EUID=0 PID=4000832 ACTION=passthru ARGS=systemctl enable elasticsearch_exporter-user-sau-main-dev.service
Created symlink /etc/systemd/system/multi-user.target.wants/elasticsearch_exporter-user-sau-main-dev.service → /etc/systemd/system/elasticsearch_exporter-user-sau-main-dev.service.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
IP Conflict Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
IP Address: 10.100.1.152
Port: 9114
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔍 Checking IP conflict for user-sau-main-dev on 10.100.1.152:9114...
✅ IP 10.100.1.152:9114 is available - no conflicts detected
🔍 Checking for orphaned processes that might conflict...
✅ No orphaned processes detected
✅ All checks passed - safe to proceed with user-sau-main-dev setup
[2026-01-18 23:30:02 UTC] USER=www-data EUID=0 PID=4001869 ACTION=passthru ARGS=systemctl restart elasticsearch_exporter-user-sau-main-dev.service
[1;32m[OK][0m elasticsearch_exporter configured on 10.100.1.152:9114
[INFO] Register this endpoint in metrics-user-sau-main-dev.fastorder.com scrape config
[1;32m[OK][0m ✓ elasticsearch_exporter integration complete
[INFO] 3.5️⃣ Configuring Prometheus to scrape Elasticsearch metrics...
[2026-01-18 23:30:05 UTC] USER=www-data EUID=0 PID=4002485 ACTION=passthru ARGS=grep -q job_name: 'elasticsearch' /etc/prometheus/obs-user-sau-main-dev/prometheus.yml
[INFO] Adding Elasticsearch scrape target to Prometheus configuration...
[INFO] Created backup: /etc/prometheus/obs-user-sau-main-dev/prometheus.yml.backup-1768779005
[2026-01-18 23:30:05 UTC] USER=www-data EUID=0 PID=4002594 ACTION=passthru ARGS=grep -q job_name: 'elasticsearch' /etc/prometheus/obs-user-sau-main-dev/prometheus.yml
[INFO] ✓ Elasticsearch job successfully inserted into config
[INFO] Validating Prometheus configuration with promtool...
Checking /etc/prometheus/obs-user-sau-main-dev/prometheus.yml
SUCCESS: 1 rule files found
SUCCESS: /etc/prometheus/obs-user-sau-main-dev/prometheus.yml is valid prometheus config file syntax
Checking /etc/prometheus/obs-user-sau-main-dev/rules/basic_alerts.yml
SUCCESS: 4 rules found
[1;32m[OK][0m ✓ Prometheus configuration validation PASSED
[1;32m[OK][0m ✓ Prometheus configuration updated successfully
[2026-01-18 23:30:05 UTC] USER=www-data EUID=0 PID=4002664 ACTION=passthru ARGS=systemctl is-active --quiet prometheus-obs-user-sau-main-dev.service
[INFO] Reloading Prometheus configuration...
[2026-01-18 23:30:05 UTC] USER=www-data EUID=0 PID=4002686 ACTION=passthru ARGS=systemctl restart prometheus-obs-user-sau-main-dev.service
[2026-01-18 23:30:09 UTC] USER=www-data EUID=0 PID=4002781 ACTION=passthru ARGS=systemctl is-active --quiet prometheus-obs-user-sau-main-dev.service
[1;32m[OK][0m ✓ Prometheus reloaded successfully
[2026-01-18 23:30:09 UTC] USER=www-data EUID=0 PID=4002802 ACTION=fsop ARGS=rm -f /tmp/prometheus_es_add.yml
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Elasticsearch Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Service: elasticsearch_exporter-user-sau-main-dev.service
[INFO] Metrics: http://localhost:9114/metrics
[INFO] Prometheus: https://metrics-user-sau-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-user-sau-main-dev.fastorder.com
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 4️⃣ Registering Elasticsearch nodes to monitoring database...
[INFO] Constructed FQDN: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com
[INFO] Registering: user-sau-main-dev-node-01
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Elasticsearch
[INFO] Identifier: user-sau-main-dev-node-01
[INFO] Identifier Parent: cluster
[INFO] IP: 10.100.1.152
[INFO] Port: 9200
[INFO] FQDN: search-user-sau-main-dev-elasticsearch-node-01.fastorder.com
[INFO] Status: running
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 04ac0ee9-53da-4099-bb77-80a64f461538
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[1;32m[OK][0m ✓ Registered: user-sau-main-dev-node-01
[1;32m[OK][0m ✓ Elasticsearch node registration completed successfully
[INFO] 5️⃣ Verifying monitoring integration...
[INFO] Checking elasticsearch_exporter service...
[1;32m[OK][0m ✓ elasticsearch_exporter-user-sau-main-dev.service is ACTIVE
[INFO] Checking Prometheus service...
[1;32m[OK][0m ✓ prometheus-obs-user-sau-main-dev.service is ACTIVE
[INFO] Validating Prometheus configuration...
[1;32m[OK][0m ✓ Prometheus configuration is VALID
[INFO] Checking Prometheus targets (waiting 35s for first scrape cycle)...
[2026-01-18 23:30:45 UTC] USER=www-data EUID=0 PID=4003621 ACTION=passthru ARGS=grep -q tls_server_config /etc/prometheus/obs-user-sau-main-dev/web-config.yml
[1;32m[OK][0m ✓ Prometheus has Elasticsearch target configured
[1;32m[OK][0m ✓ Elasticsearch target is UP and being scraped
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ All monitoring integration steps completed
[INFO] ✅ All verifications PASSED
[INFO] ✅ Elasticsearch registered to dashboard database
[INFO] ✅ Prometheus scraping Elasticsearch metrics
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[1m✓ Step 4 completed successfully![0m
[0;36m[1m════════════════════════════════════════════════════════════════[0m
[0;32m[1m🎉 All deployment tasks completed successfully![0m
[0;32m✓[0m ✅ Search infrastructure (elasticsearch) setup completed successfully
[0;34m[INFO][0m Using eventbus engine from EVENTBUS_ENGINE environment variable: kafka
[0;34m[INFO][0m Cleaning up any existing locks...
[0;32m[1mStarting eventbus engine: kafka[0m
[1;33m═══════════════════════════════════════════════[0m
[0;34m[INFO][0m Using environment from web interface: user-sau-main-dev
[0;32m[2026-01-18 23:30:45][0m Using web-provided environment: user-sau-main-dev
[0;32m[2026-01-18 23:30:45][0m Service: user, Zone: sau, Branch: main, Env: dev
[0;32m✓[0m Environment initialized successfully (mode: general)
[0;34m[INFO][0m Starting Kafka setup process...
[0;34m[INFO][0m Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Found 13 step(s) to execute
[0;34m[INFO][0m 📦 Step 1/13: install debezium connector...
═══════════════════════════════════════════════════════════════════
Fetching latest versions from Maven Central...
Installing Debezium PostgreSQL Connector
Debezium version: 3.4.0.Final
pgjdbc version: 42.7.9
═══════════════════════════════════════════════════════════════════
[OK] Debezium 3.4.0.Final with pgjdbc 42.7.9 already installed
[0;32m[OK][0m ✅ Step 1 completed: 00-install-debezium-connector.sh
[0;34m[INFO][0m 📦 Step 2/13: kafka setup...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials for secrets vault...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
🧹 Checking for orphaned Kafka processes on ports 9092, 9093, 8083...
⚠️ Found process on port 9092 (PIDs: [2026-01-18 23:30:45 UTC] USER=www-data EUID=0 PID=4003719 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 2>/dev/null || true), killing...
[2026-01-18 23:30:46 UTC] USER=www-data EUID=0 PID=4003742 ACTION=passthru ARGS=bash -c kill -9 [2026-01-18 23:30:45 UTC] USER=www-data EUID=0 PID=4003719 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 2>/dev/null || true 2>/dev/null || true
⚠️ Found process on port 9093 (PIDs: [2026-01-18 23:30:47 UTC] USER=www-data EUID=0 PID=4003756 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 2>/dev/null || true), killing...
[2026-01-18 23:30:47 UTC] USER=www-data EUID=0 PID=4003772 ACTION=passthru ARGS=bash -c kill -9 [2026-01-18 23:30:47 UTC] USER=www-data EUID=0 PID=4003756 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 2>/dev/null || true 2>/dev/null || true
⚠️ Found process on port 8083 (PIDs: [2026-01-18 23:30:48 UTC] USER=www-data EUID=0 PID=4003789 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 2>/dev/null || true
3331894
3850931), killing...
[2026-01-18 23:30:48 UTC] USER=www-data EUID=0 PID=4003799 ACTION=passthru ARGS=bash -c kill -9 [2026-01-18 23:30:48 UTC] USER=www-data EUID=0 PID=4003789 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 2>/dev/null || true
3331894
3850931 2>/dev/null || true
/usr/bin/bash: line 2: 3331894: command not found
✅ Port cleanup completed
Ensuring KAFKA application environment for coordinator...
[0;34m[INFO][0m Creating KAFKA application environment...
[INFO] 🎯 Custom Environment Creation (Example Wrapper)
[INFO] 📁 Orchestrator Library: /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] 💾 State Directory: /opt/fastorder/bash/scripts/env_app_setup/state
[INFO] 🚀 Calling centralized orchestrator: fo-env create-app
[INFO] 📋 Arguments: --service user --zone sau --branch main --env dev --domain eventbus-user-sau-main-dev-kafka-connect --app kafka-connect
[INFO] Creating application-specific environment configuration
[INFO] Environment ID: user-sau-main-dev
[INFO] Application: kafka-connect
[INFO] Base environment user-sau-main-dev already exists
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Allocated kafka-connect IP: 10.100.1.159
[INFO] Generated domain: eventbus-user-sau-main-dev-kafka-connect.fastorder.com
[INFO] Configuring network interface for kafka-connect IP: 10.100.1.159
[2026-01-18 23:30:50 UTC] USER=www-data EUID=0 PID=4004100 ACTION=passthru ARGS=ip addr add 10.100.1.159/32 dev eth0 label eth0:159
[ OK ] Configured kafka-connect IP 10.100.1.159 on interface eth0:159
[INFO] Creating systemd service for kafka-connect IP persistence...
[2026-01-18 23:30:50 UTC] USER=www-data EUID=0 PID=4004133 ACTION=passthru ARGS=systemctl daemon-reload
[ OK ] kafka-connect IP will persist across reboots
[INFO] Updating topology with application-specific configuration...
[ OK ] Topology updated with application-specific configuration
[INFO] Binding kafka-connect IP to domain: 10.100.1.159 -> eventbus-user-sau-main-dev-kafka-connect.fastorder.com
[ OK ] Successfully bound eventbus-user-sau-main-dev-kafka-connect.fastorder.com to 10.100.1.159
[ OK ] Domain correctly mapped
[ OK ] Application environment created successfully!
[INFO]
[INFO] Application Details:
[INFO] Environment ID: user-sau-main-dev
[INFO] Application: kafka-connect
[INFO] IP: 10.100.1.159
[INFO] Domain: eventbus-user-sau-main-dev-kafka-connect.fastorder.com
[INFO]
[INFO] To use this application:
[INFO] source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO] init_environment kafka-connect
[INFO] echo $VM_IP # Returns: 10.100.1.159
[ OK ] 🎉 Environment creation completed successfully!
[INFO] 📋 What happened:
[INFO] ✅ Called centralized orchestrator at /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] ✅ All topology.json management handled centrally
[INFO] ✅ Application-specific IP and domain configured
[INFO] ✅ Network interface configured and made persistent
[INFO] ✅ Domain binding added to /etc/hosts (if not skipped)
[INFO] 🔧 To use the centralized orchestrator directly:
[INFO] # Add orchestrator to PATH
[INFO] export PATH="/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/bin:$PATH"
[INFO] # Then call directly
[INFO] fo-env create-app --service auth --zone uae --env dev --app redis
[INFO] 📚 For more orchestrator commands:
[INFO] fo-env --help
[0;32mCreated KAFKA environment: eventbus-user-sau-main-dev-kafka-connect.fastorder.com (10.100.1.159)[0m
Ensuring KAFKA_BROKER_IP application environment for coordinator...
[0;34m[INFO][0m Creating KAFKA application environment...
[INFO] 🎯 Custom Environment Creation (Example Wrapper)
[INFO] 📁 Orchestrator Library: /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] 💾 State Directory: /opt/fastorder/bash/scripts/env_app_setup/state
[INFO] 🚀 Calling centralized orchestrator: fo-env create-app
[INFO] 📋 Arguments: --service user --zone sau --branch main --env dev --domain eventbus-user-sau-main-dev-kafka-broker-01 --app kafka-broker
[INFO] Creating application-specific environment configuration
[INFO] Environment ID: user-sau-main-dev
[INFO] Application: kafka-broker
[INFO] Base environment user-sau-main-dev already exists
/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/common.sh: line 261: echo: write error: Broken pipe
[INFO] Allocated kafka-broker IP: 10.100.1.141
[INFO] Generated domain: eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com
[INFO] Configuring network interface for kafka-broker IP: 10.100.1.141
[2026-01-18 23:30:52 UTC] USER=www-data EUID=0 PID=4004697 ACTION=passthru ARGS=ip addr add 10.100.1.141/32 dev eth0 label eth0:141
[ OK ] Configured kafka-broker IP 10.100.1.141 on interface eth0:141
[INFO] Creating systemd service for kafka-broker IP persistence...
[2026-01-18 23:30:52 UTC] USER=www-data EUID=0 PID=4004716 ACTION=passthru ARGS=systemctl daemon-reload
[ OK ] kafka-broker IP will persist across reboots
[INFO] Updating topology with application-specific configuration...
[ OK ] Topology updated with application-specific configuration
[INFO] Binding kafka-broker IP to domain: 10.100.1.141 -> eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com
[ OK ] Successfully bound eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com to 10.100.1.141
[ OK ] Domain correctly mapped
[ OK ] Application environment created successfully!
[INFO]
[INFO] Application Details:
[INFO] Environment ID: user-sau-main-dev
[INFO] Application: kafka-broker
[INFO] IP: 10.100.1.141
[INFO] Domain: eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com
[INFO]
[INFO] To use this application:
[INFO] source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO] init_environment kafka-broker
[INFO] echo $VM_IP # Returns: 10.100.1.141
[ OK ] 🎉 Environment creation completed successfully!
[INFO] 📋 What happened:
[INFO] ✅ Called centralized orchestrator at /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] ✅ All topology.json management handled centrally
[INFO] ✅ Application-specific IP and domain configured
[INFO] ✅ Network interface configured and made persistent
[INFO] ✅ Domain binding added to /etc/hosts (if not skipped)
[INFO] 🔧 To use the centralized orchestrator directly:
[INFO] # Add orchestrator to PATH
[INFO] export PATH="/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/bin:$PATH"
[INFO] # Then call directly
[INFO] fo-env create-app --service auth --zone uae --env dev --app redis
[INFO] 📚 For more orchestrator commands:
[INFO] fo-env --help
[0;32mCreated KAFKA_BROKER_DOMAIN environment: eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com (10.100.1.141)[0m
[0;34m[INFO][0m Kafka Broker IP: 10.100.1.141
[0;34m[INFO][0m Kafka Connect IP: 10.100.1.159
[0;34m[INFO][0m Registered /etc/hosts: eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com -> 10.100.1.141
[0;34m[INFO][0m Registered /etc/hosts: eventbus-user-sau-main-dev-kafka-connect.fastorder.com -> 10.100.1.159
🔐 Initializing keystore passwords...
[0;34m[INFO][0m 🔍 Checking secrets backend (provider: aws)...
[0;32m✅ Retrieved passwords from remote backend[0m
[0;34m[INFO][0m ✅ Using existing passwords from backend
✅ Keystore passwords initialized
- Keystore password: OV9hCGeL... (32 chars)
- Truststore password: dV4AfOMs... (32 chars)
[0;34m[INFO][0m 🔐 Vaulting kafka passwords to remote backend...
[0;32m✅ Passwords vaulted to remote backend[0m
✅ Kafka keystore passwords saved to AWS Secrets Manager
[INFO] Generating for: user-sau-main-dev (host=eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com ip=10.100.1.141)
[2026-01-18 23:30:57 UTC] USER=www-data EUID=0 PID=4004993 ACTION=fsop ARGS=rm -rf /opt/kafka/secrets/user-sau-main-dev/coordinator /data/kafka/user-sau-main-dev/coordinator
[2026-01-18 23:30:57 UTC] USER=www-data EUID=0 PID=4005004 ACTION=fsop ARGS=mkdir -p /opt/kafka/secrets/user-sau-main-dev/coordinator /opt/kafka/config/user-sau-main-dev/coordinator /opt/kafka/secrets/user-sau-main-dev/coordinator/pem /data/kafka/user-sau-main-dev_coordinator-data
[2026-01-18 23:30:57 UTC] USER=www-data EUID=0 PID=4005013 ACTION=fsop ARGS=chown -R kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator
[2026-01-18 23:30:57 UTC] USER=www-data EUID=0 PID=4005030 ACTION=fsop ARGS=chown -R kafka:kafka /opt/kafka/config/user-sau-main-dev/coordinator /data/kafka/user-sau-main-dev_coordinator-data
[2026-01-18 23:30:57 UTC] USER=www-data EUID=0 PID=4005039 ACTION=fsop ARGS=chmod 770 /opt/kafka/config/user-sau-main-dev/coordinator /data/kafka/user-sau-main-dev_coordinator-data
[2026-01-18 23:30:57 UTC] USER=www-data EUID=0 PID=4005048 ACTION=fsop ARGS=chmod 750 /opt/kafka/secrets/user-sau-main-dev/coordinator
[2026-01-18 23:30:57 UTC] USER=www-data EUID=0 PID=4005057 ACTION=fsop ARGS=chmod 750 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
[2026-01-18 23:30:57 UTC] USER=www-data EUID=0 PID=4005067 ACTION=fsop ARGS=chmod 700 /tmp/fo-tls.Ba7yLu
[2026-01-18 23:30:57 UTC] USER=www-data EUID=0 PID=4005076 ACTION=fsop ARGS=chmod 755 /tmp/fo-tls.Ba7yLu
[2026-01-18 23:30:57 UTC] USER=www-data EUID=0 PID=4005085 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/fo-tls.Ba7yLu/ra_root.crt
[2026-01-18 23:30:57 UTC] USER=www-data EUID=0 PID=4005094 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/fo-tls.Ba7yLu/ra_root.key
[2026-01-18 23:30:57 UTC] USER=www-data EUID=0 PID=4005103 ACTION=fsop ARGS=chmod 644 /tmp/fo-tls.Ba7yLu/ra_root.crt
[2026-01-18 23:30:57 UTC] USER=www-data EUID=0 PID=4005112 ACTION=fsop ARGS=chmod 644 /tmp/fo-tls.Ba7yLu/ra_root.key
Certificate was added to keystore
[2026-01-18 23:30:58 UTC] USER=www-data EUID=0 PID=4005145 ACTION=fsop ARGS=mv /tmp/fo-tls.Ba7yLu/truststore.jks /opt/kafka/secrets/user-sau-main-dev/coordinator/truststore.jks
[2026-01-18 23:30:58 UTC] USER=www-data EUID=0 PID=4005154 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/user-sau-main-dev/coordinator/truststore.jks
[2026-01-18 23:30:58 UTC] USER=www-data EUID=0 PID=4005163 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/user-sau-main-dev/coordinator/truststore.jks
Generating 4,096 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 825 days
for: CN=eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com, OU=Kafka Broker, O=FastOrder, C=AE
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /tmp/fo-tls.Ba7yLu/kafka.server.keystore.jks -destkeystore /tmp/fo-tls.Ba7yLu/kafka.server.keystore.jks -deststoretype pkcs12".
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /tmp/fo-tls.Ba7yLu/kafka.server.keystore.jks -destkeystore /tmp/fo-tls.Ba7yLu/kafka.server.keystore.jks -deststoretype pkcs12".
Certificate request self-signature ok
subject=C = AE, O = FastOrder, OU = Kafka Broker, CN = eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com
Certificate was added to keystore
Warning:
Certificate reply was installed in keystore
Warning:
[2026-01-18 23:31:00 UTC] USER=www-data EUID=0 PID=4005278 ACTION=fsop ARGS=mv /tmp/fo-tls.Ba7yLu/kafka.server.keystore.jks /opt/kafka/secrets/user-sau-main-dev/coordinator/kafka.server.keystore.jks
[2026-01-18 23:31:00 UTC] USER=www-data EUID=0 PID=4005287 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/user-sau-main-dev/coordinator/kafka.server.keystore.jks
[2026-01-18 23:31:00 UTC] USER=www-data EUID=0 PID=4005296 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/user-sau-main-dev/coordinator/kafka.server.keystore.jks
Generating 4,096 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 825 days
for: CN=eventbus-user-sau-main-dev-kafka-connect.fastorder.com, OU=Kafka Connect REST, O=FastOrder, C=AE
Certificate request self-signature ok
subject=C = AE, O = FastOrder, OU = Kafka Connect REST, CN = eventbus-user-sau-main-dev-kafka-connect.fastorder.com
Certificate was added to keystore
Certificate reply was installed in keystore
[2026-01-18 23:31:04 UTC] USER=www-data EUID=0 PID=4005491 ACTION=fsop ARGS=mv /tmp/fo-tls.Ba7yLu/connect-rest.keystore.p12 /opt/kafka/secrets/user-sau-main-dev/coordinator/connect-rest.keystore.p12
[2026-01-18 23:31:04 UTC] USER=www-data EUID=0 PID=4005506 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/user-sau-main-dev/coordinator/connect-rest.keystore.p12
[2026-01-18 23:31:04 UTC] USER=www-data EUID=0 PID=4005524 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/user-sau-main-dev/coordinator/connect-rest.keystore.p12
Certificate request self-signature ok
subject=CN = kafka-client-user-sau-main-dev, OU = Kafka Client, O = FastOrder, C = AE
[2026-01-18 23:31:04 UTC] USER=www-data EUID=0 PID=4005540 ACTION=fsop ARGS=cp /tmp/fo-tls.Ba7yLu/ra_root.crt /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem
[2026-01-18 23:31:04 UTC] USER=www-data EUID=0 PID=4005549 ACTION=fsop ARGS=cp /tmp/fo-tls.Ba7yLu/client-key.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:31:04 UTC] USER=www-data EUID=0 PID=4005558 ACTION=fsop ARGS=cp /tmp/fo-tls.Ba7yLu/client-cert.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
[2026-01-18 23:31:04 UTC] USER=www-data EUID=0 PID=4005567 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
[2026-01-18 23:31:04 UTC] USER=www-data EUID=0 PID=4005576 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
[2026-01-18 23:31:04 UTC] USER=www-data EUID=0 PID=4005585 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:31:04 UTC] USER=www-data EUID=0 PID=4005595 ACTION=fsop ARGS=mv /tmp/fo-tls.Ba7yLu/kafka.client.keystore.p12 /opt/kafka/secrets/user-sau-main-dev/coordinator/kafka.client.keystore.p12
[2026-01-18 23:31:04 UTC] USER=www-data EUID=0 PID=4005604 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/user-sau-main-dev/coordinator/kafka.client.keystore.p12
🔐 Ensuring kafka user has access to PostgreSQL certificates...
✅ kafka is already in postgres group
🧹 Cleaning up conflicting services and processes on Kafka ports on 10.100.1.141...
🔪 Killing processes on 10.100.1.141:8083: [2026-01-18 23:31:05 UTC] USER=www-data EUID=0 PID=4005651 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.141:8083 -t 2>/dev/null || true
[2026-01-18 23:31:05 UTC] USER=www-data EUID=0 PID=4005668 ACTION=passthru ARGS=bash -c kill -9 [2026-01-18 23:31:05 UTC] USER=www-data EUID=0 PID=4005651 ACTION=passthru ARGS=bash -c lsof -ti tcp:8083 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.141:8083 -t 2>/dev/null || true
🔪 Killing processes on 10.100.1.141:9092: [2026-01-18 23:31:05 UTC] USER=www-data EUID=0 PID=4005679 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.141:9092 -t 2>/dev/null || true
[2026-01-18 23:31:05 UTC] USER=www-data EUID=0 PID=4005701 ACTION=passthru ARGS=bash -c kill -9 [2026-01-18 23:31:05 UTC] USER=www-data EUID=0 PID=4005679 ACTION=passthru ARGS=bash -c lsof -ti tcp:9092 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.141:9092 -t 2>/dev/null || true
🔪 Killing processes on 10.100.1.141:9093: [2026-01-18 23:31:05 UTC] USER=www-data EUID=0 PID=4005712 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.141:9093 -t 2>/dev/null || true
[2026-01-18 23:31:05 UTC] USER=www-data EUID=0 PID=4005742 ACTION=passthru ARGS=bash -c kill -9 [2026-01-18 23:31:05 UTC] USER=www-data EUID=0 PID=4005712 ACTION=passthru ARGS=bash -c lsof -ti tcp:9093 -sTCP:LISTEN 2>/dev/null | xargs -I {} lsof -p {} -a -i @10.100.1.141:9093 -t 2>/dev/null || true
✅ Port cleanup completed
🔧 Checking for Kafka Connect internal topics with incorrect cleanup policy...
📋 Kafka broker is running, checking topic cleanup policies...
✅ Topic cleanup policy fix completed
🔧 Creating environment-specific systemd units...
🔧 Writing client properties to /etc/kafka/client-user-sau-main-dev-coordinator.properties ...
[2026-01-18 23:34:12 UTC] USER=www-data EUID=0 PID=4011093 ACTION=fsop ARGS=chown root:kafka /etc/kafka/client-user-sau-main-dev-coordinator.properties
🔧 Creating PEM certificates for PHP mTLS access...
[2026-01-18 23:34:12 UTC] USER=www-data EUID=0 PID=4011111 ACTION=passthru ARGS=bash -c openssl pkcs12 -in '/opt/kafka/secrets/user-sau-main-dev/coordinator/kafka.client.keystore.p12' -clcerts -nokeys -passin pass:'OV9hCGeLdjgcwFFaqhyU34SjH3OUk4uu' -out '/opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client.crt' 2>/dev/null
[2026-01-18 23:34:12 UTC] USER=www-data EUID=0 PID=4011121 ACTION=passthru ARGS=bash -c openssl pkcs12 -in '/opt/kafka/secrets/user-sau-main-dev/coordinator/kafka.client.keystore.p12' -nocerts -nodes -passin pass:'OV9hCGeLdjgcwFFaqhyU34SjH3OUk4uu' -out '/opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client.key' 2>/dev/null
[2026-01-18 23:34:12 UTC] USER=www-data EUID=0 PID=4011132 ACTION=passthru ARGS=bash -c keytool -exportcert -alias fastorder-ra-root -keystore '/opt/kafka/secrets/user-sau-main-dev/coordinator/truststore.jks' -storepass 'dV4AfOMsnuZ0cdEeyvgt1IHch08Rnm0j' -rfc -file '/opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.crt' 2>/dev/null
[2026-01-18 23:34:13 UTC] USER=www-data EUID=0 PID=4011220 ACTION=fsop ARGS=chown root:kafka /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client.crt /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client.key /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.crt
[2026-01-18 23:34:13 UTC] USER=www-data EUID=0 PID=4011229 ACTION=fsop ARGS=chmod 0644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client.crt /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.crt
[2026-01-18 23:34:13 UTC] USER=www-data EUID=0 PID=4011238 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client.key
🔐 Creating connector secrets file for FileConfigProvider...
[2026-01-18 23:34:13 UTC] USER=www-data EUID=0 PID=4011267 ACTION=fsop ARGS=chown kafka:kafka /opt/kafka/secrets/user-sau-main-dev/coordinator/connector-secrets.properties
[2026-01-18 23:34:13 UTC] USER=www-data EUID=0 PID=4011276 ACTION=fsop ARGS=chmod 0600 /opt/kafka/secrets/user-sau-main-dev/coordinator/connector-secrets.properties
✅ Connector secrets file created: /opt/kafka/secrets/user-sau-main-dev/coordinator/connector-secrets.properties
FileConfigProvider syntax: ${file:/opt/kafka/secrets/user-sau-main-dev/coordinator/connector-secrets.properties:key_name}
🔧 Creating Canary Event timer for pipeline verification...
[2026-01-18 23:34:13 UTC] USER=www-data EUID=0 PID=4011303 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:34:14 UTC] USER=www-data EUID=0 PID=4011352 ACTION=passthru ARGS=systemctl enable kafka-canary-user-sau-main-dev.timer
[2026-01-18 23:34:14 UTC] USER=www-data EUID=0 PID=4011405 ACTION=passthru ARGS=systemctl start kafka-canary-user-sau-main-dev.timer
✅ Canary timer installed: kafka-canary-user-sau-main-dev.timer (every 5 minutes)
[2026-01-18 23:34:14 UTC] USER=www-data EUID=0 PID=4011415 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:34:19 UTC] USER=www-data EUID=0 PID=4011518 ACTION=passthru ARGS=systemctl mask kafka-server
[2026-01-18 23:34:21 UTC] USER=www-data EUID=0 PID=4011597 ACTION=passthru ARGS=systemctl mask kafka-broker
🔒 Adjusting group ownership and permissions ...
[2026-01-18 23:34:22 UTC] USER=www-data EUID=0 PID=4011651 ACTION=fsop ARGS=chown :kafka /opt/kafka/secrets/user-sau-main-dev/coordinator/truststore.jks /opt/kafka/secrets/user-sau-main-dev/coordinator/kafka.server.keystore.jks
[2026-01-18 23:34:22 UTC] USER=www-data EUID=0 PID=4011666 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/user-sau-main-dev/coordinator/truststore.jks /opt/kafka/secrets/user-sau-main-dev/coordinator/kafka.server.keystore.jks
[2026-01-18 23:34:22 UTC] USER=www-data EUID=0 PID=4011676 ACTION=fsop ARGS=chmod 0640 /opt/kafka/secrets/user-sau-main-dev/coordinator/kafka.client.keystore.p12
[2026-01-18 23:34:22 UTC] USER=www-data EUID=0 PID=4011685 ACTION=fsop ARGS=chown root:kafka /etc/kafka/client-user-sau-main-dev-coordinator.properties
[2026-01-18 23:34:22 UTC] USER=www-data EUID=0 PID=4011694 ACTION=fsop ARGS=chmod 0644 /etc/kafka/client-user-sau-main-dev-coordinator.properties
✅ Kafka configuration complete for user-sau-main-dev_coordinator
Broker ID : 65
Broker keystore : /opt/kafka/secrets/user-sau-main-dev/coordinator/kafka.server.keystore.jks
REST keystore : /opt/kafka/secrets/user-sau-main-dev/coordinator/connect-rest.keystore.p12
Truststore : /opt/kafka/secrets/user-sau-main-dev/coordinator/truststore.jks
Client PKCS12 : /opt/kafka/secrets/user-sau-main-dev/coordinator/kafka.client.keystore.p12
Data directory : /data/kafka/user-sau-main-dev_coordinator-data
Server config : /opt/kafka/config/user-sau-main-dev/coordinator/server.properties
Connect config : /opt/kafka/config/user-sau-main-dev/coordinator/connect-distributed.properties
CLI client config : /etc/kafka/client-user-sau-main-dev-coordinator.properties
🎯 Next step: Run 03-restart-kafka-related-services.sh to start services
[0;32m[OK][0m ✅ Step 2 completed: 01-kafka-setup.sh
[0;34m[INFO][0m 📦 Step 3/13: metadata...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🎯 Kafka metadata mode: kraft
╔════════════════════════════════════════════════════════════════════╗
║ Kafka Metadata Layer Setup ║
╚════════════════════════════════════════════════════════════════════╝
Environment : user-sau-main-dev
Service : user
Zone : sau
Branch : main
Environment : dev
VM IP : 142.93.238.16
Metadata Mode : kraft
📋 KRaft Mode (Modern)
────────────────────────────────────────────────────────────────
✅ No ZooKeeper dependency
✅ Faster metadata operations
✅ Simplified architecture
✅ Recommended for new deployments
⚠️ Requires Kafka 3.3+ in production
════════════════════════════════════════════════════════════════════
[INFO] 🚀 Executing KRaft setup script...
[INFO] Script: /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps/metadata/kraft.sh
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-18 23:34:23 UTC] USER=www-data EUID=0 PID=4011769 ACTION=fsop ARGS=mkdir -p /data/kafka/user-sau-main-dev_coordinator-meta /opt/kafka/config/user-sau-main-dev/coordinator /data/kafka/user-sau-main-dev_coordinator-data
[2026-01-18 23:34:23 UTC] USER=www-data EUID=0 PID=4011778 ACTION=fsop ARGS=chown -R kafka:kafka /data/kafka/user-sau-main-dev_coordinator-meta /opt/kafka/config/user-sau-main-dev/coordinator /data/kafka/user-sau-main-dev_coordinator-data
[2026-01-18 23:34:23 UTC] USER=www-data EUID=0 PID=4011787 ACTION=fsop ARGS=chmod 770 /data/kafka/user-sau-main-dev_coordinator-meta /opt/kafka/config/user-sau-main-dev/coordinator /data/kafka/user-sau-main-dev_coordinator-data
[INFO] Adding eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com to /etc/hosts -> 10.100.1.141
[INFO] Adding eventbus-user-sau-main-dev-kafka-connect.fastorder.com to /etc/hosts -> 10.100.1.159
[INFO] Setting up KRaft for: user-sau-main-dev (host=eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com ip=10.100.1.141)
[2026-01-18 23:34:23 UTC] USER=www-data EUID=0 PID=4011811 ACTION=fsop ARGS=mkdir -p /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev
[INFO] Generated cluster.id=kykIA54TRlmOU90eJrW8zw
🔧 Configuring Kafka for KRaft mode...
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012266 ACTION=fsop ARGS=test -f /opt/kafka/config/user-sau-main-dev/coordinator/server.properties
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012276 ACTION=fsop ARGS=test -r /opt/kafka/config/user-sau-main-dev/coordinator/server.properties
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012285 ACTION=fsop ARGS=sed -i /^zookeeper\.connect=/d /opt/kafka/config/user-sau-main-dev/coordinator/server.properties
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012294 ACTION=passthru ARGS=bash -c grep -q '^process.roles=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties'
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012313 ACTION=passthru ARGS=bash -c grep -q '^node.id=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties'
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012333 ACTION=passthru ARGS=bash -c grep -q '^broker.id=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties'
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012348 ACTION=fsop ARGS=sed -i s|^broker.id=.*|broker.id=1| /opt/kafka/config/user-sau-main-dev/coordinator/server.properties
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012359 ACTION=passthru ARGS=bash -c grep -q '^controller.listener.names=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties'
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012379 ACTION=passthru ARGS=bash -c grep -q '^controller.quorum.voters=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties'
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012398 ACTION=passthru ARGS=bash -c grep -q '^metadata.log.dir=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties'
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012417 ACTION=passthru ARGS=bash -c grep -q '^log.dirs=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties'
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012426 ACTION=fsop ARGS=sed -i s|^log.dirs=.*|log.dirs=/data/kafka/user-sau-main-dev_coordinator-data| /opt/kafka/config/user-sau-main-dev/coordinator/server.properties
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012435 ACTION=passthru ARGS=bash -c grep -q '^listeners=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties'
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012444 ACTION=passthru ARGS=bash -c grep -q 'CONTROLLER://' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties'
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012453 ACTION=fsop ARGS=sed -i s|^listeners=.*|listeners=SSL://10.100.1.141:9092,CONTROLLER://10.100.1.141:9093| /opt/kafka/config/user-sau-main-dev/coordinator/server.properties
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012463 ACTION=passthru ARGS=bash -c grep -q '^advertised.listeners=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties'
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012472 ACTION=fsop ARGS=sed -i s|^advertised.listeners=.*|advertised.listeners=SSL://eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com:9092| /opt/kafka/config/user-sau-main-dev/coordinator/server.properties
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012481 ACTION=passthru ARGS=bash -c grep -q '^listener.security.protocol.map=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties'
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012490 ACTION=fsop ARGS=sed -i s|^listener.security.protocol.map=.*|listener.security.protocol.map=SSL:SSL,CONTROLLER:PLAINTEXT| /opt/kafka/config/user-sau-main-dev/coordinator/server.properties
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012499 ACTION=passthru ARGS=bash -c grep -q '^inter.broker.listener.name=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties'
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012508 ACTION=fsop ARGS=sed -i s|^inter.broker.listener.name=.*|inter.broker.listener.name=SSL| /opt/kafka/config/user-sau-main-dev/coordinator/server.properties
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012517 ACTION=passthru ARGS=bash -c grep -q '^offsets.topic.replication.factor=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties'
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012526 ACTION=fsop ARGS=sed -i s|^offsets.topic.replication.factor=.*|offsets.topic.replication.factor=1| /opt/kafka/config/user-sau-main-dev/coordinator/server.properties
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012535 ACTION=passthru ARGS=bash -c grep -q '^transaction.state.log.replication.factor=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties'
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012544 ACTION=fsop ARGS=sed -i s|^transaction.state.log.replication.factor=.*|transaction.state.log.replication.factor=1| /opt/kafka/config/user-sau-main-dev/coordinator/server.properties
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012553 ACTION=passthru ARGS=bash -c grep -q '^transaction.state.log.min.isr=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties'
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012562 ACTION=fsop ARGS=sed -i s|^transaction.state.log.min.isr=.*|transaction.state.log.min.isr=1| /opt/kafka/config/user-sau-main-dev/coordinator/server.properties
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012571 ACTION=passthru ARGS=bash -c grep -q '^min.insync.replicas=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties'
✅ KRaft configuration applied to server.properties
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012590 ACTION=fsop ARGS=test -f /data/kafka/user-sau-main-dev_coordinator-meta/meta.properties
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012599 ACTION=fsop ARGS=test -f /data/kafka/user-sau-main-dev_coordinator-data/meta.properties
[INFO] Already formatted: both /data/kafka/user-sau-main-dev_coordinator-meta and /data/kafka/user-sau-main-dev_coordinator-data have meta.properties
🔧 Creating/refreshing KRaft systemd unit...
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012619 ACTION=fsop ARGS=sed -i s|\\$MAINPID|$MAINPID|g /etc/systemd/system/confluent-kraft-user-sau-main-dev_coordinator.service
[2026-01-18 23:34:25 UTC] USER=www-data EUID=0 PID=4012636 ACTION=passthru ARGS=systemctl daemon-reload
✅ Ensured confluent-kraft-user-sau-main-dev_coordinator.service
🛑 Stopping legacy ZooKeeper-mode services and current KRaft instance...
🛑 Stopping current: confluent-kraft-user-sau-main-dev_coordinator.service
[2026-01-18 23:34:26 UTC] USER=www-data EUID=0 PID=4012683 ACTION=passthru ARGS=systemctl stop confluent-kraft-user-sau-main-dev_coordinator.service
🧹 Cleaning up rogue Kafka processes...
🧹 Killing any processes holding Kafka ports 9092, 9093...
🔪 Killing processes on port 9092: 3331893
3331894
3759563
[2026-01-18 23:34:30 UTC] USER=www-data EUID=0 PID=4012856 ACTION=passthru ARGS=bash -c kill -9 3331893
[2026-01-18 23:34:30 UTC] USER=www-data EUID=0 PID=4012865 ACTION=passthru ARGS=bash -c kill -9 3331894
🔪 Killing processes on port 9093: 3984532
3985970
[2026-01-18 23:34:32 UTC] USER=www-data EUID=0 PID=4012962 ACTION=passthru ARGS=bash -c kill -9 3984532
[2026-01-18 23:34:32 UTC] USER=www-data EUID=0 PID=4012973 ACTION=passthru ARGS=bash -c kill -9 3985970
🔪 Killing processes on port 8083: 3850931
[2026-01-18 23:34:34 UTC] USER=www-data EUID=0 PID=4013017 ACTION=passthru ARGS=bash -c kill -9 3850931
✅ Legacy services stopped and rogue processes cleaned
🔓 Removing stale lock files...
[2026-01-18 23:34:39 UTC] USER=www-data EUID=0 PID=4013203 ACTION=fsop ARGS=test -f /data/kafka/user-sau-main-dev_coordinator-meta/.lock
[2026-01-18 23:34:39 UTC] USER=www-data EUID=0 PID=4013212 ACTION=fsop ARGS=test -f /data/kafka/user-sau-main-dev_coordinator-data/.lock
✅ Lock file check complete
🚀 Starting confluent-kraft-user-sau-main-dev_coordinator.service ...
[2026-01-18 23:34:39 UTC] USER=www-data EUID=0 PID=4013221 ACTION=passthru ARGS=systemctl enable confluent-kraft-user-sau-main-dev_coordinator.service
[2026-01-18 23:34:40 UTC] USER=www-data EUID=0 PID=4013270 ACTION=passthru ARGS=systemctl restart confluent-kraft-user-sau-main-dev_coordinator.service
🔧 Patching shared Connect unit to follow KRaft broker...
[2026-01-18 23:34:43 UTC] USER=www-data EUID=0 PID=4013836 ACTION=fsop ARGS=sed -i -e s|${FULL_ENV}|user-sau-main-dev|g -e s|${IDENTIFIER}|coordinator|g -e s|${CONFIG_DIR}|/opt/kafka/config/user-sau-main-dev/coordinator|g /etc/systemd/system/confluent-connect-user-sau-main-dev_coordinator.service
[2026-01-18 23:34:43 UTC] USER=www-data EUID=0 PID=4013845 ACTION=fsop ARGS=sed -i s|\\$MAINPID|$MAINPID|g /etc/systemd/system/confluent-connect-user-sau-main-dev_coordinator.service
[2026-01-18 23:34:43 UTC] USER=www-data EUID=0 PID=4013857 ACTION=fsop ARGS=sed -i s|^After=.*|After=network-online.target confluent-kraft-user-sau-main-dev_coordinator.service| /etc/systemd/system/confluent-connect-user-sau-main-dev_coordinator.service
[2026-01-18 23:34:43 UTC] USER=www-data EUID=0 PID=4013868 ACTION=fsop ARGS=sed -i s|^Wants=.*|Wants=confluent-kraft-user-sau-main-dev_coordinator.service| /etc/systemd/system/confluent-connect-user-sau-main-dev_coordinator.service
[2026-01-18 23:34:43 UTC] USER=www-data EUID=0 PID=4013880 ACTION=fsop ARGS=sed -i s|^ExecStart=.*|ExecStart=/opt/kafka/bin/connect-distributed.sh /opt/kafka/config/user-sau-main-dev/coordinator/connect-distributed.properties| /etc/systemd/system/confluent-connect-user-sau-main-dev_coordinator.service
[2026-01-18 23:34:43 UTC] USER=www-data EUID=0 PID=4013892 ACTION=passthru ARGS=systemctl daemon-reload
✅ Connect unit patched
[2026-01-18 23:34:44 UTC] USER=www-data EUID=0 PID=4013946 ACTION=fsop ARGS=test -f /opt/kafka/config/user-sau-main-dev/coordinator/connect-distributed.properties
[2026-01-18 23:34:44 UTC] USER=www-data EUID=0 PID=4013960 ACTION=fsop ARGS=ln -sf /opt/kafka/config/user-sau-main-dev/coordinator/connect-distributed.properties /opt/kafka/config/connect-distributed.properties
⏳ Waiting for broker coordinator on SSL://eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com:9092 ...
⏳ Waiting for KRaft broker... (attempt 1, 0s/600s)
Debug: Last error was: [2026-01-18 23:34:44 UTC] USER=www-data EUID=0 PID=4013977 ACTION=passthru ARGS=bash -c timeout 5 sudo -u kafka /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server 'eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com:9092' --command-config '/etc/kafka/client-user-sau-main-dev-coordinator.properties' describe --status
[2026-01-18 23:34:48,030] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com/10.100.1.141:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2026-01-18 23:34:48,134] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com/10.100.1.141:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2026-01-18 23:34:48,236] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com/10.100.1.141:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2026-01-18 23:34:48,438] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com/10.100.1.141:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2026-01-18 23:34:48,942] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com/10.100.1.141:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
✅ coordinator responded after 45s (attempt 8)
---- server.properties (key lines) ----
[2026-01-18 23:35:51 UTC] USER=www-data EUID=0 PID=4021938 ACTION=passthru ARGS=bash -c grep -E '^(listeners|advertised\.listeners|process\.roles|controller\.quorum\.voters|controller\.listener\.names|inter\.broker\.listener\.name|log\.dirs|metadata\.log\.dir)=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties'
listeners=SSL://10.100.1.141:9092,CONTROLLER://10.100.1.141:9093
advertised.listeners=SSL://eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com:9092
inter.broker.listener.name=SSL
log.dirs=/data/kafka/user-sau-main-dev_coordinator-data
process.roles=broker,controller
controller.listener.names=CONTROLLER
controller.quorum.voters=1@10.100.1.141:9093
metadata.log.dir=/data/kafka/user-sau-main-dev_coordinator-meta
---------------------------------------
✅ KRaft setup complete for user-sau-main-dev_coordinator
server.properties : /opt/kafka/config/user-sau-main-dev/coordinator/server.properties
data dir : /data/kafka/user-sau-main-dev_coordinator-data
meta dir : /data/kafka/user-sau-main-dev_coordinator-meta
systemd unit : confluent-kraft-user-sau-main-dev_coordinator.service
🔧 Kafka Configuration Modified:
✓ process.roles, node.id, controller.quorum.voters, controller.listener.names
✓ listeners (SSL + CONTROLLER) and advertised.listeners (FQDN fallback to IP)
✓ listener.security.protocol.map, inter.broker.listener.name
✓ log.dirs -> /data/kafka/user-sau-main-dev_coordinator-data, metadata.log.dir -> /data/kafka/user-sau-main-dev_coordinator-meta
✓ removed zookeeper.connect (if present)
✓ created/refreshed dedicated KRaft systemd unit
✓ patched shared Connect unit to follow KRaft broker
✓ symlinked /opt/kafka/config/user-sau-main-dev/coordinator/connect-distributed.properties -> /opt/kafka/config/connect-distributed.properties (compat)
🔎 Check quorum:
/opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com:9092 --command-config /etc/kafka/client-user-sau-main-dev-coordinator.properties describe --status
📋 Next steps:
1) Review KRaft config: sudo grep -E 'process.roles|node.id|controller|listeners|advertised.listeners|log.dirs|metadata.log.dir' /opt/kafka/config/user-sau-main-dev/coordinator/server.properties
2) Verify topics: /opt/kafka/bin/kafka-topics.sh --bootstrap-server eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com:9092 --command-config /etc/kafka/client-user-sau-main-dev-coordinator.properties --list
✅ KRaft metadata layer setup completed successfully
Next steps:
1. Verify KRaft quorum status
2. Create Kafka topics
3. Configure Kafka Connect
[2026-01-18 23:35:51 UTC] USER=www-data EUID=0 PID=4021948 ACTION=fsop ARGS=mkdir -p /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev
[INFO] Saved metadata mode to: /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/kafka_metadata_mode
════════════════════════════════════════════════════════════════════
✅ Kafka Metadata Layer Setup Complete
Mode : kraft
Environment : user-sau-main-dev
State saved : /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/kafka_metadata_mode
KRaft cluster.id: kykIA54TRlmOU90eJrW8zw
Verify quorum:
kafka-metadata-quorum.sh --bootstrap-server ... describe
════════════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ Step 3 completed: 02-metadata.sh
[0;34m[INFO][0m 📦 Step 4/13: restart kafka related services...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-18 23:35:51 UTC] USER=www-data EUID=0 PID=4021995 ACTION=fsop ARGS=test -f /opt/kafka/config/user-sau-main-dev/coordinator/server.properties
[2026-01-18 23:35:51 UTC] USER=www-data EUID=0 PID=4022004 ACTION=passthru ARGS=bash -c grep -E '^[[:space:]]*process\.roles=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties' | grep -Eq '(broker|controller)'
[INFO] 📋 Detected mode from server.properties: kraft
[2026-01-18 23:35:52 UTC] USER=www-data EUID=0 PID=4022198 ACTION=passthru ARGS=systemctl stop confluent-connect-user-sau-main-dev_coordinator.service
[2026-01-18 23:35:57 UTC] USER=www-data EUID=0 PID=4022492 ACTION=passthru ARGS=systemctl stop confluent-kafka-zk-user-sau-main-dev_coordinator.service
[2026-01-18 23:35:58 UTC] USER=www-data EUID=0 PID=4022580 ACTION=passthru ARGS=systemctl stop confluent-zookeeper-user-sau-main-dev_coordinator.service
Failed to stop confluent-zookeeper-user-sau-main-dev_coordinator.service: Unit confluent-zookeeper-user-sau-main-dev_coordinator.service not loaded.
[INFO] 🧹 Removing stale Kafka lock files...
[2026-01-18 23:36:01 UTC] USER=www-data EUID=0 PID=4022649 ACTION=fsop ARGS=rm -f /var/lib/kafka/user-sau-main-dev_coordinator-meta/.lock
[2026-01-18 23:36:01 UTC] USER=www-data EUID=0 PID=4022658 ACTION=fsop ARGS=rm -f /var/lib/kafka/user-sau-main-dev_coordinator-data/.lock
[INFO] 🧹 Cleaning up orphaned processes on Kafka ports...
[2026-01-18 23:36:01 UTC] USER=www-data EUID=0 PID=4022667 ACTION=passthru ARGS=bash -c
for port in 9092 9093 8083 2181; do
pids=$(lsof -ti tcp:$port 2>/dev/null || true)
if [[ -n "$pids" ]]; then
echo " Killing orphaned processes on port $port: $pids"
kill -9 $pids 2>/dev/null || true
sleep 1
fi
done
Killing orphaned processes on port 9092: 4013268
4013285
4014424
4014425
Killing orphaned processes on port 9093: 4013809
🚀 Restarting Kafka components…
[INFO] 🚀 starting confluent-kraft-user-sau-main-dev_coordinator.service…
[2026-01-18 23:36:05 UTC] USER=www-data EUID=0 PID=4022825 ACTION=passthru ARGS=systemctl restart confluent-kraft-user-sau-main-dev_coordinator.service
[INFO] 🚀 starting confluent-connect-user-sau-main-dev_coordinator.service…
[2026-01-18 23:36:07 UTC] USER=www-data EUID=0 PID=4023391 ACTION=passthru ARGS=systemctl restart confluent-connect-user-sau-main-dev_coordinator.service
[INFO] ⏳ Waiting for Kafka broker readiness (FQDN: eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com, IP: 10.100.1.141) ...
[OK] ✅ Broker ready (attempt 1)
[OK] ✅ Port 9092 listening (Kafka Broker)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (1/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (2/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (3/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (4/40)
[INFO] ⏳ waiting for Kafka Connect REST port 8083 … (5/40)
[OK] ✅ Port 8083 listening (Kafka Connect REST)
[INFO] ⏳ Waiting for Connect REST at https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083 …
[OK] ✅ Connect REST is up (attempt 1)
📋 Reconciling Connect internal topics…
[ok] connect-configs exists
[ok] connect-offsets exists
[ok] connect-status exists
═══════════════════════════════════════════════════════════════════
KAFKA SUMMARY
═══════════════════════════════════════════════════════════════════
Env: user-sau-main-dev Identifier: coordinator Mode: kraft
Broker Unit : confluent-kraft-user-sau-main-dev_coordinator.service (status: active)
Connect Unit: confluent-connect-user-sau-main-dev_coordinator.service (status: active)
Bootstrap : eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com:9092
Connect URL : https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083
═══════════════════════════════════════════════════════════════════
[OK] ✅ All required services are up.
[0;32m[OK][0m ✅ Step 4 completed: 03-restart-kafka-related-services.sh
[0;34m[INFO][0m 📦 Step 5/13: checking services...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-18 23:36:54 UTC] USER=www-data EUID=0 PID=4027666 ACTION=fsop ARGS=test -f /opt/kafka/config/user-sau-main-dev/coordinator/server.properties
[2026-01-18 23:36:54 UTC] USER=www-data EUID=0 PID=4027675 ACTION=passthru ARGS=bash -c grep -E '^[[:space:]]*process\.roles=' '/opt/kafka/config/user-sau-main-dev/coordinator/server.properties' | grep -Eq '(broker|controller)'
[INFO] Detected mode from server.properties: kraft
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 1: Service status
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] confluent-kraft-user-sau-main-dev_coordinator.service status: active
[WARN] confluent-kafka-zk-user-sau-main-dev_coordinator.service present but should be stopped in KRaft
[WARN] confluent-zookeeper-user-sau-main-dev_coordinator.service present but not required in KRaft
[OK] confluent-connect-user-sau-main-dev_coordinator.service status: active
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 2: Port checks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ Port 9092 listening (Kafka Broker)
[OK] ✅ Port 8083 listening (Kafka Connect REST)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 3: Broker readiness
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] Broker API responding (attempt 1)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Step 4: Kafka Connect REST
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] Connect REST responding (attempt 1)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Kafka Services Summary
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment : user-sau-main-dev
Identifier : coordinator
Mode : kraft
Broker Unit : confluent-kraft-user-sau-main-dev_coordinator.service (status: active)
Connect Unit: confluent-connect-user-sau-main-dev_coordinator.service (status: active)
Broker FQDN : eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com:9092
Broker IP : eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com:9092
Connect URL : https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[OK] ✅ All required services are reachable.
[INFO] Creating ACLs for Kafka Connect consumer groups...
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
[INFO] Creating ACLs for Connect internal topics...
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
Error while executing ACL command: Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:474)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:100)
at kafka.admin.AclCommand$.main(AclCommand.scala:73)
at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:101)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:60)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:56)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
... 5 more
[OK] ✅ Kafka Connect ACLs configured (deny-by-default mode)
[0;32m[OK][0m ✅ Step 5 completed: 04-checking-services.sh
[0;34m[INFO][0m 📦 Step 6/13: create audit topic...
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Creating Kafka Audit Topics
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Replication Factor: 1
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 🔍 AUDIT READINESS GATE - Preflight Checks
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m [Gate 1/5] Verifying DNS resolution...
[0;32m[OK][0m ✅ Broker DNS: eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com -> 10.100.1.141
[0;32m[OK][0m ✅ Connect DNS: eventbus-user-sau-main-dev-kafka-connect.fastorder.com -> 10.100.1.159
[0;34m[INFO][0m [Gate 2/5] Verifying TLS handshake...
[0;32m[OK][0m ✅ TLS handshake: Broker certificate verified
[0;34m[INFO][0m [Gate 3/5] Verifying Kafka Connect REST API...
[0;32m[OK][0m ✅ Kafka Connect REST: Cluster ID = [2026-01-15 17:36:55 UTC] USER=www-data EUID=0 PID=455661 ACTION=passthru ARGS=bash -c cat /opt/fastorder/bash/scripts/env_app_setup/state/user-sau-main-dev/kafka_kraft_cluster_id
vGsJvzNtQGKG1HQPRIaTPQ
[0;34m[INFO][0m [Gate 4/5] Verifying required internal topics...
[0;32m[OK][0m ✅ Topic exists: connect-configs
[0;32m[OK][0m ✅ Topic exists: connect-offsets
[0;32m[OK][0m ✅ Topic exists: connect-status
[0;34m[INFO][0m [Gate 5/5] Verifying broker metadata access...
[0;32m[OK][0m ✅ Broker metadata: API versions accessible
[0;32m[OK][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ AUDIT READINESS GATE: ALL CHECKS PASSED
[0;32m[OK][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Waiting for Kafka to be ready...
[0;32m[OK][0m ✅ Kafka is ready
[0;34m[INFO][0m Creating audit topic: audit.events.user.sau.main.dev
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Error while executing topic command : Topic 'audit.events.user.sau.main.dev' already exists.
[2026-01-18 23:37:29,070] ERROR org.apache.kafka.common.errors.TopicExistsException: Topic 'audit.events.user.sau.main.dev' already exists.
(kafka.admin.TopicCommand$)
[0;32m[OK][0m ✅ Audit topic already exists: audit.events.user.sau.main.dev
[0;32m[OK][0m ✅ Topic verified: audit.events.user.sau.main.dev
Topic: audit.events.user.sau.main.dev TopicId: vYh4z_jEQ1a-0MR4arL3dA PartitionCount: 3 ReplicationFactor: 1 Configs: compression.type=lz4,min.insync.replicas=1,cleanup.policy=delete,segment.bytes=1073741824,retention.ms=7776000000,message.timestamp.type=LogAppendTime,segment.ms=604800000
Topic: audit.events.user.sau.main.dev Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: audit.events.user.sau.main.dev Partition: 1 Leader: 1 Replicas: 1 Isr: 1
Topic: audit.events.user.sau.main.dev Partition: 2 Leader: 1 Replicas: 1 Isr: 1
[0;34m[INFO][0m Creating audit producer credentials...
Completed updating config for user audit-producer-user-sau-main-dev.
[0;32m[OK][0m ✅ Audit producer user created: audit-producer-user-sau-main-dev
[0;34m[INFO][0m Creating ACLs for audit producer...
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.user.sau.main.dev, patternType=LITERAL)`:
(principal=User:audit-producer-user-sau-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.user.sau.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:audit-producer-user-sau-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:audit-producer-user-sau-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.user.sau.main.dev, patternType=LITERAL)`:
(principal=User:audit-producer-user-sau-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.user.sau.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:audit-producer-user-sau-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:audit-producer-user-sau-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.user.sau.main.dev, patternType=LITERAL)`:
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.user.sau.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:audit-producer-user-sau-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:audit-producer-user-sau-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.user.sau.main.dev, patternType=LITERAL)`:
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.user.sau.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:audit-producer-user-sau-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:audit-producer-user-sau-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.user.sau.main.dev, patternType=LITERAL)`:
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.user.sau.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:audit-producer-user-sau-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:audit-producer-user-sau-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.user.sau.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=audit.events.user.sau.main.dev, patternType=LITERAL)`:
(principal=User:*, host=*, operation=ALTER, permissionType=DENY)
(principal=User:audit-producer-user-sau-main-dev, host=*, operation=WRITE, permissionType=ALLOW)
(principal=User:audit-producer-user-sau-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:*, host=*, operation=DELETE, permissionType=DENY)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=READ, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
(principal=User:C=AE,O=FastOrder,OU=Kafka Client,CN=kafka-client-user-sau-main-dev, host=*, operation=READ, permissionType=ALLOW)
(principal=User:CN=kafka-client-user-sau-main-dev,OU=Kafka Client,O=FastOrder,C=AE, host=*, operation=DESCRIBE, permissionType=ALLOW)
[0;32m[OK][0m ✅ ACLs configured (producer: write-only, sinks: read-only, immutability: protected)
[0;34m[INFO][0m Storing audit producer credentials in AWS Secrets Manager...
{
"ARN": "arn:aws:secretsmanager:me-central-1:464621692046:secret:fastorder/eventbus/user/sau/main/dev/kafka/audit/producer-jFPb1p",
"Name": "fastorder/eventbus/user/sau/main/dev/kafka/audit/producer",
"VersionId": "b6fe91ac-3166-49c0-a83f-fdf1f0ca6a86"
}
[0;32m[OK][0m ✅ Credentials stored in: fastorder/eventbus/user/sau/main/dev/kafka/audit/producer
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Configuring S3 Sink for Audit Cold Storage
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
PRE audit/
[0;34m[INFO][0m S3 staging bucket exists: fastorder-audit-staging
[0;34m[INFO][0m Data flow: Kafka → fastorder-audit-staging → (replication) → fastorder-audit-immutable
[0;34m[INFO][0m Updating existing S3 sink connector...
{"name":"audit-s3-sink-user_sau_main_dev","config":{"connector.class":"io.confluent.connect.s3.S3SinkConnector","tasks.max":"1","topics":"audit.events.user.sau.main.dev","topics.dir":"audit/user-sau-main-dev","s3.bucket.name":"fastorder-audit-staging","s3.region":"me-central-1","s3.part.size":"5242880","flush.size":"1000","rotate.interval.ms":"3600000","rotate.schedule.interval.ms":"86400000","storage.class":"io.confluent.connect.s3.storage.S3Storage","format.class":"io.confluent.connect.s3.format.json.JsonFormat","partitioner.class":"io.confluent.connect.storage.partitioner.TimeBasedPartitioner","path.format":"'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH","partition.duration.ms":"3600000","locale":"en-US","timezone":"UTC","timestamp.extractor":"Record","key.converter":"org.apache.kafka.connect.json.JsonConverter","value.converter":"org.apache.kafka.connect.json.JsonConverter","key.converter.schemas.enable":"false","value.converter.schemas.enable":"false","behavior.on.null.values":"ignore","errors.tolerance":"all","errors.log.enable":"true","errors.log.include.messages":"true","name":"audit-s3-sink-user_sau_main_dev"},"tasks":[{"connector":"audit-s3-sink-user_sau_main_dev","task":0}],"type":"sink"}[0;32m[OK][0m ✅ S3 Sink connector configured for audit cold storage
[0;34m[INFO][0m Staging Bucket: fastorder-audit-staging (Kafka Connect writes here)
[0;34m[INFO][0m Immutable Bucket: fastorder-audit-immutable (via S3 Replication)
[0;34m[INFO][0m Path: audit/user-sau-main-dev/
[0;34m[INFO][0m Final Retention: WORM-enabled (Object Lock COMPLIANCE mode, 1-year)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Kafka Audit Topic Created Successfully
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Topic: audit.events.user.sau.main.dev
[0;34m[INFO][0m Partitions: 3
[0;34m[INFO][0m Replication Factor: 1
[0;34m[INFO][0m Retention: 90 days
[0;34m[INFO][0m Producer: audit-producer-user-sau-main-dev (write-only)
[0;34m[INFO][0m Application Integration:
[0;34m[INFO][0m - Use credentials from: fastorder/eventbus/user/sau/main/dev/kafka/audit/producer
[0;34m[INFO][0m - Connect to: eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com:9092
[0;34m[INFO][0m - Produce to: audit.events.user.sau.main.dev
[0;34m[INFO][0m - Security: SASL_SSL (SCRAM-SHA-512)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 📋 PCI-DSS Compliance Status
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m
[0;34m[INFO][0m ✅ Kafka Hot Storage: 90 days (meets PCI-DSS 3-month immediate access)
[0;34m[INFO][0m ✅ ACL Authorization: deny-by-default (allow.everyone.if.no.acl.found=false)
[0;34m[INFO][0m ✅ Immutability: DENY DELETE/ALTER on audit topic
[0;34m[INFO][0m ✅ S3 Cold Storage: fastorder-audit-immutable (Object Lock COMPLIANCE, 1-year)
[0;34m[INFO][0m
[0;34m[INFO][0m S3 Audit Storage:
[0;34m[INFO][0m Bucket: s3://fastorder-audit-staging
[0;34m[INFO][0m Path: audit/user-sau-main-dev/
[0;34m[INFO][0m Object Lock: COMPLIANCE mode, 1-year retention
[0;34m[INFO][0m Immutability: Objects cannot be deleted or modified for 1 year
[0;34m[INFO][0m
[0;34m[INFO][0m Verify compliance with:
[0;34m[INFO][0m bash 04-eventbus/engine/kafka/steps/11-audit-compliance-check.sh
[0;34m[INFO][0m
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 📋 AWS Roles Anywhere - Credential Refresh Setup
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m
[0;34m[INFO][0m For S3 sink to write to S3, Kafka Connect needs AWS credentials.
[0;34m[INFO][0m Use IAM Roles Anywhere with systemd timer for automatic refresh.
[0;34m[INFO][0m
[0;34m[INFO][0m Required files:
[0;34m[INFO][0m Certificate: /etc/fastorder/rolesanywhere/client-bundle.crt
[0;34m[INFO][0m Private Key: /etc/fastorder/rolesanywhere/client.key
[0;34m[INFO][0m Helper: /usr/local/bin/aws_signing_helper
[0;34m[INFO][0m
[0;34m[INFO][0m Systemd timer: kafka-aws-credential-refresh.timer
[0;34m[INFO][0m Runs every 30 minutes to refresh credentials to /var/lib/kafka/.aws/credentials
[0;34m[INFO][0m
[0;34m[INFO][0m Verify timer is active:
[0;34m[INFO][0m systemctl status kafka-aws-credential-refresh.timer
[0;34m[INFO][0m
[0;34m[INFO][0m Documentation: https://docs.aws.amazon.com/rolesanywhere/latest/userguide/
[0;32m[OK][0m ✅ Step 6 completed: 05-create-audit-topic.sh
[0;34m[INFO][0m 📦 Step 7/13: setup backups...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Kafka Backup Configuration
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Environment: user-sau-main-dev
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /home/ab/.aws/credentials
[0;34m[INFO][0m 1️⃣ Creating S3 bucket for Kafka backups...
make_bucket failed: s3://fastorder-kafka-backups-user-sau-main-dev An error occurred (AccessDenied) when calling the CreateBucket operation: User: arn:aws:iam::464621692046:user/fo-dev is not authorized to perform: s3:CreateBucket on resource: "arn:aws:s3:::fastorder-kafka-backups-user-sau-main-dev" because no identity-based policy allows the s3:CreateBucket action
An error occurred (NoSuchBucket) when calling the PutBucketVersioning operation: The specified bucket does not exist
Parameter validation failed:
Unknown parameter in LifecycleConfiguration.Rules[0]: "Id", must be one of: Expiration, ID, Prefix, Filter, Status, Transitions, NoncurrentVersionTransitions, NoncurrentVersionExpiration, AbortIncompleteMultipartUpload
[0;32m[OK][0m ✅ S3 bucket created: fastorder-kafka-backups-user-sau-main-dev
[0;34m[INFO][0m 2️⃣ Creating local backup directory...
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4030955 ACTION=fsop ARGS=mkdir -p /var/backups/kafka/user-sau-main-dev
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4030976 ACTION=fsop ARGS=mkdir -p /var/backups/kafka/user-sau-main-dev/topics
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4030986 ACTION=fsop ARGS=mkdir -p /var/backups/kafka/user-sau-main-dev/metadata
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4030995 ACTION=fsop ARGS=mkdir -p /var/log/kafka/backups
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4031004 ACTION=fsop ARGS=chown -R kafka:kafka /var/backups/kafka/user-sau-main-dev
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4031013 ACTION=fsop ARGS=chown -R kafka:kafka /var/log/kafka/backups
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4031022 ACTION=fsop ARGS=chmod 750 /var/backups/kafka/user-sau-main-dev
[0;32m[OK][0m ✅ Local backup directory created
[0;34m[INFO][0m 3️⃣ Creating topic backup script...
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4031040 ACTION=fsop ARGS=sed -i s|__ENV_ID__|user-sau-main-dev|g /usr/local/bin/kafka-backup-user-sau-main-dev.sh
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4031049 ACTION=fsop ARGS=sed -i s|__KAFKA_BROKER__|eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com:9092|g /usr/local/bin/kafka-backup-user-sau-main-dev.sh
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4031058 ACTION=fsop ARGS=sed -i s|__BACKUP_DIR__|/var/backups/kafka/user-sau-main-dev|g /usr/local/bin/kafka-backup-user-sau-main-dev.sh
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4031067 ACTION=fsop ARGS=sed -i s|__S3_BUCKET__|fastorder-kafka-backups-user-sau-main-dev|g /usr/local/bin/kafka-backup-user-sau-main-dev.sh
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4031076 ACTION=fsop ARGS=sed -i s|__S3_REGION__|me-central-1|g /usr/local/bin/kafka-backup-user-sau-main-dev.sh
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4031085 ACTION=fsop ARGS=chmod 750 /usr/local/bin/kafka-backup-user-sau-main-dev.sh
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4031094 ACTION=fsop ARGS=chown root:kafka /usr/local/bin/kafka-backup-user-sau-main-dev.sh
[0;32m[OK][0m ✅ Backup script created: /usr/local/bin/kafka-backup-user-sau-main-dev.sh
[0;34m[INFO][0m 4️⃣ Setting up cron jobs for automated backups...
[0;32m[OK][0m ✅ Cron job configured: Daily backups at 2:00 AM
[0;34m[INFO][0m 5️⃣ Creating restore documentation...
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4031131 ACTION=fsop ARGS=sed -i s|__S3_BUCKET__|fastorder-kafka-backups-user-sau-main-dev|g /var/backups/kafka/user-sau-main-dev/RESTORE_INSTRUCTIONS.md
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4031149 ACTION=fsop ARGS=sed -i s|__KAFKA_BROKER__|eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com|g /var/backups/kafka/user-sau-main-dev/RESTORE_INSTRUCTIONS.md
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4031158 ACTION=fsop ARGS=chmod 644 /var/backups/kafka/user-sau-main-dev/RESTORE_INSTRUCTIONS.md
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4031167 ACTION=fsop ARGS=chown kafka:kafka /var/backups/kafka/user-sau-main-dev/RESTORE_INSTRUCTIONS.md
[0;32m[OK][0m ✅ Restore documentation created: /var/backups/kafka/user-sau-main-dev/RESTORE_INSTRUCTIONS.md
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Kafka Backup Configured
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m S3 Bucket: fastorder-kafka-backups-user-sau-main-dev
[0;34m[INFO][0m Region: me-central-1
[0;34m[INFO][0m Local backup dir: /var/backups/kafka/user-sau-main-dev
[0;34m[INFO][0m Schedule: Daily at 2:00 AM
[0;34m[INFO][0m Script: /usr/local/bin/kafka-backup-user-sau-main-dev.sh
[0;34m[INFO][0m Restore docs: /var/backups/kafka/user-sau-main-dev/RESTORE_INSTRUCTIONS.md
[1;33m[WARN][0m ⚠️ Note: This backs up Kafka metadata only (topics, configs, offsets)
[1;33m[WARN][0m For full message data backup, configure Kafka Connect S3 Sink
[0;32m[OK][0m ✅ Step 7 completed: 06-setup-backups.sh
[0;34m[INFO][0m 📦 Step 8/13: monitoring setup...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 🔍 Kafka Monitoring Integration for user-sau-main-dev
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] 1️⃣ Checking observability cell readiness...
[INFO] Checking observability cell readiness: obs-user-sau-main-dev
[1;32m[OK][0m Observability cell endpoints registered for user-sau-main-dev
[1;32m[OK][0m ✓ Observability cell is ready
[INFO] 2️⃣ Setting up Kafka JMX exporter integration...
[INFO] JMX Exporter port calculated for user-sau-main-dev: 9357 (offset: 49)
[INFO] Checking observability cell readiness: obs-user-sau-main-dev
[1;32m[OK][0m Observability cell endpoints registered for user-sau-main-dev
[INFO] Setting up Kafka JMX exporter for user-sau-main-dev
[INFO] JMX Prometheus Java Agent already exists at /opt/kafka/libs/jmx_prometheus_javaagent.jar
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4031210 ACTION=passthru ARGS=mv /tmp/jmx_exporter.yml /opt/kafka/config/jmx_exporter.yml
[2026-01-18 23:38:02 UTC] USER=www-data EUID=0 PID=4031219 ACTION=passthru ARGS=chmod 644 /opt/kafka/config/jmx_exporter.yml
[1;32m[OK][0m JMX exporter configuration created at /opt/kafka/config/jmx_exporter.yml
[1;32m[OK][0m JMX exporter configuration created
[INFO] Configuring Kafka systemd services to use JMX exporter...
[2026-01-18 23:38:03 UTC] USER=www-data EUID=0 PID=4031243 ACTION=fsop ARGS=test -f /etc/systemd/system/[2026-01-18
[INFO] All Kafka services already configured with JMX exporter
[1;32m[OK][0m Kafka JMX exporter integration complete
[INFO] Metrics endpoint: http://142.93.238.16:9357/metrics
[INFO] Prometheus will automatically scrape: https://metrics-user-sau-main-dev.fastorder.com:9090
[INFO] View dashboards at: https://dashboards-user-sau-main-dev.fastorder.com
[1;32m[OK][0m ✓ Kafka JMX exporter integration complete
[INFO] Configuring KAFKA_OPTS environment variable for kafka user...
[2026-01-18 23:38:03 UTC] USER=www-data EUID=0 PID=4031264 ACTION=passthru ARGS=grep -q KAFKA_OPTS.*javaagent.*jmx_prometheus_javaagent.*=9357: /home/kafka/.bashrc
[2026-01-18 23:38:03 UTC] USER=www-data EUID=0 PID=4031285 ACTION=passthru ARGS=sed -i /export KAFKA_OPTS=.*jmx_prometheus_javaagent/d /home/kafka/.bashrc
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[1;32m[OK][0m ✓ KAFKA_OPTS configured in /home/kafka/.bashrc (port 9357)
[INFO] 2.5️⃣ Enabling JMX exporter in Kafka systemd service...
[2026-01-18 23:38:03 UTC] USER=www-data EUID=0 PID=4031330 ACTION=passthru ARGS=grep -q javaagent.*jmx_prometheus_javaagent /etc/systemd/system/confluent-kraft-user-sau-main-dev_coordinator.service
[INFO] Updating confluent-kraft-user-sau-main-dev_coordinator.service to enable JMX exporter...
[1;32m[OK][0m ✓ Updated confluent-kraft-user-sau-main-dev_coordinator.service
[INFO] Reloading systemd daemon and restarting Kafka services...
[2026-01-18 23:38:03 UTC] USER=www-data EUID=0 PID=4031381 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:38:03 UTC] USER=www-data EUID=0 PID=4031439 ACTION=passthru ARGS=systemctl is-active --quiet confluent-kraft-user-sau-main-dev_coordinator
[INFO] Restarting confluent-kraft-user-sau-main-dev_coordinator...
[2026-01-18 23:38:04 UTC] USER=www-data EUID=0 PID=4031460 ACTION=passthru ARGS=systemctl restart confluent-kraft-user-sau-main-dev_coordinator
[2026-01-18 23:38:08 UTC] USER=www-data EUID=0 PID=4031964 ACTION=passthru ARGS=systemctl is-active --quiet confluent-kraft-user-sau-main-dev_coordinator
[1;32m[OK][0m ✓ confluent-kraft-user-sau-main-dev_coordinator restarted successfully
[1;32m[OK][0m ✓ JMX exporter enabled in Kafka systemd services
[INFO] 2.6️⃣ Configuring Prometheus to scrape Kafka metrics...
[2026-01-18 23:38:08 UTC] USER=www-data EUID=0 PID=4031986 ACTION=passthru ARGS=grep -q job_name: 'kafka' /etc/prometheus/obs-user-sau-main-dev/prometheus.yml
[INFO] Adding Kafka scrape target to Prometheus configuration...
[ERROR] No passwordless sudo and wrapper does not allow 'bash'. Run as root or extend wrapper.
[2026-01-18 23:38:08 UTC] USER=www-data EUID=0 PID=4032019 ACTION=passthru ARGS=sed -i /# Prometheus self-monitoring/r /tmp/prometheus_kafka_add.yml /etc/prometheus/obs-user-sau-main-dev/prometheus.yml
[ERROR] Invalid Prometheus configuration - rolling back
[2026-01-18 23:38:08 UTC] USER=www-data EUID=0 PID=4032053 ACTION=passthru ARGS=sed -i /job_name: 'kafka'/,+6d /etc/prometheus/obs-user-sau-main-dev/prometheus.yml
[2026-01-18 23:38:09 UTC] USER=www-data EUID=0 PID=4032075 ACTION=fsop ARGS=rm -f /tmp/prometheus_kafka_add.yml
[INFO] 3️⃣ Registering Kafka nodes to monitoring database...
[INFO] Detected Kafka version: 3.9.1
[INFO] Registering Kafka Broker to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Kafka Broker
[INFO] Identifier: user-sau-main-dev-broker-01
[INFO] Identifier Parent: cluster
[INFO] IP: 142.93.238.16
[INFO] Port: 9092
[INFO] FQDN: eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com
[INFO] Status: running
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: f13110f1-e44a-4c54-a00b-2cf550959aef
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[1;32m[OK][0m ✓ Kafka broker registered
[INFO] Registering Kafka Connect to monitoring dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Kafka Connect
[INFO] Identifier: user-sau-main-dev-connect-01
[INFO] Identifier Parent: cluster
[INFO] IP: 142.93.238.16
[INFO] Port: 8083
[INFO] FQDN: eventbus-user-sau-main-dev-kafka-connect.fastorder.com
[INFO] Status: running
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 44032700-39d0-4bd0-aa84-ed929b0f2345
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[1;32m[OK][0m ✓ Kafka Connect registered
[INFO] Schema Registry not running, skipping registration
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Kafka Monitoring Setup Complete
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Metrics: http://localhost:9357/metrics
[INFO] Prometheus: https://metrics-user-sau-main-dev.fastorder.com:9090
[INFO] Grafana: https://dashboards-user-sau-main-dev.fastorder.com
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Step 8 completed: 10-monitoring-setup.sh
[0;34m[INFO][0m 📦 Step 9/13: audit compliance check...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[1m KAFKA AUDIT COMPLIANCE DASHBOARD - PCI-DSS Verification[0m
[1m Environment: user-sau-main-dev[0m
[1m Timestamp: 2026-01-18 23:38:12 UTC[0m
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[1m[1/5] Kafka Deny-by-Default ACL Posture[0m
Requirement: allow.everyone.if.no.acl.found=false
[0;32mPASS[0m Deny-by-default is ENABLED (allow.everyone.if.no.acl.found=false)
[1m[2/5] Audit Topic Hot Retention (90 days)[0m
Requirement: retention.ms >= 7776000000 (90 days)
[0;32mPASS[0m Retention is 90 days (7776000000 ms)
[1m[3/5] Kafka Connect S3 Sink Status[0m
Requirement: Connector and all tasks RUNNING
[0;31mFAIL[0m Connector state: UNASSIGNED (expected RUNNING)
[1m[4/5] S3 Freshness Evidence[0m
Requirement: Newest object < 120 minutes old
[1;33mWARN[0m No objects found in s3://fastorder-audit-immutable/audit/user-sau-main-dev/
This may be normal if no audit events have been generated yet
[1m[5/5] S3 Object Lock Immutability[0m
Requirement: COMPLIANCE mode with 1-year retention
[0;31mFAIL[0m Cannot verify Object Lock configuration - access denied
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[1m COMPLIANCE SUMMARY[0m
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;31m[1mCOMPLIANCE ISSUES DETECTED[0m
Passed: 2/5
Failed: 3/5
Review failed checks above and remediate.
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[0;32m[OK][0m ✅ Step 9 completed: 11-audit-compliance-check.sh
[0;34m[INFO][0m 📦 Step 10/13: audit canary test...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[1m KAFKA AUDIT CANARY TEST - End-to-End Verification[0m
[1m Environment: user-sau-main-dev[0m
[1m Canary ID: canary-1768779504-4033368[0m
[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m
[1m[Step 1/4] Producing canary audit event to Kafka[0m
Topic: audit.events.user.sau.main.dev
Event: canary-1768779504-4033368
[0;31mFailed to produce event[0m
[1;33m(Topic may not exist yet - normal during initial setup)[0m
[0;32m[OK][0m ✅ Step 10 completed: 12-audit-canary-test.sh
[0;34m[INFO][0m 📦 Step 11/13: setup audit s3 staging...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
════════════════════════════════════════════════════════════════════════
Kafka Audit S3 Staging + Replication Setup
════════════════════════════════════════════════════════════════════════
Staging Bucket: fastorder-audit-staging
Immutable Bucket: fastorder-audit-immutable
Region: me-central-1
Environment: --auto
════════════════════════════════════════════════════════════════════════
[INFO] 1️⃣ Checking AWS credentials...
[WARN] No AWS credentials found - skipping S3 staging setup
To configure S3 audit storage later, add credentials to /home/ab/.aws/credentials:
[admin]
aws_access_key_id = AKIA...
aws_secret_access_key = ...
Then run: AWS_PROFILE=admin /opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps/13-setup-audit-s3-staging.sh --auto
[0;32m[OK][0m ✅ Step 11 completed: 13-setup-audit-s3-staging.sh
[0;34m[INFO][0m 📦 Step 12/13: install ksqldb...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ksqlDB Installation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Identifier: --auto
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Allocating new VM_IP for ksqlDB: 10.100.1.230
[2026-01-18 23:38:26 UTC] USER=www-data EUID=0 PID=4033894 ACTION=fsop ARGS=tee -a /etc/hosts
10.100.1.230 eventbus-user-sau-main-dev-ksqldb---auto.fastorder.com
VM_IP: 10.100.1.230
FQDN: eventbus-user-sau-main-dev-ksqldb---auto.fastorder.com
📦 Step 1: Checking Confluent Platform installation...
✅ ksqlDB already installed (version: )
📁 Step 2: Creating directories...
[2026-01-18 23:38:28 UTC] USER=www-data EUID=0 PID=4034050 ACTION=fsop ARGS=mkdir -p /var/lib/ksqldb/user-sau-main-dev/--auto
[2026-01-18 23:38:29 UTC] USER=www-data EUID=0 PID=4034078 ACTION=fsop ARGS=mkdir -p /var/log/ksqldb/user-sau-main-dev/--auto
[2026-01-18 23:38:29 UTC] USER=www-data EUID=0 PID=4034099 ACTION=fsop ARGS=mkdir -p /etc/ksqldb/user-sau-main-dev/--auto
[2026-01-18 23:38:29 UTC] USER=www-data EUID=0 PID=4034120 ACTION=fsop ARGS=chown -R kafka:kafka /var/lib/ksqldb/user-sau-main-dev/--auto /var/log/ksqldb/user-sau-main-dev/--auto /etc/ksqldb/user-sau-main-dev/--auto
✅ Directories created
⚙️ Step 3: Generating ksqlDB configuration...
[2026-01-18 23:38:29 UTC] USER=www-data EUID=0 PID=4034164 ACTION=fsop ARGS=chown kafka:kafka /etc/ksqldb/user-sau-main-dev/--auto/ksql-server.properties
[2026-01-18 23:38:29 UTC] USER=www-data EUID=0 PID=4034186 ACTION=fsop ARGS=chmod 640 /etc/ksqldb/user-sau-main-dev/--auto/ksql-server.properties
✅ Configuration generated: /etc/ksqldb/user-sau-main-dev/--auto/ksql-server.properties
🔧 Step 4: Creating systemd service...
[2026-01-18 23:38:29 UTC] USER=www-data EUID=0 PID=4034208 ACTION=fsop ARGS=mv /tmp/ksqldb-user-sau-main-dev---auto.service /etc/systemd/system/ksqldb-user-sau-main-dev---auto.service
[2026-01-18 23:38:29 UTC] USER=www-data EUID=0 PID=4034229 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:38:30 UTC] USER=www-data EUID=0 PID=4034294 ACTION=passthru ARGS=systemctl enable ksqldb-user-sau-main-dev---auto.service
✅ Systemd service created: ksqldb-user-sau-main-dev---auto.service
🚀 Step 5: Starting ksqlDB service...
🔍 Checking Kafka broker connectivity...
✅ Kafka broker is accessible
[2026-01-18 23:38:30 UTC] USER=www-data EUID=0 PID=4034377 ACTION=passthru ARGS=systemctl start ksqldb-user-sau-main-dev---auto.service
✅ ksqlDB service started
⏳ Waiting for ksqlDB to be ready...
..............................
🔍 Step 6: Verifying installation...
📊 Service Status:
[2026-01-18 23:39:41 UTC] USER=www-data EUID=0 PID=4036244 ACTION=passthru ARGS=systemctl status ksqldb-user-sau-main-dev---auto.service --no-pager -l
● ksqldb-user-sau-main-dev---auto.service - ksqlDB Server (user-sau-main-dev --auto)
Loaded: loaded (/etc/systemd/system/ksqldb-user-sau-main-dev---auto.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Sun 2026-01-18 23:39:37 UTC; 4s ago
Docs: https://docs.ksqldb.io/
Process: 4035962 ExecStart=/usr/bin/ksql-server-start /etc/ksqldb/user-sau-main-dev/--auto/ksql-server.properties (code=exited, status=255/EXCEPTION)
Main PID: 4035962 (code=exited, status=255/EXCEPTION)
CPU: 4.798s
📊 ksqlDB Info:
⚠️ ksqlDB not responding yet (may still be starting)
📡 Step 7: Registering ksqlDB to Observability API...
🔄 Registering ksqlDB node to observability dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: ksqlDB
[INFO] Identifier: user-sau-main-dev-ksqldb---auto
[INFO] Identifier Parent: eventbus
[INFO] IP: 10.100.1.230
[INFO] Port: 8088
[INFO] FQDN: eventbus-user-sau-main-dev-ksqldb---auto.fastorder.com
[INFO] Status: starting
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[WARN] Registration API call failed (HTTP 500), retrying (1/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[22P02]: Invalid text representation: 7 ERROR: invalid input value for enum obs.instance_status: \"starting\"\nCONTEXT: unnamed portal parameter $8 = '...'"}
[WARN] Registration API call failed (HTTP 500), retrying (2/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[22P02]: Invalid text representation: 7 ERROR: invalid input value for enum obs.instance_status: \"starting\"\nCONTEXT: unnamed portal parameter $8 = '...'"}
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ REGISTRATION FAILED AFTER 3 ATTEMPTS
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] HTTP Status: 500
[ERROR] Response: {"success":false,"error":"Registration failed: SQLSTATE[22P02]: Invalid text representation: 7 ERROR: invalid input value for enum obs.instance_status: \"starting\"\nCONTEXT: unnamed portal parameter $8 = '...'"}
[ERROR]
[ERROR] API endpoint: https://skeleton.dev.fastorder.com/api/obs/register
[ERROR]
[ERROR] Troubleshooting:
[ERROR] 1. Check if skeleton.dev.fastorder.com is accessible
[ERROR] 2. Verify web application is running
[ERROR] 3. Check web application logs: /var/www/html/skeleton.dev.fastorder.com/logs/
[ERROR] 4. Test API manually:
[ERROR] curl -k -X POST 'https://skeleton.dev.fastorder.com/api/obs/register' \
[ERROR] -H 'Content-Type: application/json' \
[ERROR] -H 'X-Internal-Token: $OBS_INTERNAL_API_TOKEN' \
[ERROR] -d '$PAYLOAD'
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
⚠️ Failed to register ksqlDB (non-fatal)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ksqlDB Installation Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Service: ksqldb-user-sau-main-dev---auto
VM_IP: 10.100.1.230
FQDN: eventbus-user-sau-main-dev-ksqldb---auto.fastorder.com
Port: 8088
Config: /etc/ksqldb/user-sau-main-dev/--auto/ksql-server.properties
Data: /var/lib/ksqldb/user-sau-main-dev/--auto
Logs: /var/log/ksqldb/user-sau-main-dev/--auto
Dashboard:
https://skeleton.dev.fastorder.com/dashboard/monitoring/environment2/<env-id>/service/ksqldb
CLI Access (with SSL):
ksql --ssl https://eventbus-user-sau-main-dev-ksqldb---auto.fastorder.com:8088
REST API (HTTPS):
curl -k https://eventbus-user-sau-main-dev-ksqldb---auto.fastorder.com:8088/info
curl -k https://eventbus-user-sau-main-dev-ksqldb---auto.fastorder.com:8088/ksql -H 'Content-Type: application/vnd.ksql.v1+json' -d '{"ksql": "SHOW STREAMS;"}'
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Step 12 completed: 20-install-ksqldb.sh
[0;34m[INFO][0m 📦 Step 13/13: update www data certs...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
📋 Post-Kafka Setup: Updating www-data Kafka certificates...
Environment: user-sau-main-dev
Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/user-sau-main-dev
✓ Kafka certificates found
✓ www-data user exists
[2026-01-18 23:39:47 UTC] USER=www-data EUID=0 PID=4036403 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:39:47 UTC] USER=www-data EUID=0 PID=4036415 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:39:47 UTC] USER=www-data EUID=0 PID=4036438 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-01-18 23:39:47 UTC] USER=www-data EUID=0 PID=4036455 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-01-18 23:39:47 UTC] USER=www-data EUID=0 PID=4036470 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-01-18 23:39:47 UTC] USER=www-data EUID=0 PID=4036497 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:39:47 UTC] USER=www-data EUID=0 PID=4036507 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:39:47 UTC] USER=www-data EUID=0 PID=4036516 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks created for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✓ Post-Kafka setup complete
[0;32m[OK][0m ✅ Step 13 completed: 99-update-www-data-certs.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m ✅ Kafka setup completed successfully!
[0;32m[OK][0m Executed all 13 steps
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Service: user
[0;34m[INFO][0m Zone: sau
[0;34m[INFO][0m Branch: main
[0;34m[INFO][0m Env: dev
[0;34m[INFO][0m Registering Kafka nodes via API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Kafka
[INFO] Identifier: user-sau-main-dev_coordinator
[INFO] Identifier Parent: coordinator
[INFO] IP: 10.100.1.141
[INFO] Port: 9092
[INFO] FQDN: eventbus-user-sau-main-dev-kafka-broker-01.fastorder.com
[INFO] Status: running
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[WARN] Registration API call failed (HTTP 500), retrying (1/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint \"application_kind_key\"\nDETAIL: Key (kind)=(kafka) already exists."}
[WARN] Registration API call failed (HTTP 500), retrying (2/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint \"application_kind_key\"\nDETAIL: Key (kind)=(kafka) already exists."}
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ REGISTRATION FAILED AFTER 3 ATTEMPTS
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] HTTP Status: 500
[ERROR] Response: {"success":false,"error":"Registration failed: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint \"application_kind_key\"\nDETAIL: Key (kind)=(kafka) already exists."}
[ERROR]
[ERROR] API endpoint: https://skeleton.dev.fastorder.com/api/obs/register
[ERROR]
[ERROR] Troubleshooting:
[ERROR] 1. Check if skeleton.dev.fastorder.com is accessible
[ERROR] 2. Verify web application is running
[ERROR] 3. Check web application logs: /var/www/html/skeleton.dev.fastorder.com/logs/
[ERROR] 4. Test API manually:
[ERROR] curl -k -X POST 'https://skeleton.dev.fastorder.com/api/obs/register' \
[ERROR] -H 'Content-Type: application/json' \
[ERROR] -H 'X-Internal-Token: $OBS_INTERNAL_API_TOKEN' \
[ERROR] -d '$PAYLOAD'
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: Kafka Connect
[INFO] Identifier: user-sau-main-dev_coordinator
[INFO] Identifier Parent: coordinator
[INFO] IP: 10.100.1.159
[INFO] Port: 8083
[INFO] FQDN: eventbus-user-sau-main-dev-kafka-connect.fastorder.com
[INFO] Status: running
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 6996dd68-05d6-44fb-bdba-84855c75d261
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[0;32m[OK][0m ✔ Kafka node registration completed
[0;34m[INFO][0m Setting up Kafka observability integration...
[0;34m[INFO][0m Checking observability cell readiness: obs-user-sau-main-dev
[0;32m[OK][0m Observability cell endpoints registered for user-sau-main-dev
[0;34m[INFO][0m Observability cell verified for user-sau-main-dev
[0;34m[INFO][0m Monitoring will be configured after Kafka deployment (step 10-monitoring-setup.sh)
[0;34m[INFO][0m Cleaning up temporary files...
[INFO] Starting cleanup of temporary files...
[INFO] Cleaning up SSL temp files for user-sau-main-dev...
[INFO] Cleaning up old provisioning logs...
[INFO] Cleaning up old configuration backups...
[0;32m[OK][0m ✔ Cleanup completed
[0;32m✓[0m ✅ Event bus infrastructure (kafka) setup completed successfully
[0;34m[INFO][0m Using database engine from DB_ENGINE environment variable: postgresql
[0;34m[INFO][0m Cleaning up any existing locks...
[0;32m[1mStarting database engine: postgresql[0m
[1;33m═══════════════════════════════════════════════[0m
[0;34m[INFO][0m Using environment from web interface: user-sau-main-dev
[0;32m[2026-01-18 23:39:53][0m Using web-provided environment: user-sau-main-dev
[0;32m[2026-01-18 23:39:53][0m Service: user, Zone: sau, Branch: main, Env: dev
[0;32m✓[0m Environment initialized successfully (mode: general)
[0;34m[INFO][0m Checking observability cell readiness: obs-user-sau-main-dev
[1;32m[OK][0m Observability cell endpoints registered for user-sau-main-dev
[0;34m[INFO][0m Observability cell verified for user-sau-main-dev
[0;34m[INFO][0m Monitoring will be configured after PostgreSQL deployment (step 10-monitoring-setup.sh)
[0;34m[INFO][0m Citus mode ENABLED
[0;34m[INFO][0m → Coordinator + 1 worker(s) + 1 standby node(s) per worker
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up coordinator (Citus control plane)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-18 23:39:54 UTC] USER=unknown EUID=33 PID=4036921 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-18 23:39:54 UTC] USER=unknown EUID=33 PID=4036928 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-18 23:39:54 UTC] USER=unknown EUID=33 PID=4036935 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-18 23:39:54 UTC] USER=unknown EUID=33 PID=4036942 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-18 23:39:54 UTC] USER=unknown EUID=33 PID=4036950 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-18 23:39:54 UTC] USER=unknown EUID=33 PID=4036957 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for user in sau-dev...
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61 JOB_UUID=c535671c-4f96-43e7-95ee-3e02cbcf2d2f
[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.95
[0;34m[INFO][0m Primary hostname: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[0;34m[INFO][0m Adding /etc/hosts entries for coordinator...
[0;34m[INFO][0m 1. db-user-sau-main-dev-postgresql.fastorder.com → 10.100.1.95 (primary/short)
[0;34m[INFO][0m 2. db-user-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.95 (compatibility)
[0;34m[INFO][0m ➕ Adding db-user-sau-main-dev-postgresql.fastorder.com → 10.100.1.95
[0;32m✅[0m ✅ Added: db-user-sau-main-dev-postgresql.fastorder.com → 10.100.1.95
[0;34m[INFO][0m ➕ Adding db-user-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.95
[0;32m✅[0m ✅ Added: db-user-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.95
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
10.100.1.95 db-user-sau-main-dev-postgresql.fastorder.com
10.100.1.95 db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau (Saudi Arabia)
Branch: main
Env: dev
Node: coordinator
Primary CN: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
Alt CN: user-sau-main-dev.fastorder.com
VM IP: 142.93.238.16
Coordinator variants:
- db-user-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
- db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-18 23:39:57 UTC] USER=www-data EUID=0 PID=4037141 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator and /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:39:57 UTC] USER=www-data EUID=0 PID=4037150 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-18 23:39:57 UTC] USER=www-data EUID=0 PID=4037160 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-4037097
[2026-01-18 23:39:57 UTC] USER=www-data EUID=0 PID=4037169 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-4037097/ra_root.crt
[2026-01-18 23:39:57 UTC] USER=www-data EUID=0 PID=4037178 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-4037097/ra_root.key
[2026-01-18 23:39:57 UTC] USER=www-data EUID=0 PID=4037187 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4037097/ra_root.crt
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-18 23:39:57 UTC] USER=www-data EUID=0 PID=4037241 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4037097/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-18 23:39:57 UTC] USER=www-data EUID=0 PID=4037250 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4037097/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-18 23:39:57 UTC] USER=www-data EUID=0 PID=4037259 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-18 23:39:57 UTC] USER=www-data EUID=0 PID=4037268 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4037097/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:39:57 UTC] USER=www-data EUID=0 PID=4037277 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:39:57 UTC] USER=www-data EUID=0 PID=4037286 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:39:57 UTC] USER=www-data EUID=0 PID=4037295 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-18 23:39:58 UTC] USER=www-data EUID=0 PID=4037306 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-18 23:39:58 UTC] USER=www-data EUID=0 PID=4037315 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-18 23:39:58 UTC] USER=www-data EUID=0 PID=4037324 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-18 23:39:58 UTC] USER=www-data EUID=0 PID=4037333 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-18 23:39:58 UTC] USER=www-data EUID=0 PID=4037342 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:39:58 UTC] USER=www-data EUID=0 PID=4037351 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🔍 Verifying certificate...
Certificate details:
Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-coordinator.fastorder.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
--
X509v3 Subject Alternative Name:
DNS:db-user-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:user-sau-main-dev.fastorder.com, DNS:db-user-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:db-user-sau-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-user-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-user-sau-main-dev-postgresql.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
X509v3 Subject Key Identifier:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Node: coordinator
Primary CN: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-coordinator.service
3. Test SSL connection:
psql "host=db-user-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username: postgres
Identifier: coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau
Branch: main
Env: dev
Node: coordinator
User (CN): postgres
Hostname: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:39:58 UTC] USER=www-data EUID=0 PID=4037405 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-01-18 23:39:58 UTC] USER=www-data EUID=0 PID=4037414 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-18 23:39:58 UTC] USER=www-data EUID=0 PID=4037423 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-01-18 23:39:58 UTC] USER=www-data EUID=0 PID=4037432 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-18 23:39:58 UTC] USER=www-data EUID=0 PID=4037441 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037457 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037466 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037475 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037484 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037493 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037502 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037511 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037521 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037530 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037539 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037548 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037557 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037569 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037580 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037589 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037598 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037607 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037616 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037642 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037651 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037660 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037669 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037678 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037687 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037696 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037705 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037714 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037723 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037732 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037741 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037751 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037764 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037773 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037782 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037791 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037800 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037809 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037818 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037827 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037836 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037845 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037854 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037863 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037873 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:39:59 UTC] USER=www-data EUID=0 PID=4037883 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4037892 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4037901 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4037910 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4037919 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4037933 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4037944 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4037953 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4037962 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4037971 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4037980 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4037989 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4037999 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038009 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038018 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038029 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038038 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038057 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038066 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038075 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038084 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038093 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038102 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038111 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038121 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038144 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038165 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038180 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038192 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038207 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038217 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038227 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038236 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038246 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:40:00 UTC] USER=www-data EUID=0 PID=4038256 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: user-sau-main-dev
User: postgres
Node: coordinator
FQDN: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt"
export PGSSLMODE="verify-full"
psql -h db-user-sau-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username: postgres
Identifier: coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau
Branch: main
Env: dev
Node: coordinator
User (CN): postgres
Hostname: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:40:01 UTC] USER=www-data EUID=0 PID=4038309 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-postgres
[2026-01-18 23:40:01 UTC] USER=www-data EUID=0 PID=4038318 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-18 23:40:01 UTC] USER=www-data EUID=0 PID=4038327 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-postgres/ra_root.key
[2026-01-18 23:40:01 UTC] USER=www-data EUID=0 PID=4038336 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.crt
[2026-01-18 23:40:01 UTC] USER=www-data EUID=0 PID=4038345 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:40:01 UTC] USER=www-data EUID=0 PID=4038396 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:40:01 UTC] USER=www-data EUID=0 PID=4038409 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:40:01 UTC] USER=www-data EUID=0 PID=4038431 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt
[2026-01-18 23:40:01 UTC] USER=www-data EUID=0 PID=4038440 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:01 UTC] USER=www-data EUID=0 PID=4038458 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038467 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038476 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038494 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038503 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038521 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038532 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038549 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038561 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038570 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038579 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038589 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038598 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038624 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038635 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038644 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038654 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038665 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038684 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038705 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038717 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-18 23:40:02 UTC] USER=www-data EUID=0 PID=4038726 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038748 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038760 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038770 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038779 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038788 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038797 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038806 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038815 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038824 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038833 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038846 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038855 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038864 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038876 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038890 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038903 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038912 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038921 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038931 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038940 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038949 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038958 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038967 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038977 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038986 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4038995 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4039005 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4039020 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4039035 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4039046 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4039055 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4039065 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:03 UTC] USER=www-data EUID=0 PID=4039074 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key
[2026-01-18 23:40:04 UTC] USER=www-data EUID=0 PID=4039083 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
[2026-01-18 23:40:04 UTC] USER=www-data EUID=0 PID=4039092 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:04 UTC] USER=www-data EUID=0 PID=4039101 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:40:04 UTC] USER=www-data EUID=0 PID=4039110 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key.pkcs1
[2026-01-18 23:40:04 UTC] USER=www-data EUID=0 PID=4039119 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_der.key
[2026-01-18 23:40:04 UTC] USER=www-data EUID=0 PID=4039128 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres_pk8.der
[2026-01-18 23:40:04 UTC] USER=www-data EUID=0 PID=4039138 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:40:04 UTC] USER=www-data EUID=0 PID=4039148 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:40:04 UTC] USER=www-data EUID=0 PID=4039157 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:40:04 UTC] USER=www-data EUID=0 PID=4039166 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-01-18 23:40:04 UTC] USER=www-data EUID=0 PID=4039175 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-01-18 23:40:04 UTC] USER=www-data EUID=0 PID=4039184 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-01-18 23:40:04 UTC] USER=www-data EUID=0 PID=4039193 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:40:04 UTC] USER=www-data EUID=0 PID=4039202 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:40:04 UTC] USER=www-data EUID=0 PID=4039211 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:40:04 UTC] USER=www-data EUID=0 PID=4039220 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: user-sau-main-dev
User: postgres
Node: coordinator
FQDN: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt"
export PGSSLMODE="verify-full"
psql -h db-user-sau-main-dev-postgresql-coordinator.fastorder.com -U postgres -d postgres
[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 📦 02 setup pg instance...
[0;34m[DEADLOCK-PREVENTION][0m Deadlock prevention library loaded
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Creating Postgresql Ident:db-coordinator-postgresql application environment...
[INFO] 🎯 Custom Environment Creation (Example Wrapper)
[INFO] 📁 Orchestrator Library: /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] 💾 State Directory: /opt/fastorder/bash/scripts/env_app_setup/state
[INFO] 🚀 Calling centralized orchestrator: fo-env create-app
[INFO] 📋 Arguments: --service user --zone sau --branch main --env dev --domain db-user-sau-main-dev-postgresql-coordinator --app db-coordinator-postgresql
[INFO] Creating application-specific environment configuration
[INFO] Environment ID: user-sau-main-dev
[INFO] Application: db-coordinator-postgresql
[INFO] Base environment user-sau-main-dev already exists
[INFO] Allocated db-coordinator-postgresql IP: 10.100.1.231
[INFO] Generated domain: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[INFO] Configuring network interface for db-coordinator-postgresql IP: 10.100.1.231
[INFO] IP 10.100.1.231 is already configured
[INFO] Updating topology with application-specific configuration...
[ OK ] Topology updated with application-specific configuration
[INFO] Binding db-coordinator-postgresql IP to domain: 10.100.1.231 -> db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[WARN] Domain 'db-user-sau-main-dev-postgresql-coordinator.fastorder.com' already exists in /etc/hosts
[INFO] Removing old entries for domain: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-18 23:40:06 UTC] USER=www-data EUID=0 PID=4039764 ACTION=fsop ARGS=sed -i /\sdb-user-sau-main-dev-postgresql-coordinator.fastorder.com\(\s\|$\)/d /etc/hosts
[ OK ] Successfully bound db-user-sau-main-dev-postgresql-coordinator.fastorder.com to 10.100.1.231
[ OK ] Domain correctly mapped
[ OK ] Application environment created successfully!
[INFO]
[INFO] Application Details:
[INFO] Environment ID: user-sau-main-dev
[INFO] Application: db-coordinator-postgresql
[INFO] IP: 10.100.1.231
[INFO] Domain: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[INFO]
[INFO] To use this application:
[INFO] source /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/lib/config_management.sh
[INFO] init_environment db-coordinator-postgresql
[INFO] echo $VM_IP # Returns: 10.100.1.231
[ OK ] 🎉 Environment creation completed successfully!
[INFO] 📋 What happened:
[INFO] ✅ Called centralized orchestrator at /opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator
[INFO] ✅ All topology.json management handled centrally
[INFO] ✅ Application-specific IP and domain configured
[INFO] ✅ Network interface configured and made persistent
[INFO] ✅ Domain binding added to /etc/hosts (if not skipped)
[INFO] 🔧 To use the centralized orchestrator directly:
[INFO] # Add orchestrator to PATH
[INFO] export PATH="/opt/fastorder/bash/scripts/env_app_setup/lib/env-orchestrator/bin:$PATH"
[INFO] # Then call directly
[INFO] fo-env create-app --service auth --zone uae --env dev --app redis
[INFO] 📚 For more orchestrator commands:
[INFO] fo-env --help
[ OK ] Created db-coordinator-postgresql environment: db-user-sau-main-dev-postgresql-coordinator.fastorder.com (10.100.1.231)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.231
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m Data dir: /data/postgresql/17/user-sau-main-dev/coordinator
[0;34m[INFO][0m Port: 5432
[0;34m[INFO][0m Hostname: db-user-sau-main-dev-postgresql-coordinator
[2026-01-18 23:40:06 UTC] USER=www-data EUID=0 PID=4039818 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:40:06 UTC] USER=www-data EUID=0 PID=4039839 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:40:06 UTC] USER=www-data EUID=0 PID=4039860 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:40:06 UTC] USER=www-data EUID=0 PID=4039881 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[1;33m[WARN][0m Server certificate not found at /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[0;34m[INFO][0m Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau (Saudi Arabia)
Branch: main
Env: dev
Node: coordinator
Primary CN: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
Alt CN: user-sau-main-dev.fastorder.com
VM IP: 142.93.238.16
Coordinator variants:
- db-user-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
- db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-18 23:40:07 UTC] USER=www-data EUID=0 PID=4039921 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator and /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:40:07 UTC] USER=www-data EUID=0 PID=4039930 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🔐 Generating 4096-bit private key...
[2026-01-18 23:40:07 UTC] USER=www-data EUID=0 PID=4039940 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-4039888
[2026-01-18 23:40:07 UTC] USER=www-data EUID=0 PID=4039949 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-4039888/ra_root.crt
[2026-01-18 23:40:07 UTC] USER=www-data EUID=0 PID=4039958 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-4039888/ra_root.key
[2026-01-18 23:40:07 UTC] USER=www-data EUID=0 PID=4039967 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4039888/ra_root.crt
[2026-01-18 23:40:07 UTC] USER=www-data EUID=0 PID=4039976 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4039888/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[2026-01-18 23:40:10 UTC] USER=www-data EUID=0 PID=4040071 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4039888/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-18 23:40:10 UTC] USER=www-data EUID=0 PID=4040080 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4039888/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-18 23:40:10 UTC] USER=www-data EUID=0 PID=4040089 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
📋 Setting up CA certificate...
[2026-01-18 23:40:10 UTC] USER=www-data EUID=0 PID=4040098 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4039888/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:10 UTC] USER=www-data EUID=0 PID=4040107 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:10 UTC] USER=www-data EUID=0 PID=4040116 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:10 UTC] USER=www-data EUID=0 PID=4040125 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-18 23:40:10 UTC] USER=www-data EUID=0 PID=4040145 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
[2026-01-18 23:40:10 UTC] USER=www-data EUID=0 PID=4040154 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-18 23:40:10 UTC] USER=www-data EUID=0 PID=4040163 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-18 23:40:10 UTC] USER=www-data EUID=0 PID=4040172 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:40:10 UTC] USER=www-data EUID=0 PID=4040181 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🔍 Verifying certificate...
Certificate details:
Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-coordinator.fastorder.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
--
X509v3 Subject Alternative Name:
DNS:db-user-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:user-sau-main-dev.fastorder.com, DNS:db-user-sau-main-dev-postgresql-coordinator.fastorder.com, DNS:db-user-sau-main-dev-postgresql-coordinator, DNS:localhost, DNS:db-user-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-sau-main-dev-postgresql-coordinator-coordinator.fastorder.com, DNS:db-user-sau-main-dev-postgresql.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
X509v3 Subject Key Identifier:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Node: coordinator
Primary CN: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-coordinator.service
3. Test SSL connection:
psql "host=db-user-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-18 23:40:10 UTC] USER=www-data EUID=0 PID=4040210 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/server.crt
[2026-01-18 23:40:10 UTC] USER=www-data EUID=0 PID=4040228 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
[0;32m[OK][0m mTLS certificates OK (server cert + client certs verified) and keys secured
[0;34m[INFO][0m Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-18 23:40:10 UTC] USER=www-data EUID=0 PID=4040249 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-coordinator.service
[2026-01-18 23:40:10 UTC] USER=www-data EUID=0 PID=4040270 ACTION=passthru ARGS=systemctl stop postgresql
[1;33m[WARN][0m Cleaning stale socket directory /var/run/postgresql-user-sau-main-dev-coordinator
[2026-01-18 23:40:10 UTC] USER=www-data EUID=0 PID=4040301 ACTION=fsop ARGS=rm -rf /var/run/postgresql-user-sau-main-dev-coordinator
[0;32m[OK][0m No conflicting Postgres left on port 5432
[0;32m[OK][0m Using postgres password from vault provider
[2026-01-18 23:40:12 UTC] USER=www-data EUID=0 PID=4040436 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.NgisRJ
[2026-01-18 23:40:12 UTC] USER=www-data EUID=0 PID=4040479 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/user-sau-main-dev
[2026-01-18 23:40:12 UTC] USER=www-data EUID=0 PID=4040501 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev
[2026-01-18 23:40:12 UTC] USER=www-data EUID=0 PID=4040528 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/user-sau-main-dev
[0;34m[INFO][0m Initializing cluster in /data/postgresql/17/user-sau-main-dev/coordinator (SCRAM; pwfile)
[1;33m[WARN][0m Removing existing data directory: /data/postgresql/17/user-sau-main-dev/coordinator
[2026-01-18 23:40:12 UTC] USER=www-data EUID=0 PID=4040551 ACTION=fsop ARGS=rm -rf /data/postgresql/17/user-sau-main-dev/coordinator
[2026-01-18 23:40:13 UTC] USER=www-data EUID=0 PID=4040606 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev/coordinator
[2026-01-18 23:40:13 UTC] USER=www-data EUID=0 PID=4040627 ACTION=fsop ARGS=chmod 700 /data/postgresql/17/user-sau-main-dev/coordinator
[2026-01-18 23:40:13 UTC] USER=www-data EUID=0 PID=4040648 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-user-sau-main-dev-coordinator
[2026-01-18 23:40:13 UTC] USER=www-data EUID=0 PID=4040669 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-user-sau-main-dev-coordinator
[2026-01-18 23:40:13 UTC] USER=www-data EUID=0 PID=4040690 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-user-sau-main-dev-coordinator
[2026-01-18 23:40:13 UTC] USER=www-data EUID=0 PID=4040699 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/user-sau-main-dev/coordinator --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.NgisRJ
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /data/postgresql/17/user-sau-main-dev/coordinator ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
/usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/user-sau-main-dev/coordinator -l logfile start
[0;32m[OK][0m initdb complete
[2026-01-18 23:40:14 UTC] USER=www-data EUID=0 PID=4040753 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.NgisRJ
[0;34m[INFO][0m Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[0;32m[OK][0m postgresql.conf updated successfully
[0;34m[INFO][0m Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-18 23:40:14 UTC] USER=www-data EUID=0 PID=4040824 ACTION=fsop ARGS=cp /tmp/tmp.u8m7g3MQp5 /data/postgresql/17/user-sau-main-dev/coordinator/pg_hba.conf
[2026-01-18 23:40:14 UTC] USER=www-data EUID=0 PID=4040846 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev/coordinator/pg_hba.conf
[2026-01-18 23:40:14 UTC] USER=www-data EUID=0 PID=4040867 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/user-sau-main-dev/coordinator/pg_hba.conf
[0;32m[OK][0m pg_hba.conf updated
[0;34m[INFO][0m Creating systemd unit: /etc/systemd/system/postgresql@user-sau-main-dev-coordinator.service
[0;32m[OK][0m systemd unit written
[2026-01-18 23:40:15 UTC] USER=www-data EUID=0 PID=4040955 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-18 23:40:15 UTC] USER=www-data EUID=0 PID=4040976 ACTION=passthru ARGS=systemctl daemon-reload
[0;34m[INFO][0m Starting PostgreSQL instance...
[2026-01-18 23:40:16 UTC] USER=www-data EUID=0 PID=4041130 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-coordinator.service
[0;34m[INFO][0m Waiting for ACTIVE (systemd)…
[2026-01-18 23:40:17 UTC] USER=www-data EUID=0 PID=4041173 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[0;32m[OK][0m Service ACTIVE
[0;34m[INFO][0m Waiting for port 5432 bind…
[0;32m[OK][0m Port bound
[0;34m[INFO][0m Waiting pg_isready (socket)…
[0;32m[OK][0m Readiness via socket OK
[0;34m[INFO][0m Waiting pg_isready (TCP db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432)…
[0;32m[OK][0m Startup sequence complete
[0;34m[INFO][0m Validating core security GUCs (via local socket)…
[0;32m[OK][0m Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[0;34m[INFO][0m Provisioning application database and Debezium role (if not exists)...
[0;34m[INFO][0m Checking if database fastorder_user_sau_main_dev_db exists...
[0;34m[INFO][0m DB check result: exit_code=0, output='[2026-01-18 23:40:18 UTC] USER=www-data EUID=0 PID=4041337 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_user_sau_main_dev_db''
[0;34m[INFO][0m Creating database fastorder_user_sau_main_dev_db...
[2026-01-18 23:40:18 UTC] USER=www-data EUID=0 PID=4041365 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_user_sau_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[0;32m[OK][0m Database fastorder_user_sau_main_dev_db created
[0;34m[INFO][0m Checking if role debezium_user exists...
[0;34m[INFO][0m Role check result: exit_code=0, output='[2026-01-18 23:40:18 UTC] USER=www-data EUID=0 PID=4041389 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[0;34m[INFO][0m Creating role debezium_user...
[2026-01-18 23:40:18 UTC] USER=www-data EUID=0 PID=4041417 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD 'iLaMZwuZ3LeD6ygrLFPsdLRm';
CREATE ROLE
[0;32m[OK][0m Role debezium_user created
[2026-01-18 23:40:18 UTC] USER=www-data EUID=0 PID=4041445 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_user_sau_main_dev_db" TO debezium_user;
GRANT
[0;32m[OK][0m Application DB (fastorder_user_sau_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[0;34m[INFO][0m Applying connection and memory optimizations...
[0;34m[INFO][0m Current settings: max_connections=100, work_mem=4MB
[0;34m[INFO][0m Target settings (coordinator): max_connections=150, work_mem=8MB
[2026-01-18 23:40:19 UTC] USER=www-data EUID=0 PID=4041546 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 150;
ALTER SYSTEM
[2026-01-18 23:40:19 UTC] USER=www-data EUID=0 PID=4041569 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-18 23:40:19 UTC] USER=www-data EUID=0 PID=4041595 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
pg_reload_conf
----------------
t
(1 row)
[0;32m[OK][0m Settings applied to postgresql.auto.conf
[2026-01-18 23:40:19 UTC] USER=www-data EUID=0 PID=4041610 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/user-sau-main-dev/coordinator/standby.signal
[0;34m[INFO][0m Service recently started (3s ago) - restarting to apply max_connections...
[0;34m[INFO][0m Stopping service...
[2026-01-18 23:40:19 UTC] USER=www-data EUID=0 PID=4041632 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-coordinator.service
[0;34m[INFO][0m Waiting for port 5432 to be released...
[0;32m[OK][0m Port 5432 released
[0;34m[INFO][0m Starting service...
[2026-01-18 23:40:23 UTC] USER=www-data EUID=0 PID=4041708 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-coordinator.service
[2026-01-18 23:40:28 UTC] USER=www-data EUID=0 PID=4041835 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[0;32m[OK][0m ✅ Optimization complete: max_connections=150, work_mem=8MB
[0;34m[INFO][0m Setting postgres password via centralized script... for coordinator
[0;34m[INFO][0m Temporarily disabling synchronous_commit on coordinator for password setting...
[0;32m[OK][0m Disabled synchronous_commit (was: on)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: user[0m
[0;34m Zone: sau[0m
[0;34m Environment: dev[0m
[0;34m Identifier: coordinator[0m
[0;34mAWS Secret: fastorder/db/user/sau/main/dev/postgresql/coordinator[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-user-sau-main-dev-coordinator[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mInitial setup: Using password from initdb[0m
[0;32m✓ PostgreSQL password already set during initdb[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/coordinator
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/user/sau/main/dev/postgresql/coordinator[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials coordinator[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
[0;34m[INFO][0m Restoring synchronous_commit on coordinator...
[0;32m[OK][0m Restored synchronous_commit to: on
[0;32m[OK][0m Password set and persisted
[0;34m[INFO][0m Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Identifier: coordinator
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.231
[0;34m[INFO][0m Primary hostname: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[0;34m[INFO][0m Adding /etc/hosts entries for coordinator...
[0;34m[INFO][0m 1. db-user-sau-main-dev-postgresql.fastorder.com → 10.100.1.231 (primary/short)
[0;34m[INFO][0m 2. db-user-sau-main-dev-postgresql-coordinator.fastorder.com → 10.100.1.231 (compatibility)
[0;34m[INFO][0m 🔄 Updating db-user-sau-main-dev-postgresql.fastorder.com → 10.100.1.231
[0;32m✅[0m ✅ Updated: db-user-sau-main-dev-postgresql.fastorder.com → 10.100.1.231
[0;34m[INFO][0m ✅ db-user-sau-main-dev-postgresql-coordinator.fastorder.com already exists with correct IP
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
10.100.1.231 db-user-sau-main-dev-postgresql-coordinator.fastorder.com
10.100.1.231 db-user-sau-main-dev-postgresql.fastorder.com
[0;32m[OK][0m PostgreSQL 'user-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt \
sslcert=/home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.crt \
sslkey=/home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/postgres.key \
host=db-user-sau-main-dev-postgresql-coordinator port=5432 dbname=postgres user=postgres"
File been compeleted perfectly: 02-setup-pg-instance
[0;34m[INFO][0m Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: PostgreSQL
[INFO] Identifier: user-sau-main-dev-postgresql-coordinator
[INFO] Identifier Parent: coordinator
[INFO] IP: 10.100.1.231
[INFO] Port: 5432
[INFO] FQDN: db-user-sau-main-dev-postgresql-coordinator
[INFO] Status: running
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 83d5bc7d-3699-4f7e-98b2-72fdfea60e05
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[0;32m[OK][0m PostgreSQL node registered to observability API
[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 📦 03 role...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-18 23:40:38 UTC] USER=www-data EUID=0 PID=4042465 ACTION=fsop ARGS=test -f /data/postgresql/17/user-sau-main-dev/coordinator/standby.signal
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username: debezium_user
Identifier: coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau
Branch: main
Env: dev
Node: coordinator
User (CN): debezium_user
Hostname: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:40:38 UTC] USER=www-data EUID=0 PID=4042621 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-debezium_user
[2026-01-18 23:40:38 UTC] USER=www-data EUID=0 PID=4042630 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-01-18 23:40:38 UTC] USER=www-data EUID=0 PID=4042639 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-debezium_user/ra_root.key
[2026-01-18 23:40:38 UTC] USER=www-data EUID=0 PID=4042648 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.crt
[2026-01-18 23:40:38 UTC] USER=www-data EUID=0 PID=4042657 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042673 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042682 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042691 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042700 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042709 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042718 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042727 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042736 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042745 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042754 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042763 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042772 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042781 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042790 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042799 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042808 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042817 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042826 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042852 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042861 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042870 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042879 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042888 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042897 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042906 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042915 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042924 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042933 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042942 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042951 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_pk8.der
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042961 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042971 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042980 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042989 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4042998 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4043007 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4043016 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4043025 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4043034 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4043043 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:40:39 UTC] USER=www-data EUID=0 PID=4043052 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043061 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043070 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_pk8.der
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043080 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043090 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043099 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043108 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043117 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043126 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043135 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043144 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043153 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043162 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043171 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043180 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043189 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_pk8.der
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043199 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043209 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043218 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043227 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043236 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043245 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043254 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043263 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043272 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043281 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043290 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key.pkcs1
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043299 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_der.key
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043308 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user_pk8.der
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043318 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043328 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043337 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043346 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043355 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043364 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043373 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043382 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043391 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:40:40 UTC] USER=www-data EUID=0 PID=4043400 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: user-sau-main-dev
User: debezium_user
Node: coordinator
FQDN: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt
database.sslkey: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/debezium_user.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt"
export PGSSLMODE="verify-full"
psql -h db-user-sau-main-dev-postgresql-coordinator.fastorder.com -U debezium_user -d postgres
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: user[0m
[0;34m Zone: sau[0m
[0;34m Environment: dev[0m
[0;34m Identifier: coordinator[0m
[0;34mAWS Secret: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-user-sau-main-dev-coordinator[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mGenerating new secure password...[0m
[0;34mUser fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: fastorder_admin_gd[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials coordinator[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: coordinator/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-user-sau-main-dev-postgresql.fastorder.com
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username: fastorder_admin_gd
Identifier: coordinator
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau
Branch: main
Env: dev
Node: coordinator
User (CN): fastorder_admin_gd
Hostname: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:40:47 UTC] USER=www-data EUID=0 PID=4043932 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-coordinator-fastorder_admin_gd
[2026-01-18 23:40:47 UTC] USER=www-data EUID=0 PID=4043941 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-18 23:40:47 UTC] USER=www-data EUID=0 PID=4043951 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
[2026-01-18 23:40:47 UTC] USER=www-data EUID=0 PID=4043963 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt
[2026-01-18 23:40:47 UTC] USER=www-data EUID=0 PID=4043974 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044033 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044043 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044057 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044067 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044076 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044085 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044097 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044108 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044123 ACTION=fsop ARGS=cp -f /tmp/pg-client-coordinator-fastorder_admin_gd/fastorder_admin_gd_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044133 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044142 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044151 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044160 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044170 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044179 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044188 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044197 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044206 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044215 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044226 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044235 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044261 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044270 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044279 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044288 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044297 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044306 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044315 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044324 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044333 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044351 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044360 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044370 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044380 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044389 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:48 UTC] USER=www-data EUID=0 PID=4044398 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044407 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044416 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044425 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044434 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044443 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044452 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044461 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044470 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044480 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044490 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044500 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044509 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044518 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044528 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044541 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044550 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044559 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044568 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044577 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044586 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key.pkcs1
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044595 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044604 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044614 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044627 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044636 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044645 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044654 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044663 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044672 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044681 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044690 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044699 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044717 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_der.key
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044726 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd_pk8.der
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044736 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator → /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044746 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:40:49 UTC] USER=www-data EUID=0 PID=4044755 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:40:50 UTC] USER=www-data EUID=0 PID=4044764 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-01-18 23:40:50 UTC] USER=www-data EUID=0 PID=4044773 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-01-18 23:40:50 UTC] USER=www-data EUID=0 PID=4044782 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-01-18 23:40:50 UTC] USER=www-data EUID=0 PID=4044791 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:40:50 UTC] USER=www-data EUID=0 PID=4044800 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:40:50 UTC] USER=www-data EUID=0 PID=4044818 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: user-sau-main-dev
User: fastorder_admin_gd
Node: coordinator
FQDN: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt
database.sslkey: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/fastorder_admin_gd.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt"
export PGSSLMODE="verify-full"
psql -h db-user-sau-main-dev-postgresql-coordinator.fastorder.com -U fastorder_admin_gd -d postgres
🧱 Connecting via Unix socket to create role and database...
Socket: /var/run/postgresql-user-sau-main-dev-coordinator:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️ Database fastorder_user_sau_main_dev_db already exists, skipping creation
[2026-01-18 23:40:50 UTC] USER=www-data EUID=0 PID=4044876 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️ Using pg_hba.conf: /data/postgresql/17/user-sau-main-dev/coordinator/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-18 23:40:50 UTC] USER=www-data EUID=0 PID=4044913 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-coordinator.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[0;34m=== Pre-flight Checks ===[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;32m✓[0m AWS Secrets Manager accessible
[0;34m=== Retrieving Credentials from AWS ===[0m
ℹ️ Retrieving PostgreSQL credentials for: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
ℹ️ Fetching secret: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
✅ PostgreSQL credentials loaded for coordinator/fastorder_admin_gd: fastorder_admin_gd@db-user-sau-main-dev-postgresql.fastorder.com:5432/fastorder_user_sau_main_dev_db
[0;32m✓[0m Credentials retrieved: fastorder_admin_gd@db-user-sau-main-dev-postgresql.fastorder.com:5432/fastorder_user_sau_main_dev_db
[0;34m╔════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Test Suite (AWS Secrets MGR) ║[0m
[0;34m╚════════════════════════════════════════════╝[0m
[0;34m=== PostgreSQL Authentication Test ===[0m
[0;31m✗[0m PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-user-sau-main-dev-postgresql.fastorder.com" (10.100.1.231), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/user/sau/main/dev/postgresql/coordinator/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-18 23:40:55 UTC] USER=www-data EUID=0 PID=4045248 ACTION=fsop ARGS=test -f /data/postgresql/17/user-sau-main-dev/coordinator/standby.signal
── fast setup ─────────────────────────────────────────────
NAME : user-sau-main-dev
IDENTIFIER : coordinator
PG HOST : db-user-sau-main-dev-postgresql.fastorder.com:5432
ROLE : debezium_user
DB : fastorder_user_sau_main_dev_db
SCHEMA : user
AUTH MODE : scram (scram=password over TLS | cert=mTLS)
SUBNET ALLOW: 10.201.0.0/16
CONNECT /32 : 142.93.238.16
SSL DIR : /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator
DNS → 10.100.1.231
CA : /home/www-data/ssl/.postgresql/user-sau-main-dev/coordinator/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: user[0m
[0;34m Zone: sau[0m
[0;34m Environment: dev[0m
[0;34m Identifier: coordinator[0m
[0;34mAWS Secret: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-user-sau-main-dev-coordinator[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mGenerating new secure password...[0m
[0;34mUser debezium_user does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: debezium_user[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/user/sau/main/dev/postgresql/coordinator/debezium_user[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials coordinator[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: coordinator/debezium_user
✓ Retrieved password from secrets vault
password : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️ Role debezium_user exists, updating
[2026-01-18 23:41:02 UTC] USER=www-data EUID=0 PID=4045676 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️ Database fastorder_user_sau_main_dev_db already exists
[2026-01-18 23:41:02 UTC] USER=www-data EUID=0 PID=4045704 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-coordinator -p 5432 -d fastorder_user_sau_main_dev_db --no-psqlrc
ERROR: syntax error at or near "user"
LINE 1: CREATE SCHEMA IF NOT EXISTS user;
^
GRANT
ERROR: syntax error at or near "user"
LINE 1: GRANT USAGE ON SCHEMA user TO debezium_user;
^
ERROR: syntax error at or near "user"
LINE 1: GRANT SELECT ON ALL TABLES IN SCHEMA user TO debezium_user;
^
ERROR: syntax error at or near "user"
LINE 1: GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA user TO debez...
^
ERROR: syntax error at or near "user"
LINE 1: ALTER DEFAULT PRIVILEGES IN SCHEMA user GRANT SELECT ON TABL...
^
✅ Role/DB/grants ensured.
⚠️ Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/user-sau-main-dev/coordinator/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
ℹ️ Service-specific setup (user) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)
🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=user
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/user/run.sh
🔍 DEBUG_CHECKPOINT_03: No specific folder for user, using default
[DEBUG] Tracking substep start: steps/01-install/steps/default (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 🔸 Service: user (using default contracts schema)
🔍 DEBUG_CHECKPOINT_04: Executing default: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/run.sh
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting default contracts schema provisioning for SERVICE=user
[INFO] Environment: user-sau-main-dev
[INFO] Schema: user (contracts tables)
[INFO] Identifier: coordinator
[INFO] VM IP: 142.93.238.16
🔍 DEBUG: Looking for contracts steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/default/contracts/steps
[INFO] 📁 Running contracts schema setup for: user
[INFO] 📁 Steps directory: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/default/contracts/steps
[INFO] 📦 01 init schema...
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Initializing user schema (contracts tables)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Schema: user
Identifier: coordinator
Database: fastorder_user_sau_main_dev_db
Host: db-user-sau-main-dev-postgresql.fastorder.com:5432
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 Connecting to PostgreSQL over SSL (verify-full + mTLS)...
🗄️ Checking database: fastorder_user_sau_main_dev_db
ℹ️ Database fastorder_user_sau_main_dev_db already exists
✅ Connected to database: fastorder_user_sau_main_dev_db
ℹ️ Checking synchronous replication configuration...
synchronous_standby_names: ''
Connected standbys: 0
ℹ️ Synchronous replication not configured (standbys will be added later)
🔧 Installing extensions...
CREATE EXTENSION
CREATE EXTENSION
🔧 Installing Citus extension on coordinator...
CREATE EXTENSION
✅ Citus extension installed
✅ Extensions installed
🔧 Installing UUIDv7 function...
✅ UUIDv7 function installed
🔧 Creating user schema...
CREATE SCHEMA
✅ Schema created
🔧 Creating contracts tables in user schema...
Creating "user".contract_key...
CREATE TABLE
Creating "user".contract_type...
CREATE TABLE
Creating "user".contracts...
CREATE TABLE
Adding columns to "user".contracts (safe migration)...
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
UPDATE 0
UPDATE 0
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
Creating "user".contract_vars...
CREATE TABLE
Creating "user".contract_datetime...
CREATE TABLE
Creating "user".contract_decimal...
CREATE TABLE
Creating "user".contract_float...
CREATE TABLE
Creating "user".contract_int...
CREATE TABLE
Creating "user".contract_json...
CREATE TABLE
Creating "user".contract_terms...
CREATE TABLE
Creating "user".contract_term_contracts...
CREATE TABLE
Creating "user".contract_term_datetime...
CREATE TABLE
Creating "user".contract_term_decimal...
CREATE TABLE
Creating "user".contract_term_float...
CREATE TABLE
Creating "user".contract_term_int...
CREATE TABLE
Creating "user".contract_term_items...
CREATE TABLE
Creating "user".contract_term_json...
CREATE TABLE
Creating "user".contract_term_vars...
CREATE TABLE
Creating "user".user_id_uuid_mapping...
CREATE TABLE
✅ All 19 tables created
🔧 Creating indexes...
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
✅ All indexes created
🔧 Creating foreign keys...
DO
DO
✅ Foreign keys created
🔧 Configuring Citus distribution...
Creating reference table: contract_key
create_reference_table
------------------------
(1 row)
Creating reference table: contract_type
create_reference_table
------------------------
(1 row)
Creating distributed table: contracts
Creating distributed table: contract_vars
Creating distributed table: contract_datetime
Creating distributed table: contract_decimal
Creating distributed table: contract_float
Creating distributed table: contract_int
Creating distributed table: contract_json
Creating distributed table: contract_terms
Creating distributed table: contract_term_contracts
Creating distributed table: contract_term_datetime
Creating distributed table: contract_term_decimal
Creating distributed table: contract_term_float
Creating distributed table: contract_term_int
Creating distributed table: contract_term_items
Creating distributed table: contract_term_json
create_distributed_table
--------------------------
(1 row)
Creating distributed table: contract_term_vars
create_distributed_table
--------------------------
(1 row)
✅ Citus distribution configured
🎉 Schema initialization complete for user in fastorder_user_sau_main_dev_db
ℹ️ Skipping LISTEN/NOTIFY trigger on coordinator
CDC via Debezium is the primary change tracking mechanism
==========================================
✅ user schema initialization complete!
Tables: 19
Indexes: 54
==========================================
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Default contracts schema setup complete for: user
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✓[0m ✅ Coordinator setup completed
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up 1 worker(s) (Citus data nodes)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up worker: worker-01
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-18 23:41:16 UTC] USER=unknown EUID=33 PID=4046446 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-18 23:41:16 UTC] USER=unknown EUID=33 PID=4046454 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-18 23:41:16 UTC] USER=unknown EUID=33 PID=4046461 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-18 23:41:16 UTC] USER=unknown EUID=33 PID=4046468 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-18 23:41:16 UTC] USER=unknown EUID=33 PID=4046475 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-18 23:41:16 UTC] USER=unknown EUID=33 PID=4046482 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for user in sau-dev...
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61 JOB_UUID=c535671c-4f96-43e7-95ee-3e02cbcf2d2f
[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[WARN] Could not find PostgreSQL IP for worker-01 in topology.json, allocating new VM IP...
[INFO] Allocated new VM IP: 10.100.1.232 for db-worker-01-postgresql
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: worker-01
[INFO] PostgreSQL IP: 10.100.1.232
[INFO] Primary hostname: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO] Adding /etc/hosts entry for worker-01...
[INFO] db-user-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.232
[INFO] ➕ Adding db-user-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.232
[0;32m✅[0m ✅ Added: db-user-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.232
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
10.100.1.232 db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau (Saudi Arabia)
Branch: main
Env: dev
Node: worker-01
Primary CN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
Alt CN: user-sau-main-dev.fastorder.com
VM IP: 142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-18 23:41:19 UTC] USER=www-data EUID=0 PID=4047030 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01 and /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:41:20 UTC] USER=www-data EUID=0 PID=4047045 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-18 23:41:20 UTC] USER=www-data EUID=0 PID=4047057 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-4046995
[2026-01-18 23:41:20 UTC] USER=www-data EUID=0 PID=4047066 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-4046995/ra_root.crt
[2026-01-18 23:41:20 UTC] USER=www-data EUID=0 PID=4047075 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-4046995/ra_root.key
[2026-01-18 23:41:20 UTC] USER=www-data EUID=0 PID=4047084 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4046995/ra_root.crt
[2026-01-18 23:41:20 UTC] USER=www-data EUID=0 PID=4047093 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4046995/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[2026-01-18 23:41:22 UTC] USER=www-data EUID=0 PID=4047186 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4046995/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-18 23:41:22 UTC] USER=www-data EUID=0 PID=4047195 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4046995/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-18 23:41:22 UTC] USER=www-data EUID=0 PID=4047204 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-18 23:41:22 UTC] USER=www-data EUID=0 PID=4047213 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4046995/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:22 UTC] USER=www-data EUID=0 PID=4047222 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:22 UTC] USER=www-data EUID=0 PID=4047231 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:22 UTC] USER=www-data EUID=0 PID=4047243 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-18 23:41:22 UTC] USER=www-data EUID=0 PID=4047254 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-18 23:41:22 UTC] USER=www-data EUID=0 PID=4047263 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-18 23:41:22 UTC] USER=www-data EUID=0 PID=4047272 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-18 23:41:22 UTC] USER=www-data EUID=0 PID=4047281 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-18 23:41:22 UTC] USER=www-data EUID=0 PID=4047290 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:41:22 UTC] USER=www-data EUID=0 PID=4047299 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🔍 Verifying certificate...
Certificate details:
Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-worker-01.fastorder.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
--
X509v3 Subject Alternative Name:
DNS:db-user-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:user-sau-main-dev.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01, DNS:localhost, IP Address:142.93.238.16, IP Address:127.0.0.1
X509v3 Subject Key Identifier:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Node: worker-01
Primary CN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-worker-01.service
3. Test SSL connection:
psql "host=db-user-sau-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username: postgres
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau
Branch: main
Env: dev
Node: worker-01
User (CN): postgres
Hostname: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:41:23 UTC] USER=www-data EUID=0 PID=4047372 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-18 23:41:23 UTC] USER=www-data EUID=0 PID=4047381 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-postgres/ra_root.key
[2026-01-18 23:41:23 UTC] USER=www-data EUID=0 PID=4047390 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-18 23:41:23 UTC] USER=www-data EUID=0 PID=4047399 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:41:23 UTC] USER=www-data EUID=0 PID=4047457 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:41:23 UTC] USER=www-data EUID=0 PID=4047467 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:41:23 UTC] USER=www-data EUID=0 PID=4047476 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047486 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047495 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047505 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047514 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047532 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047541 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047551 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047567 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047576 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047593 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047624 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047636 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047650 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047663 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047697 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047706 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047715 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047726 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047738 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047798 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047809 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047819 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047830 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047841 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047856 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047866 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:24 UTC] USER=www-data EUID=0 PID=4047876 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4047885 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4047894 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4047913 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4047922 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4047931 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4047940 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4047950 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4047964 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4047976 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4047985 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4048003 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4048012 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4048021 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4048030 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4048048 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4048057 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4048066 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4048081 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4048094 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4048122 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4048131 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4048150 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4048159 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4048168 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4048177 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-18 23:41:25 UTC] USER=www-data EUID=0 PID=4048186 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-18 23:41:26 UTC] USER=www-data EUID=0 PID=4048195 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-18 23:41:26 UTC] USER=www-data EUID=0 PID=4048205 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:41:26 UTC] USER=www-data EUID=0 PID=4048215 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:41:26 UTC] USER=www-data EUID=0 PID=4048224 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:41:26 UTC] USER=www-data EUID=0 PID=4048235 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-01-18 23:41:26 UTC] USER=www-data EUID=0 PID=4048244 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-01-18 23:41:26 UTC] USER=www-data EUID=0 PID=4048261 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-01-18 23:41:26 UTC] USER=www-data EUID=0 PID=4048271 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:41:26 UTC] USER=www-data EUID=0 PID=4048280 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:41:26 UTC] USER=www-data EUID=0 PID=4048289 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:41:26 UTC] USER=www-data EUID=0 PID=4048298 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: user-sau-main-dev
User: postgres
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username: postgres
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau
Branch: main
Env: dev
Node: worker-01
User (CN): postgres
Hostname: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:41:26 UTC] USER=www-data EUID=0 PID=4048350 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-postgres
[2026-01-18 23:41:26 UTC] USER=www-data EUID=0 PID=4048361 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-18 23:41:26 UTC] USER=www-data EUID=0 PID=4048384 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.crt
[2026-01-18 23:41:26 UTC] USER=www-data EUID=0 PID=4048403 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048426 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048444 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048453 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048462 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048476 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048486 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048513 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-postgres/postgres_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048522 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048531 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048545 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048554 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048563 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048572 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048582 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048591 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048600 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048609 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048622 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048631 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048657 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048666 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:41:27 UTC] USER=www-data EUID=0 PID=4048675 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048684 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048693 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048702 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048711 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048720 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048738 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048747 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048756 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048766 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048776 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048785 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048794 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048803 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048812 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048821 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048830 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048839 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048848 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048857 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048866 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048885 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048895 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048904 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048913 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048937 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:28 UTC] USER=www-data EUID=0 PID=4048951 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4048962 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4048971 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4048980 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4048989 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4048998 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049007 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049016 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049028 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049046 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049055 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049064 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049075 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049096 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049105 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049115 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049124 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049133 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049142 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key.pkcs1
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049152 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_der.key
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049161 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres_pk8.der
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049171 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049181 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049190 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049199 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049208 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049217 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049226 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049235 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:41:29 UTC] USER=www-data EUID=0 PID=4049244 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: user-sau-main-dev
User: postgres
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U postgres -d postgres
[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 📦 02 setup pg instance...
[0;34m[DEADLOCK-PREVENTION][0m Deadlock prevention library loaded
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Using existing db-worker-01-postgresql environment: db-user-sau-main-dev-postgresql-worker-01.fastorder.com (10.100.1.232)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.232
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m Data dir: /data/postgresql/17/user-sau-main-dev/worker-01
[0;34m[INFO][0m Port: 5432
[0;34m[INFO][0m Hostname: db-user-sau-main-dev-postgresql-worker-01
[2026-01-18 23:41:31 UTC] USER=www-data EUID=0 PID=4049349 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:41:31 UTC] USER=www-data EUID=0 PID=4049370 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[1;33m[WARN][0m Server certificate not found at /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[0;34m[INFO][0m Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau (Saudi Arabia)
Branch: main
Env: dev
Node: worker-01
Primary CN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
Alt CN: user-sau-main-dev.fastorder.com
VM IP: 142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-18 23:41:31 UTC] USER=www-data EUID=0 PID=4049455 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01 and /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:41:31 UTC] USER=www-data EUID=0 PID=4049464 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🔐 Generating 4096-bit private key...
[2026-01-18 23:41:31 UTC] USER=www-data EUID=0 PID=4049474 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-4049419
[2026-01-18 23:41:31 UTC] USER=www-data EUID=0 PID=4049484 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-4049419/ra_root.crt
[2026-01-18 23:41:31 UTC] USER=www-data EUID=0 PID=4049493 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-4049419/ra_root.key
[2026-01-18 23:41:31 UTC] USER=www-data EUID=0 PID=4049502 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4049419/ra_root.crt
[2026-01-18 23:41:31 UTC] USER=www-data EUID=0 PID=4049511 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4049419/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[2026-01-18 23:41:33 UTC] USER=www-data EUID=0 PID=4049569 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4049419/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-18 23:41:33 UTC] USER=www-data EUID=0 PID=4049578 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4049419/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-18 23:41:33 UTC] USER=www-data EUID=0 PID=4049587 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
📋 Setting up CA certificate...
[2026-01-18 23:41:33 UTC] USER=www-data EUID=0 PID=4049596 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4049419/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:33 UTC] USER=www-data EUID=0 PID=4049605 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:33 UTC] USER=www-data EUID=0 PID=4049614 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:41:33 UTC] USER=www-data EUID=0 PID=4049623 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-18 23:41:33 UTC] USER=www-data EUID=0 PID=4049634 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-18 23:41:33 UTC] USER=www-data EUID=0 PID=4049643 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-18 23:41:33 UTC] USER=www-data EUID=0 PID=4049670 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:41:34 UTC] USER=www-data EUID=0 PID=4049679 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🔍 Verifying certificate...
Certificate details:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Node: worker-01
Primary CN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-worker-01.service
3. Test SSL connection:
psql "host=db-user-sau-main-dev-postgresql-worker-01.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-18 23:41:34 UTC] USER=www-data EUID=0 PID=4049708 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.crt
[2026-01-18 23:41:34 UTC] USER=www-data EUID=0 PID=4049717 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/server.key
[2026-01-18 23:41:34 UTC] USER=www-data EUID=0 PID=4049726 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[0;32m[OK][0m mTLS certificates OK (server cert + client certs verified) and keys secured
[0;34m[INFO][0m Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-18 23:41:34 UTC] USER=www-data EUID=0 PID=4049747 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-worker-01.service
[2026-01-18 23:41:34 UTC] USER=www-data EUID=0 PID=4049768 ACTION=passthru ARGS=systemctl stop postgresql
[1;33m[WARN][0m Cleaning stale socket directory /var/run/postgresql-user-sau-main-dev-worker-01
[0;32m[OK][0m No conflicting Postgres left on port 5432
[0;32m[OK][0m Using postgres password from vault provider
[2026-01-18 23:41:35 UTC] USER=www-data EUID=0 PID=4049882 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.CQxZEW
[2026-01-18 23:41:35 UTC] USER=www-data EUID=0 PID=4049903 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.CQxZEW
[2026-01-18 23:41:35 UTC] USER=www-data EUID=0 PID=4049925 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/user-sau-main-dev
[2026-01-18 23:41:35 UTC] USER=www-data EUID=0 PID=4049947 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev
[2026-01-18 23:41:36 UTC] USER=www-data EUID=0 PID=4049969 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/user-sau-main-dev
[0;34m[INFO][0m Initializing cluster in /data/postgresql/17/user-sau-main-dev/worker-01 (SCRAM; pwfile)
[1;33m[WARN][0m Removing existing data directory: /data/postgresql/17/user-sau-main-dev/worker-01
[2026-01-18 23:41:36 UTC] USER=www-data EUID=0 PID=4049990 ACTION=fsop ARGS=rm -rf /data/postgresql/17/user-sau-main-dev/worker-01
[2026-01-18 23:41:36 UTC] USER=www-data EUID=0 PID=4050012 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/user-sau-main-dev/worker-01
[2026-01-18 23:41:36 UTC] USER=www-data EUID=0 PID=4050054 ACTION=fsop ARGS=chmod 700 /data/postgresql/17/user-sau-main-dev/worker-01
[2026-01-18 23:41:36 UTC] USER=www-data EUID=0 PID=4050076 ACTION=fsop ARGS=mkdir -p /var/run/postgresql-user-sau-main-dev-worker-01
[2026-01-18 23:41:36 UTC] USER=www-data EUID=0 PID=4050097 ACTION=fsop ARGS=chown postgres:postgres /var/run/postgresql-user-sau-main-dev-worker-01
[2026-01-18 23:41:36 UTC] USER=www-data EUID=0 PID=4050129 ACTION=passthru ARGS=sudo -u postgres /usr/lib/postgresql/17/bin/initdb -D /data/postgresql/17/user-sau-main-dev/worker-01 --locale=en_US.UTF-8 --encoding=UTF8 --auth-local=scram-sha-256 --auth-host=scram-sha-256 --pwfile=/tmp/.pg_pwfile.CQxZEW
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /data/postgresql/17/user-sau-main-dev/worker-01 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
/usr/lib/postgresql/17/bin/pg_ctl -D /data/postgresql/17/user-sau-main-dev/worker-01 -l logfile start
[0;32m[OK][0m initdb complete
[2026-01-18 23:41:37 UTC] USER=www-data EUID=0 PID=4050181 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.CQxZEW
[0;34m[INFO][0m Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[0;32m[OK][0m postgresql.conf updated successfully
[0;34m[INFO][0m Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-18 23:41:37 UTC] USER=www-data EUID=0 PID=4050230 ACTION=fsop ARGS=cp /tmp/tmp.sJF3om9nwd /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[2026-01-18 23:41:37 UTC] USER=www-data EUID=0 PID=4050272 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[0;32m[OK][0m pg_hba.conf updated
[0;34m[INFO][0m Creating systemd unit: /etc/systemd/system/postgresql@user-sau-main-dev-worker-01.service
[2026-01-18 23:41:37 UTC] USER=www-data EUID=0 PID=4050302 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.ikIL5q /etc/systemd/system/postgresql@user-sau-main-dev-worker-01.service
[2026-01-18 23:41:37 UTC] USER=www-data EUID=0 PID=4050344 ACTION=fsop ARGS=chmod 0644 /etc/systemd/system/postgresql@user-sau-main-dev-worker-01.service
[0;32m[OK][0m systemd unit written
[2026-01-18 23:41:38 UTC] USER=www-data EUID=0 PID=4050379 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-18 23:41:38 UTC] USER=www-data EUID=0 PID=4050404 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-18 23:41:38 UTC] USER=www-data EUID=0 PID=4050427 ACTION=passthru ARGS=systemctl daemon-reload
[0;34m[INFO][0m Starting PostgreSQL instance...
[2026-01-18 23:41:39 UTC] USER=www-data EUID=0 PID=4050566 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-worker-01.service
[0;34m[INFO][0m Waiting for ACTIVE (systemd)…
[2026-01-18 23:41:39 UTC] USER=www-data EUID=0 PID=4050606 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-worker-01.service
[0;32m[OK][0m Service ACTIVE
[0;34m[INFO][0m Waiting for port 5432 bind…
[0;32m[OK][0m Port bound
[0;34m[INFO][0m Waiting pg_isready (socket)…
[0;32m[OK][0m Readiness via socket OK
[0;34m[INFO][0m Waiting pg_isready (TCP db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432)…
[0;32m[OK][0m Startup sequence complete
[0;34m[INFO][0m Validating core security GUCs (via local socket)…
[0;32m[OK][0m Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[0;34m[INFO][0m Provisioning application database and Debezium role (if not exists)...
[0;34m[INFO][0m Checking if database fastorder_user_sau_main_dev_db exists...
[0;34m[INFO][0m DB check result: exit_code=0, output='[2026-01-18 23:41:41 UTC] USER=www-data EUID=0 PID=4050769 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_database WHERE datname = 'fastorder_user_sau_main_dev_db''
[0;34m[INFO][0m Creating database fastorder_user_sau_main_dev_db...
[2026-01-18 23:41:41 UTC] USER=www-data EUID=0 PID=4050792 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE DATABASE "fastorder_user_sau_main_dev_db" ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0;
CREATE DATABASE
[0;32m[OK][0m Database fastorder_user_sau_main_dev_db created
[0;34m[INFO][0m Checking if role debezium_user exists...
[0;34m[INFO][0m Role check result: exit_code=0, output='[2026-01-18 23:41:41 UTC] USER=www-data EUID=0 PID=4050821 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -Atqc SELECT 1 FROM pg_roles WHERE rolname = 'debezium_user''
[0;34m[INFO][0m Creating role debezium_user...
[2026-01-18 23:41:41 UTC] USER=www-data EUID=0 PID=4050850 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c CREATE ROLE debezium_user LOGIN PASSWORD '5WF4RDzreIF8CwuTXQIgebIK';
CREATE ROLE
[0;32m[OK][0m Role debezium_user created
[2026-01-18 23:41:41 UTC] USER=www-data EUID=0 PID=4050887 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c GRANT CONNECT ON DATABASE "fastorder_user_sau_main_dev_db" TO debezium_user;
GRANT
[0;32m[OK][0m Application DB (fastorder_user_sau_main_dev_db) + Debezium role (debezium_user) provisioned (idempotent)
[0;34m[INFO][0m Applying connection and memory optimizations...
[0;34m[INFO][0m Current settings: max_connections=100, work_mem=4MB
[0;34m[INFO][0m Target settings (worker): max_connections=100, work_mem=8MB
[2026-01-18 23:41:42 UTC] USER=www-data EUID=0 PID=4050988 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET max_connections = 100;
ALTER SYSTEM
[2026-01-18 23:41:42 UTC] USER=www-data EUID=0 PID=4051011 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET work_mem = '8MB';
ALTER SYSTEM
[2026-01-18 23:41:42 UTC] USER=www-data EUID=0 PID=4051037 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
pg_reload_conf
----------------
t
(1 row)
[0;32m[OK][0m Settings applied to postgresql.auto.conf
[2026-01-18 23:41:42 UTC] USER=www-data EUID=0 PID=4051052 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/user-sau-main-dev/worker-01/standby.signal
[0;34m[INFO][0m Service recently started (3s ago) - restarting to apply max_connections...
[0;34m[INFO][0m Stopping service...
[2026-01-18 23:41:42 UTC] USER=www-data EUID=0 PID=4051074 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-worker-01.service
[0;34m[INFO][0m Waiting for port 5432 to be released...
[0;32m[OK][0m Port 5432 released
[0;34m[INFO][0m Starting service...
[2026-01-18 23:41:46 UTC] USER=www-data EUID=0 PID=4051143 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-worker-01.service
[2026-01-18 23:41:51 UTC] USER=www-data EUID=0 PID=4051278 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-worker-01.service
[0;32m[OK][0m ✅ Optimization complete: max_connections=100, work_mem=8MB
[0;32m[OK][0m Synchronous replication already configured (synchronous_commit: on)
[0;34m[INFO][0m Setting postgres password via centralized script... for worker-01
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: user[0m
[0;34m Zone: sau[0m
[0;34m Environment: dev[0m
[0;34m Identifier: worker-01[0m
[0;34mAWS Secret: fastorder/db/user/sau/main/dev/postgresql/worker-01[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-user-sau-main-dev-worker-01[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mInitial setup: Using password from initdb[0m
[0;32m✓ PostgreSQL password already set during initdb[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/worker-01
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/user/sau/main/dev/postgresql/worker-01[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials worker-01[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
[0;32m[OK][0m Password set and persisted
[0;34m[INFO][0m Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Identifier: worker-01
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.232
[0;34m[INFO][0m Primary hostname: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[0;34m[INFO][0m Adding /etc/hosts entry for worker-01...
[0;34m[INFO][0m db-user-sau-main-dev-postgresql-worker-01.fastorder.com → 10.100.1.232
[0;34m[INFO][0m ✅ db-user-sau-main-dev-postgresql-worker-01.fastorder.com already exists with correct IP
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
10.100.1.232 db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[0;32m[OK][0m PostgreSQL 'user-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt \
sslcert=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.crt \
sslkey=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/postgres.key \
host=db-user-sau-main-dev-postgresql-worker-01 port=5432 dbname=postgres user=postgres"
File been compeleted perfectly: 02-setup-pg-instance
[0;34m[INFO][0m Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: PostgreSQL
[INFO] Identifier: user-sau-main-dev-postgresql-worker-01
[INFO] Identifier Parent: worker-01
[INFO] IP: 10.100.1.232
[INFO] Port: 5432
[INFO] FQDN: db-user-sau-main-dev-postgresql-worker-01
[INFO] Status: running
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 87ccba48-d8e0-43e4-97b8-d87917a5d35c
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[0;32m[OK][0m PostgreSQL node registered to observability API
[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 📦 03 role...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-18 23:42:00 UTC] USER=www-data EUID=0 PID=4051947 ACTION=fsop ARGS=test -f /data/postgresql/17/user-sau-main-dev/worker-01/standby.signal
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username: debezium_user
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau
Branch: main
Env: dev
Node: worker-01
User (CN): debezium_user
Hostname: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:42:01 UTC] USER=www-data EUID=0 PID=4052130 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-debezium_user
[2026-01-18 23:42:01 UTC] USER=www-data EUID=0 PID=4052140 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-01-18 23:42:01 UTC] USER=www-data EUID=0 PID=4052157 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-debezium_user/ra_root.key
[2026-01-18 23:42:01 UTC] USER=www-data EUID=0 PID=4052170 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.crt
[2026-01-18 23:42:01 UTC] USER=www-data EUID=0 PID=4052185 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-debezium_user/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = debezium_user
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052228 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052244 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052262 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052271 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052299 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052308 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-debezium_user/debezium_user_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_pk8.der
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052317 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052326 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052335 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052345 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052356 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052365 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052374 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_pk8.der
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052383 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052392 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052418 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052427 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052436 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052445 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052461 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052470 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052479 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052488 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052506 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052515 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052524 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_pk8.der
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052534 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:02 UTC] USER=www-data EUID=0 PID=4052544 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052553 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052562 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052571 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052580 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052589 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052598 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052607 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052616 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052625 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052634 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052648 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_pk8.der
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052658 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052668 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052677 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052695 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052704 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052713 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052722 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052731 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052743 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052752 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key.pkcs1
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052761 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052770 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_pk8.der
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052780 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052790 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052799 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052817 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052826 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:03 UTC] USER=www-data EUID=0 PID=4052862 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:04 UTC] USER=www-data EUID=0 PID=4052880 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_der.key
[2026-01-18 23:42:04 UTC] USER=www-data EUID=0 PID=4052889 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/debezium_user_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user_pk8.der
[2026-01-18 23:42:04 UTC] USER=www-data EUID=0 PID=4052899 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:04 UTC] USER=www-data EUID=0 PID=4052909 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:04 UTC] USER=www-data EUID=0 PID=4052918 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:04 UTC] USER=www-data EUID=0 PID=4052927 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-01-18 23:42:04 UTC] USER=www-data EUID=0 PID=4052936 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
✅ Symlinked client-key.pem
[2026-01-18 23:42:04 UTC] USER=www-data EUID=0 PID=4052954 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:04 UTC] USER=www-data EUID=0 PID=4052963 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:42:04 UTC] USER=www-data EUID=0 PID=4052974 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: user-sau-main-dev
User: debezium_user
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt
database.sslkey: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/debezium_user.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U debezium_user -d postgres
🔐 Generating replicator client certificate for worker-01...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username: replicator
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau
Branch: main
Env: dev
Node: worker-01
User (CN): replicator
Hostname: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:42:04 UTC] USER=www-data EUID=0 PID=4053042 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-18 23:42:04 UTC] USER=www-data EUID=0 PID=4053071 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-18 23:42:04 UTC] USER=www-data EUID=0 PID=4053080 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053095 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053104 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053113 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053122 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053131 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053140 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053149 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053158 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053167 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053176 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053185 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053194 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053203 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053212 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053221 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053230 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053239 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053248 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053257 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053266 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053275 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053326 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053341 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053368 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053387 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:42:05 UTC] USER=www-data EUID=0 PID=4053397 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053415 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053424 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053433 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053442 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053452 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053471 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053480 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053489 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053499 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053508 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053517 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053535 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053544 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053553 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053562 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053572 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053583 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053594 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053603 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053612 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053632 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053650 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053659 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053668 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053677 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-18 23:42:06 UTC] USER=www-data EUID=0 PID=4053686 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053696 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053706 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053715 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053724 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053733 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053742 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053751 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053760 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053769 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053778 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053816 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053842 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053852 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053861 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053870 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053879 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053896 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053905 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053914 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053923 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:42:07 UTC] USER=www-data EUID=0 PID=4053933 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: user-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
database.sslkey: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres
✅ Replicator certificate generated for worker-01
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
📦 Start executing 03-create-role.sh
📦 Setting password for user: fastorder_admin_gd
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: user[0m
[0;34m Zone: sau[0m
[0;34m Environment: dev[0m
[0;34m Identifier: worker-01[0m
[0;34mAWS Secret: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-user-sau-main-dev-worker-01[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mGenerating new secure password...[0m
[0;34mUser fastorder_admin_gd does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: fastorder_admin_gd[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials worker-01[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: worker-01/fastorder_admin_gd
✓ Retrieved password from centralized secrets vault
🌐 Using PostgreSQL host: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username: fastorder_admin_gd
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau
Branch: main
Env: dev
Node: worker-01
User (CN): fastorder_admin_gd
Hostname: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:42:15 UTC] USER=www-data EUID=0 PID=4054339 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-fastorder_admin_gd
[2026-01-18 23:42:15 UTC] USER=www-data EUID=0 PID=4054348 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt
[2026-01-18 23:42:15 UTC] USER=www-data EUID=0 PID=4054357 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
[2026-01-18 23:42:15 UTC] USER=www-data EUID=0 PID=4054375 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = fastorder_admin_gd
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:42:15 UTC] USER=www-data EUID=0 PID=4054390 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:42:15 UTC] USER=www-data EUID=0 PID=4054399 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:42:15 UTC] USER=www-data EUID=0 PID=4054408 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-18 23:42:15 UTC] USER=www-data EUID=0 PID=4054417 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-18 23:42:15 UTC] USER=www-data EUID=0 PID=4054426 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:15 UTC] USER=www-data EUID=0 PID=4054435 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:15 UTC] USER=www-data EUID=0 PID=4054444 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-18 23:42:15 UTC] USER=www-data EUID=0 PID=4054453 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-fastorder_admin_gd/fastorder_admin_gd_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-18 23:42:15 UTC] USER=www-data EUID=0 PID=4054471 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054489 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054498 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054507 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054516 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054525 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054534 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054543 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054552 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054561 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054570 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054596 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054614 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054623 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054650 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054659 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054669 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054678 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054696 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054716 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054734 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054745 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054765 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054791 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054800 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054809 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054818 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_der.key
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054827 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054837 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:16 UTC] USER=www-data EUID=0 PID=4054847 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4054865 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4054874 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4054883 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4054892 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4054901 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4054910 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4054919 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4054928 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key.pkcs1
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4054947 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4054958 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4054968 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4054977 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4054986 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4054996 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4055005 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4055014 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4055023 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4055032 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4055052 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4055085 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd_pk8.der
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4055095 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4055105 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4055114 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4055123 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
✅ Symlinked ca.pem
✅ Symlinked client-cert.pem
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4055141 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4055150 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4055159 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4055168 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:42:17 UTC] USER=www-data EUID=0 PID=4055177 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: user-sau-main-dev
User: fastorder_admin_gd
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt
database.sslkey: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/fastorder_admin_gd.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U fastorder_admin_gd -d postgres
🧱 Connecting via Unix socket to create role and database...
Socket: /var/run/postgresql-user-sau-main-dev-worker-01:5432
📦 Creating role fastorder_admin_gd...
✅ Role fastorder_admin_gd created
ℹ️ Database fastorder_user_sau_main_dev_db already exists, skipping creation
[2026-01-18 23:42:18 UTC] USER=www-data EUID=0 PID=4055235 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
GRANT
✅ Role and DB created via SSL
🔐 Adding user to pg_hba.conf for SSL access...
ℹ️ Using pg_hba.conf: /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
✅ Added fastorder_admin_gd to pg_hba.conf
🔄 Reloading PostgreSQL configuration...
[2026-01-18 23:42:18 UTC] USER=www-data EUID=0 PID=4055272 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-worker-01.service
✅ PostgreSQL configuration reloaded
🧪 Testing connection for user: fastorder_admin_gd
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[0;34m=== Pre-flight Checks ===[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;32m✓[0m AWS Secrets Manager accessible
[0;34m=== Retrieving Credentials from AWS ===[0m
ℹ️ Retrieving PostgreSQL credentials for: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
ℹ️ Fetching secret: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ Retrieved from cache: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
✅ PostgreSQL credentials loaded for worker-01/fastorder_admin_gd: fastorder_admin_gd@db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_user_sau_main_dev_db
[0;32m✓[0m Credentials retrieved: fastorder_admin_gd@db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432/fastorder_user_sau_main_dev_db
[0;34m╔════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Test Suite (AWS Secrets MGR) ║[0m
[0;34m╚════════════════════════════════════════════╝[0m
[0;34m=== PostgreSQL Authentication Test ===[0m
[0;31m✗[0m PostgreSQL authentication failed
---- Error Details ----
psql: error: connection to server at "db-user-sau-main-dev-postgresql-worker-01.fastorder.com" (10.100.1.232), port 5432 failed: root certificate file "/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/fastorder_admin_gd/root.crt" does not exist
Either provide the file, use the system's trusted roots with sslrootcert=system, or change sslmode to disable server certificate verification.
----------------------
❌ User authentication test failed
📋 Password stored securely in AWS Secrets Manager
📋 Secret path: fastorder/db/user/sau/main/dev/postgresql/worker-01/fastorder_admin_gd
📦 End executing 03-create-role.sh
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-18 23:42:23 UTC] USER=www-data EUID=0 PID=4055535 ACTION=fsop ARGS=test -f /data/postgresql/17/user-sau-main-dev/worker-01/standby.signal
── fast setup ─────────────────────────────────────────────
NAME : user-sau-main-dev
IDENTIFIER : worker-01
PG HOST : db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432
ROLE : debezium_user
DB : fastorder_user_sau_main_dev_db
SCHEMA : user
AUTH MODE : scram (scram=password over TLS | cert=mTLS)
SUBNET ALLOW: 10.201.0.0/16
CONNECT /32 : 142.93.238.16
SSL DIR : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
DNS → 10.100.1.232
CA : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
🔐 Setting password for user: debezium_user
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
⚠️ ~/.aws/credentials file not found
⚠️ Using environment-based AWS authentication
[0;34m╔════════════════════════════════════════════════════════════╗[0m
[0;34m║ PostgreSQL Password Rotation via AWS Secrets Manager ║[0m
[0;34m╚════════════════════════════════════════════════════════════╝[0m
[0;34mEnvironment Configuration:[0m
[0;34m Service: user[0m
[0;34m Zone: sau[0m
[0;34m Environment: dev[0m
[0;34m Identifier: worker-01[0m
[0;34mAWS Secret: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user[0m
[0;34mConnection Info:[0m
[0;34m Socket Dir: /var/run/postgresql-user-sau-main-dev-worker-01[0m
[0;34m Port: 5432[0m
[0;34mTesting AWS Secrets Manager connectivity...[0m
ℹ️ Testing AWS IAM credentials...
✅ AWS IAM credentials are valid
{
"UserId": "AIDAWYLM4MSHFSCGU7QUM",
"Account": "464621692046",
"Arn": "arn:aws:iam::464621692046:user/fo-dev"
}
[0;34mMethod 1 (PREFERRED): AWS Secrets Manager Rotation[0m
[0;34m────────────────────────────────────────────────────────────[0m
[0;34mThis method uses AWS Secrets Manager's built-in rotation:[0m
[0;34m ✓ Zero-downtime (dual-password window)[0m
[0;34m ✓ Automatic rollback on failure[0m
[0;34m ✓ CloudTrail audit log[0m
[0;34m ✓ CloudWatch metrics[0m
[0;34m ✓ No secret exposure in scripts[0m
[0;34mNon-interactive mode: Proceeding with password rotation automatically[0m
[0;34mGenerating new secure password...[0m
[0;34mUser debezium_user does not exist yet - skipping ALTER, will be created by calling script[0m
[0;32m✓ Password generated for new user: debezium_user[0m
[0;34mStoring password in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user[0m
ℹ️ Setting PostgreSQL credentials in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user
ℹ️ Setting secret in AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user
✅ Secret updated: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user
✅ PostgreSQL credentials set in vault: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user
[0;32m✓ Password stored in AWS Secrets Manager[0m
[0;34mVerifying new credentials...[0m
[0;32m✓ New credentials retrieved from AWS Secrets Manager[0m
[0;34mTesting PostgreSQL connection with new credentials...[0m
[0;32m✓ PostgreSQL connection successful (socket authentication)[0m
[0;32m✓ ╔════════════════════════════════════════════════════════════╗[0m
[0;32m✓ ║ Password Rotation Complete! ║[0m
[0;32m✓ ╚════════════════════════════════════════════════════════════╝[0m
[0;34mSecret: fastorder/db/user/sau/main/dev/postgresql/worker-01/debezium_user[0m
[0;34mMethod: Direct Update (stored in AWS Secrets Manager)[0m
[0;34mStatus: Completed[0m
[0;34mTo retrieve credentials:[0m
[0;34m # Using Bash library[0m
[0;34m source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh[0m
[0;34m get_pg_credentials worker-01[0m
[0;34mAudit trail: AWS CloudTrail (for Secrets Manager operations)[0m
[0;32m✓ Done![0m
🔍 Retrieving password from vault with identifier: worker-01/debezium_user
✓ Retrieved password from secrets vault
password : (stored in AWS Secrets Manager)
🔍 TLS chain check...
🔧 Ensuring role and grants…
ℹ️ Role debezium_user exists, updating
[2026-01-18 23:42:29 UTC] USER=www-data EUID=0 PID=4055954 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc
ALTER ROLE
ℹ️ Database fastorder_user_sau_main_dev_db already exists
[2026-01-18 23:42:29 UTC] USER=www-data EUID=0 PID=4055981 ACTION=passthru ARGS=sudo -u postgres psql -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d fastorder_user_sau_main_dev_db --no-psqlrc
ERROR: syntax error at or near "user"
LINE 1: CREATE SCHEMA IF NOT EXISTS user;
^
GRANT
ERROR: syntax error at or near "user"
LINE 1: GRANT USAGE ON SCHEMA user TO debezium_user;
^
ERROR: syntax error at or near "user"
LINE 1: GRANT SELECT ON ALL TABLES IN SCHEMA user TO debezium_user;
^
ERROR: syntax error at or near "user"
LINE 1: GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA user TO debez...
^
ERROR: syntax error at or near "user"
LINE 1: ALTER DEFAULT PRIVILEGES IN SCHEMA user GRANT SELECT ON TABL...
^
✅ Role/DB/grants ensured.
⚠️ Could not find pg_hba.conf (skipping HBA edits): /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
🧪 Testing ROLE connection (scram)...
✅ SCRAM+TLS probe OK
🎉 Done.
🔐 Creating replicator role for worker-01...
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
NAME : user-sau-main-dev
IDENTIFIER : worker-01
PG HOST : db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432
ROLE : replicator
SSL DIR : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
DNS → 10.100.1.232
CA : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️ Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE: Creating role: replicator with password
SET
CREATE ROLE
✅ Replicator role ensured with password authentication.
ℹ️ Password stored in: AWS Secrets Manager
Secret name: fastorder/db/user/sau/main/dev/postgresql/replicator
🔄 MIGRATION PATH: Password → Certificate Authentication
Current: SCRAM-SHA-256 password auth (production-ready)
Future: Certificate-based auth (requires CA automation)
To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
and configure standby to use SSL certificates instead of password
🎉 Done.
✅ Replicator role created for worker-01
[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
ℹ️ Service-specific setup (user) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)
🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=user
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/user/run.sh
🔍 DEBUG_CHECKPOINT_03: No specific folder for user, using default
[DEBUG] Tracking substep start: steps/01-install/steps/default (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 🔸 Service: user (using default contracts schema)
🔍 DEBUG_CHECKPOINT_04: Executing default: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/run.sh
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting default contracts schema provisioning for SERVICE=user
[INFO] Environment: user-sau-main-dev
[INFO] Schema: user (contracts tables)
[INFO] Identifier: worker-01
[INFO] VM IP: 142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Skipping Schema Setup on worker-01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ℹ️ Schema setup only runs on coordinator
ℹ️ This is a worker-01 node - schemas replicate automatically
✅ Nothing to do on this node
[0;32m✓[0m ✅ Worker worker-01 setup completed
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Setting up standby replicas (1 per worker)…
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
→ Setting up standby: worker-01-standby-01 (replica of worker-01)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 📁 Initializing log directories...
[2026-01-18 23:42:34 UTC] USER=unknown EUID=33 PID=4056212 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-18 23:42:34 UTC] USER=unknown EUID=33 PID=4056219 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-18 23:42:34 UTC] USER=unknown EUID=33 PID=4056226 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/provisioning
[2026-01-18 23:42:34 UTC] USER=unknown EUID=33 PID=4056233 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-18 23:42:34 UTC] USER=unknown EUID=33 PID=4056240 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[2026-01-18 23:42:34 UTC] USER=unknown EUID=33 PID=4056247 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/provisioning
/opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/run.sh: line 41: ok: command not found
[INFO] 🟢 Starting PostgreSQL provisioning for user in sau-dev...
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] VM IP: 142.93.238.16
[DEBUG] RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61 JOB_UUID=c535671c-4f96-43e7-95ee-3e02cbcf2d2f
[DEBUG] Tracking substep start: steps/01-install/steps/00-configure-network-hosts (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 📦 00 configure network hosts...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[WARN] Could not find PostgreSQL IP for worker-01-standby-01 in topology.json, allocating new VM IP...
[INFO] Allocated new VM IP: 10.100.1.233 for db-worker-01-standby-01-postgresql
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] CONFIGURING POSTGRESQL NETWORK & DNS
[INFO] ═══════════════════════════════════════════════════════════════
[INFO] Environment: user-sau-main-dev
[INFO] Identifier: worker-01-standby-01
[INFO] PostgreSQL IP: 10.100.1.233
[INFO] Primary hostname: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
[INFO] Adding /etc/hosts entry for worker-01-standby-01...
[INFO] db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.233
[INFO] ➕ Adding db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.233
[0;32m✅[0m ✅ Added: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.233
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[INFO] Verifying /etc/hosts entries:
10.100.1.233 db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
[DEBUG] Tracking substep start: steps/01-install/steps/01-prepare-ssl-server-postgres (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 📦 01 prepare ssl server postgres...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau (Saudi Arabia)
Branch: main
Env: dev
Node: worker-01-standby-01
Primary CN: user-sau-main-dev.fastorder.com
Alt CN: user-sau-main-dev.fastorder.com
VM IP: 142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-18 23:42:37 UTC] USER=www-data EUID=0 PID=4056885 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:37 UTC] USER=www-data EUID=0 PID=4056910 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-18 23:42:37 UTC] USER=www-data EUID=0 PID=4056920 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-4056852
[2026-01-18 23:42:37 UTC] USER=www-data EUID=0 PID=4056929 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-4056852/ra_root.crt
[2026-01-18 23:42:38 UTC] USER=www-data EUID=0 PID=4056938 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-4056852/ra_root.key
[2026-01-18 23:42:38 UTC] USER=www-data EUID=0 PID=4056947 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4056852/ra_root.crt
[2026-01-18 23:42:38 UTC] USER=www-data EUID=0 PID=4056956 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4056852/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = user-sau-main-dev.fastorder.com
[2026-01-18 23:42:39 UTC] USER=www-data EUID=0 PID=4057065 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4056852/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
[2026-01-18 23:42:39 UTC] USER=www-data EUID=0 PID=4057074 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4056852/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-18 23:42:39 UTC] USER=www-data EUID=0 PID=4057083 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-18 23:42:39 UTC] USER=www-data EUID=0 PID=4057092 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4056852/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-18 23:42:39 UTC] USER=www-data EUID=0 PID=4057101 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-18 23:42:39 UTC] USER=www-data EUID=0 PID=4057110 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-18 23:42:39 UTC] USER=www-data EUID=0 PID=4057119 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-18 23:42:39 UTC] USER=www-data EUID=0 PID=4057130 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
[2026-01-18 23:42:39 UTC] USER=www-data EUID=0 PID=4057139 ACTION=fsop ARGS=chmod 600 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
[2026-01-18 23:42:39 UTC] USER=www-data EUID=0 PID=4057148 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-18 23:42:39 UTC] USER=www-data EUID=0 PID=4057157 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-18 23:42:39 UTC] USER=www-data EUID=0 PID=4057167 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:39 UTC] USER=www-data EUID=0 PID=4057176 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
🔍 Verifying certificate...
Certificate details:
Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = user-sau-main-dev.fastorder.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
--
X509v3 Subject Alternative Name:
DNS:user-sau-main-dev.fastorder.com, DNS:user-sau-main-dev.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
X509v3 Subject Key Identifier:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Node: worker-01-standby-01
Primary CN: user-sau-main-dev.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-worker-01-standby-01.service
3. Test SSL connection:
psql "host=user-sau-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username: postgres
Identifier: worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau
Branch: main
Env: dev
Node: worker-01-standby-01
User (CN): postgres
Hostname: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:40 UTC] USER=www-data EUID=0 PID=4057289 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:40 UTC] USER=www-data EUID=0 PID=4057298 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:40 UTC] USER=www-data EUID=0 PID=4057307 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-18 23:42:40 UTC] USER=www-data EUID=0 PID=4057316 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-18 23:42:40 UTC] USER=www-data EUID=0 PID=4057325 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-18 23:42:40 UTC] USER=www-data EUID=0 PID=4057334 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-18 23:42:40 UTC] USER=www-data EUID=0 PID=4057343 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-18 23:42:40 UTC] USER=www-data EUID=0 PID=4057352 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-18 23:42:40 UTC] USER=www-data EUID=0 PID=4057397 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-18 23:42:40 UTC] USER=www-data EUID=0 PID=4057434 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-18 23:42:40 UTC] USER=www-data EUID=0 PID=4057444 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-18 23:42:40 UTC] USER=www-data EUID=0 PID=4057453 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:40 UTC] USER=www-data EUID=0 PID=4057479 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:40 UTC] USER=www-data EUID=0 PID=4057488 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:40 UTC] USER=www-data EUID=0 PID=4057497 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:40 UTC] USER=www-data EUID=0 PID=4057506 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057515 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057524 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057533 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057542 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057551 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057560 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057569 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057578 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057588 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057598 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057607 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057616 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057625 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057643 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057652 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057670 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057679 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057697 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057718 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057727 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057745 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057754 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057763 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057772 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057790 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057801 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057810 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057819 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057829 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057839 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057848 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057857 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057866 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057875 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057884 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057893 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057902 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057911 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057920 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057929 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-18 23:42:41 UTC] USER=www-data EUID=0 PID=4057938 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4057948 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4057958 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4057967 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
✅ Symlinked ca.pem
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4057987 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4058004 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4058014 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4058023 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4058032 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4058041 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: user-sau-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username: postgres
Identifier: worker-01-standby-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau
Branch: main
Env: dev
Node: worker-01-standby-01
User (CN): postgres
Hostname: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4058084 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-standby-01-postgres
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4058102 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4058120 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-standby-01-postgres/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = postgres
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4058136 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4058154 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4058163 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4058172 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4058181 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-18 23:42:42 UTC] USER=www-data EUID=0 PID=4058190 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058199 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-standby-01-postgres/postgres_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058217 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058226 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058235 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058244 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058262 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058271 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058280 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058289 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058298 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058307 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058350 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058359 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058377 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058395 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058404 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058413 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058422 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058431 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058449 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058459 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058478 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058487 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058505 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058514 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058523 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058532 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058541 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058551 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058560 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058569 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-18 23:42:43 UTC] USER=www-data EUID=0 PID=4058579 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058589 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058598 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058607 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058616 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058625 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058634 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058643 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058661 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058670 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058679 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058688 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058698 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058708 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058726 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058735 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058744 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058753 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058762 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058771 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058780 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058789 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key.pkcs1
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058798 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_der.key
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058807 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres_pk8.der
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058817 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058827 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058836 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058845 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058854 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058865 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058874 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058892 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:42:44 UTC] USER=www-data EUID=0 PID=4058901 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: user-sau-main-dev
User: postgres
Node: worker-01-standby-01
FQDN: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt
database.sslkey: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com -U postgres -d postgres
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username: replicator
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau
Branch: main
Env: dev
Node: worker-01
User (CN): replicator
Hostname: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:42:45 UTC] USER=www-data EUID=0 PID=4058942 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-18 23:42:45 UTC] USER=www-data EUID=0 PID=4058953 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-18 23:42:45 UTC] USER=www-data EUID=0 PID=4058969 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-18 23:42:45 UTC] USER=www-data EUID=0 PID=4058979 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-18 23:42:45 UTC] USER=www-data EUID=0 PID=4058988 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:42:45 UTC] USER=www-data EUID=0 PID=4059005 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:42:45 UTC] USER=www-data EUID=0 PID=4059023 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:42:45 UTC] USER=www-data EUID=0 PID=4059044 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt
[2026-01-18 23:42:45 UTC] USER=www-data EUID=0 PID=4059053 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:45 UTC] USER=www-data EUID=0 PID=4059062 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:45 UTC] USER=www-data EUID=0 PID=4059071 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:42:45 UTC] USER=www-data EUID=0 PID=4059080 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-18 23:42:45 UTC] USER=www-data EUID=0 PID=4059089 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-18 23:42:45 UTC] USER=www-data EUID=0 PID=4059107 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059116 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059134 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059152 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059161 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059170 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059197 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059223 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059232 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059241 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059250 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059259 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059268 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059277 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059286 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059295 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059324 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059334 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059353 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059362 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059400 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059409 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059427 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059443 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059455 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059473 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059490 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059511 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:46 UTC] USER=www-data EUID=0 PID=4059527 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059545 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059554 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059583 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059592 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059601 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059610 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059619 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059628 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059638 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059658 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059667 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059676 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059686 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059695 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059704 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059713 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059722 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059731 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059740 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059759 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059769 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059780 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:47 UTC] USER=www-data EUID=0 PID=4059789 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
✅ Symlinked ca.pem
✅ Symlinked client-cert.pem
[2026-01-18 23:42:48 UTC] USER=www-data EUID=0 PID=4059822 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-01-18 23:42:48 UTC] USER=www-data EUID=0 PID=4059831 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:42:48 UTC] USER=www-data EUID=0 PID=4059840 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:42:48 UTC] USER=www-data EUID=0 PID=4059849 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:42:48 UTC] USER=www-data EUID=0 PID=4059858 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: user-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
database.sslkey: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres
[DEBUG] Tracking substep start: steps/01-install/steps/02-setup-pg-instance (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 📦 02 setup pg instance...
[0;34m[DEADLOCK-PREVENTION][0m Deadlock prevention library loaded
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔑 Configuring AWS credentials...
[WARN] ~/.aws/credentials file not found
[WARN] AWS operations may require SSO login
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Using existing db-worker-01-standby-01-postgresql environment: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com (10.100.1.233)
[INFO] PostgreSQL will listen on application-specific IP: 10.100.1.233
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Identifier: worker-01-standby-01
[0;34m[INFO][0m Data dir: /data/postgresql/17/user-sau-main-dev/worker-01-standby-01
[0;34m[INFO][0m Port: 5432
[0;34m[INFO][0m Hostname: db-user-sau-main-dev-postgresql-worker-01-standby-01
[2026-01-18 23:42:49 UTC] USER=www-data EUID=0 PID=4059971 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:49 UTC] USER=www-data EUID=0 PID=4059997 ACTION=fsop ARGS=chmod 755 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:49 UTC] USER=www-data EUID=0 PID=4060019 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:49 UTC] USER=www-data EUID=0 PID=4060040 ACTION=fsop ARGS=chown root:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[1;33m[WARN][0m Server certificate not found at /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[0;34m[INFO][0m Generating server certificate using ssl/server.sh...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📦 PostgreSQL Server Certificate Generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau (Saudi Arabia)
Branch: main
Env: dev
Node: worker-01-standby-01
Primary CN: user-sau-main-dev.fastorder.com
Alt CN: user-sau-main-dev.fastorder.com
VM IP: 142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Removing existing server certificates (preserving client certs)...
[2026-01-18 23:42:50 UTC] USER=www-data EUID=0 PID=4060087 ACTION=fsop ARGS=rm -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
✅ Ensuring directories exist: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01 and /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:42:50 UTC] USER=www-data EUID=0 PID=4060106 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01
🔐 Generating 4096-bit private key...
[2026-01-18 23:42:50 UTC] USER=www-data EUID=0 PID=4060123 ACTION=fsop ARGS=chmod 755 /tmp/pg-cert-gen-4060047
[2026-01-18 23:42:50 UTC] USER=www-data EUID=0 PID=4060132 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-cert-gen-4060047/ra_root.crt
[2026-01-18 23:42:50 UTC] USER=www-data EUID=0 PID=4060141 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-cert-gen-4060047/ra_root.key
[2026-01-18 23:42:50 UTC] USER=www-data EUID=0 PID=4060150 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4060047/ra_root.crt
[2026-01-18 23:42:50 UTC] USER=www-data EUID=0 PID=4060159 ACTION=fsop ARGS=chmod 644 /tmp/pg-cert-gen-4060047/ra_root.key
📝 Creating certificate signing request (CSR)...
📜 Signing certificate with internal CA...
Certificate request self-signature ok
subject=C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = user-sau-main-dev.fastorder.com
[2026-01-18 23:42:51 UTC] USER=www-data EUID=0 PID=4060207 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4060047/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
[2026-01-18 23:42:51 UTC] USER=www-data EUID=0 PID=4060216 ACTION=fsop ARGS=cp /tmp/pg-cert-gen-4060047/server.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-18 23:42:51 UTC] USER=www-data EUID=0 PID=4060227 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
📋 Setting up CA certificate...
[2026-01-18 23:42:51 UTC] USER=www-data EUID=0 PID=4060253 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt
✅ Using CA certificate: /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt
🚚 Setting up key in private directory...
Key already in correct location (CERT_DIR == KEY_DIR)
🔒 Securing key and cert permissions...
[2026-01-18 23:42:51 UTC] USER=www-data EUID=0 PID=4060302 ACTION=fsop ARGS=chown postgres:postgres /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
🔍 Verifying certificate...
Certificate details:
Subject: C = SA, ST = Riyadh, L = Riyadh, O = FastOrder, OU = PostgreSQL, CN = user-sau-main-dev.fastorder.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
--
X509v3 Subject Alternative Name:
DNS:user-sau-main-dev.fastorder.com, DNS:user-sau-main-dev.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, DNS:db-user-sau-main-dev-postgresql-worker-01-standby-01, DNS:localhost, DNS:db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com, IP Address:142.93.238.16, IP Address:127.0.0.1
X509v3 Subject Key Identifier:
⚠️ Certificate chain verification: FAILED (but certificate may still work)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL Server Certificate Generated Successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Node: worker-01-standby-01
Primary CN: user-sau-main-dev.fastorder.com
Certificate files installed:
📜 Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
🔑 Server key: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
🏛️ CA cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt (ca.crt symlink also available)
To use these certificates in PostgreSQL:
1. Update postgresql.conf:
ssl = on
ssl_cert_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt'
ssl_key_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key'
ssl_ca_file = '/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/root.crt'
2. Restart PostgreSQL:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart postgresql@user-sau-main-dev-worker-01-standby-01.service
3. Test SSL connection:
psql "host=user-sau-main-dev.fastorder.com port=5432 user=postgres sslmode=verify-full"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ✅ Using canonical certificate path (hardened, ProtectHome=true compatible)
[2026-01-18 23:42:51 UTC] USER=www-data EUID=0 PID=4060358 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.crt
[2026-01-18 23:42:51 UTC] USER=www-data EUID=0 PID=4060367 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/server.key
[2026-01-18 23:42:51 UTC] USER=www-data EUID=0 PID=4060376 ACTION=fsop ARGS=test -f /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt
[0;32m[OK][0m mTLS certificates OK (server cert + client certs verified) and keys secured
[0;34m[INFO][0m Preflight: stopping any conflicting Postgres services/processes on port 5432…
[2026-01-18 23:42:51 UTC] USER=www-data EUID=0 PID=4060397 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-worker-01-standby-01.service
[2026-01-18 23:42:52 UTC] USER=www-data EUID=0 PID=4060418 ACTION=passthru ARGS=systemctl stop postgresql
[1;33m[WARN][0m Cleaning stale socket directory /var/run/postgresql-user-sau-main-dev-worker-01-standby-01
[2026-01-18 23:42:52 UTC] USER=www-data EUID=0 PID=4060449 ACTION=fsop ARGS=rm -rf /var/run/postgresql-user-sau-main-dev-worker-01-standby-01
[0;32m[OK][0m No conflicting Postgres left on port 5432
[0;32m[OK][0m Generated new postgres password for initdb
[2026-01-18 23:43:13 UTC] USER=www-data EUID=0 PID=4061068 ACTION=fsop ARGS=chown postgres:postgres /tmp/.pg_pwfile.ixntXx
[2026-01-18 23:43:13 UTC] USER=www-data EUID=0 PID=4061116 ACTION=fsop ARGS=chmod 600 /tmp/.pg_pwfile.ixntXx
[2026-01-18 23:43:13 UTC] USER=www-data EUID=0 PID=4061161 ACTION=fsop ARGS=mkdir -p /data/postgresql/17/user-sau-main-dev
[2026-01-18 23:43:13 UTC] USER=www-data EUID=0 PID=4061185 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev
[2026-01-18 23:43:13 UTC] USER=www-data EUID=0 PID=4061207 ACTION=fsop ARGS=chmod 755 /data/postgresql/17/user-sau-main-dev
[0;34m[INFO][0m This is a standby. Using pg_basebackup from primary (worker-01)...
[0;34m[INFO][0m Setting up replicator role and slot on primary (worker-01)...
ℹ️ Scanning primary for stuck queries from previous failed attempts...
ℹ️ Scanning for stuck queries (timeout: 30s)...
ℹ️ No stuck queries found
[WARN] Deadlock prevention library not found: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/role/lib/pg-deadlock-prevention.sh
🔑 Configuring AWS credentials...
✅ Using permanent AWS credentials from /var/www/.aws/credentials
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
── replicator setup ───────────────────────────────────────
NAME : user-sau-main-dev
IDENTIFIER : worker-01
PG HOST : db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432
ROLE : replicator
SLOT : worker_01_standby_01
SSL DIR : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
DNS → 10.100.1.232
CA : /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
🔍 TLS chain check...
🔧 Ensuring replicator role…
🔐 Checking AWS Secrets Manager for replicator password...
✅ Retrieved replicator password from AWS Secrets Manager
ℹ️ Temporarily disabling synchronous_commit to prevent replication deadlock...
NOTICE: Role replicator already exists, updating password and ensuring REPLICATION privilege
SET
ALTER ROLE
✅ Replicator role ensured with password authentication.
ℹ️ Password stored in: AWS Secrets Manager
Secret name: fastorder/db/user/sau/main/dev/postgresql/replicator
🔄 MIGRATION PATH: Password → Certificate Authentication
Current: SCRAM-SHA-256 password auth (production-ready)
Future: Certificate-based auth (requires CA automation)
To migrate: Update pg_hba.conf rules from 'scram-sha-256' to 'cert clientcert=verify-full'
and configure standby to use SSL certificates instead of password
🔧 Ensuring replication slot: worker_01_standby_01…
🆕 Creating replication slot worker_01_standby_01
SET
pg_create_physical_replication_slot
-------------------------------------
(worker_01_standby_01,)
(1 row)
✅ Replication slot worker_01_standby_01 created.
🎉 Done.
[0;32m[OK][0m Replicator role and slot created on primary
[0;34m[INFO][0m Creating replicator client certificates for connecting to primary (worker-01)...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username: replicator
Identifier: worker-01
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau
Branch: main
Env: dev
Node: worker-01
User (CN): replicator
Hostname: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:43:16 UTC] USER=www-data EUID=0 PID=4061368 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-worker-01-replicator
[2026-01-18 23:43:16 UTC] USER=www-data EUID=0 PID=4061388 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-18 23:43:16 UTC] USER=www-data EUID=0 PID=4061397 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-worker-01-replicator/ra_root.key
[2026-01-18 23:43:16 UTC] USER=www-data EUID=0 PID=4061406 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.crt
[2026-01-18 23:43:16 UTC] USER=www-data EUID=0 PID=4061420 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-worker-01-replicator/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = replicator
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061435 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061456 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061473 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061482 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061492 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061501 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061511 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061520 ACTION=fsop ARGS=cp -f /tmp/pg-client-worker-01-replicator/replicator_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061529 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061538 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061548 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061561 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061578 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061597 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061615 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061624 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061633 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061642 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:43:17 UTC] USER=www-data EUID=0 PID=4061668 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061677 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061686 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061695 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061704 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061713 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061740 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061749 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061767 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061787 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061805 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061814 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061823 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061832 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061850 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061860 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061869 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061878 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061887 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061896 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_der.key
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061905 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061915 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061925 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061934 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061943 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061952 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061961 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061988 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4061997 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4062006 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4062034 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4062044 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4062053 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4062062 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:43:18 UTC] USER=www-data EUID=0 PID=4062080 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01
[2026-01-18 23:43:19 UTC] USER=www-data EUID=0 PID=4062089 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:43:19 UTC] USER=www-data EUID=0 PID=4062107 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:43:19 UTC] USER=www-data EUID=0 PID=4062116 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
[2026-01-18 23:43:19 UTC] USER=www-data EUID=0 PID=4062143 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator_pk8.der
[2026-01-18 23:43:19 UTC] USER=www-data EUID=0 PID=4062153 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01 → /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:43:19 UTC] USER=www-data EUID=0 PID=4062163 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:43:19 UTC] USER=www-data EUID=0 PID=4062181 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
✅ Symlinked ca.pem
✅ Symlinked client-cert.pem
[2026-01-18 23:43:19 UTC] USER=www-data EUID=0 PID=4062199 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-01-18 23:43:19 UTC] USER=www-data EUID=0 PID=4062208 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:43:19 UTC] USER=www-data EUID=0 PID=4062217 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:43:19 UTC] USER=www-data EUID=0 PID=4062226 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:43:19 UTC] USER=www-data EUID=0 PID=4062235 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: user-sau-main-dev
User: replicator
Node: worker-01
FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
database.sslkey: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt"
export PGSSLMODE="verify-full"
psql -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -U replicator -d postgres
[0;32m[OK][0m Replicator certificate created for worker-01 in /home/postgres/
[0;34m[INFO][0m Using replicator certificates from primary worker-01...
[2026-01-18 23:43:19 UTC] USER=www-data EUID=0 PID=4062263 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key.pkcs1
[2026-01-18 23:43:19 UTC] USER=www-data EUID=0 PID=4062285 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.key
[2026-01-18 23:43:19 UTC] USER=www-data EUID=0 PID=4062306 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/replicator.crt
[0;32m[OK][0m Replicator certificates verified at /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[0;32m[OK][0m root.crt verified at /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01
[0;34m[INFO][0m Updating primary pg_hba.conf to allow replication...
[0;34m[INFO][0m Standby IP: 10.100.1.233/32 (standby's source IP)
[0;34m[INFO][0m Primary application IP: 10.100.1.232/32 (for local pg_basebackup)
[0;34m[INFO][0m Primary DNS IP: 10.100.1.232/32 (DNS resolution of db-user-sau-main-dev-postgresql-worker-01.fastorder.com)
[2026-01-18 23:43:19 UTC] USER=www-data EUID=0 PID=4062337 ACTION=passthru ARGS=grep -qxF # BEGIN standby-replication (managed) /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[2026-01-18 23:43:19 UTC] USER=www-data EUID=0 PID=4062382 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl replication replicator 10.100.1.233/32 scram-sha-256
$0==begin {inside=1}
inside && $0==rule {found=1}
$0==end {inside=0}
END {exit found?0:1}
/data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[2026-01-18 23:43:20 UTC] USER=www-data EUID=0 PID=4062406 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl replication replicator 10.100.1.233/32 scram-sha-256 /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[2026-01-18 23:43:20 UTC] USER=www-data EUID=0 PID=4062428 ACTION=passthru ARGS=awk -v begin=# BEGIN standby-replication (managed) -v end=# END standby-replication (managed) -v rule=hostssl replication replicator 10.100.1.232/32 scram-sha-256
$0==begin {inside=1}
inside && $0==rule {found=1}
$0==end {inside=0}
END {exit found?0:1}
/data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[2026-01-18 23:43:20 UTC] USER=www-data EUID=0 PID=4062452 ACTION=passthru ARGS=sed -i /^# END standby-replication (managed)$/i hostssl replication replicator 10.100.1.232/32 scram-sha-256 /data/postgresql/17/user-sau-main-dev/worker-01/pg_hba.conf
[0;34m[INFO][0m Reloading primary PostgreSQL service...
[2026-01-18 23:43:20 UTC] USER=www-data EUID=0 PID=4062475 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-worker-01.service
[0;32m[OK][0m Primary pg_hba.conf updated and service reloaded
[1;33m[WARN][0m Removing existing data directory: /data/postgresql/17/user-sau-main-dev/worker-01-standby-01
[2026-01-18 23:43:20 UTC] USER=www-data EUID=0 PID=4062505 ACTION=fsop ARGS=rm -rf /data/postgresql/17/user-sau-main-dev/worker-01-standby-01
[0;34m[INFO][0m Primary host: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[0;34m[INFO][0m Using replicator cert: /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt
[0;34m[INFO][0m Using replicator key: /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key (PKCS#8 format)
[0;34m[INFO][0m Using CA cert: /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[0;34m[INFO][0m Verifying postgres user can access certificates...
[0;31m[ERR][0m postgres user CANNOT read /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[0;34m[INFO][0m File permissions:
lrwxrwxrwx 1 postgres ssl-cert 68 Jan 18 23:43 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt -> /etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01/root.crt
[0;34m[INFO][0m Parent directory permissions:
drwx------ 2 postgres postgres 4096 Jan 18 23:43 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
drwx------ 5 postgres postgres 4096 Jan 18 23:42 /home/postgres/ssl/.postgresql/user-sau-main-dev
[1;33m[WARN][0m Attempting to fix permissions (/usr/local/bin/fastorder-provisioning-wrapper.sh required)...
[0;34m[INFO][0m Fixing /home/postgres/ directory...
[0;34m[INFO][0m Fixing /home/postgres/ssl/.postgresql/...
[2026-01-18 23:43:21 UTC] USER=www-data EUID=0 PID=4062594 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/
[0;34m[INFO][0m Fixing parent directory: /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:43:21 UTC] USER=www-data EUID=0 PID=4062617 ACTION=fsop ARGS=chmod 755 /home/postgres/ssl/.postgresql/user-sau-main-dev
[0;34m[INFO][0m Fixing certificate directory: /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01
[0;34m[INFO][0m Fixing CA certificate: /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[2026-01-18 23:43:21 UTC] USER=www-data EUID=0 PID=4062659 ACTION=fsop ARGS=chmod 644 /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt
[0;32m[OK][0m Permissions fixed
[0;32m[OK][0m postgres user can now read /home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt after permission fix
[2026-01-18 23:43:21 UTC] USER=www-data EUID=0 PID=4062724 ACTION=fsop ARGS=chmod 2775 /var/run/postgresql-user-sau-main-dev-worker-01-standby-01
[0;34m[INFO][0m Checking primary database size before pg_basebackup...
[0;34m[INFO][0m Total primary database size: 29 MB
[0;34m[INFO][0m Estimated transfer time: ~0 minutes (at 10MB/s with compression)
[0;34m[INFO][0m Retrieving replicator password from AWS Secrets Manager: fastorder/db/user/sau/main/dev/postgresql/replicator
[0;32m[OK][0m Replicator password retrieved successfully
[0;34m[INFO][0m Starting pg_basebackup...
[2026-01-18 23:43:23 UTC] USER=www-data EUID=0 PID=4062810 ACTION=passthru ARGS=sudo -u postgres env PGPASSWORD=4fdUrcEKNirjtl6pfO2YEuBbBDxOb2hE PGSSLMODE=verify-full PGSSLCERT=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.crt PGSSLKEY=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/replicator.key PGSSLROOTCERT=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01/root.crt /usr/lib/postgresql/17/bin/pg_basebackup -h db-user-sau-main-dev-postgresql-worker-01.fastorder.com -p 5432 -U replicator -D /data/postgresql/17/user-sau-main-dev/worker-01-standby-01 -Fp -Xs -P -R --checkpoint=fast --wal-method=stream --verbose
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/2000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_4062819"
30526/30526 kB (100%), 0/1 tablespace (...-01-standby-01/global/pg_control)
30526/30526 kB (100%), 1/1 tablespace
pg_basebackup: write-ahead log end point: 0/2000120
pg_basebackup: waiting for background process to finish streaming ...
pg_basebackup: syncing data to disk ...
pg_basebackup: renaming backup_manifest.tmp to backup_manifest
pg_basebackup: base backup completed
[0;32m[OK][0m pg_basebackup complete
[0;34m[INFO][0m Fixing postgresql.auto.conf to use IP-based primary_conninfo (matching golden backup)...
[2026-01-18 23:43:23 UTC] USER=www-data EUID=0 PID=4062833 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-18 23:43:23 UTC] USER=www-data EUID=0 PID=4062855 ACTION=fsop ARGS=chmod 600 /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-18 23:43:23 UTC] USER=www-data EUID=0 PID=4062876 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/standby.signal
[2026-01-18 23:43:23 UTC] USER=www-data EUID=0 PID=4062886 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/standby.signal
[0;32m[OK][0m standby.signal verified and permissions set
[0;34m[INFO][0m Fixing postgresql.conf with standby-specific settings...
[1;33m[WARN][0m postgresql.conf not found at /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/postgresql.conf
[0;34m[INFO][0m Verifying postgresql.auto.conf...
[1;33m[WARN][0m postgresql.auto.conf not found - pg_basebackup may have failed
[2026-01-18 23:43:23 UTC] USER=www-data EUID=0 PID=4062916 ACTION=fsop ARGS=rm -f /tmp/.pg_pwfile.ixntXx
[0;34m[INFO][0m Writing postgresql.conf (TLS≥1.2, SCRAM, audit logs)
[0;32m[OK][0m postgresql.conf updated successfully
[0;34m[INFO][0m Writing pg_hba.conf (mTLS with client certificates + SCRAM, least-privilege)
[2026-01-18 23:43:24 UTC] USER=www-data EUID=0 PID=4062965 ACTION=fsop ARGS=cp /tmp/tmp.X09s7qGDYl /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/pg_hba.conf
[2026-01-18 23:43:24 UTC] USER=www-data EUID=0 PID=4062992 ACTION=fsop ARGS=chown postgres:postgres /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/pg_hba.conf
[0;32m[OK][0m pg_hba.conf updated
[0;34m[INFO][0m Creating systemd unit: /etc/systemd/system/postgresql@user-sau-main-dev-worker-01-standby-01.service
[2026-01-18 23:43:24 UTC] USER=www-data EUID=0 PID=4063039 ACTION=fsop ARGS=mv -f /tmp/.pg_unit.05Fv2R /etc/systemd/system/postgresql@user-sau-main-dev-worker-01-standby-01.service
[0;32m[OK][0m systemd unit written
[2026-01-18 23:43:24 UTC] USER=www-data EUID=0 PID=4063109 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest /var/spool/pgbackrest /var/log/pgbackrest
[2026-01-18 23:43:24 UTC] USER=www-data EUID=0 PID=4063130 ACTION=passthru ARGS=systemctl daemon-reload
[0;34m[INFO][0m Starting PostgreSQL instance...
[2026-01-18 23:43:25 UTC] USER=www-data EUID=0 PID=4063253 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-worker-01-standby-01.service
[0;34m[INFO][0m Waiting for ACTIVE (systemd)…
[0;32m[OK][0m Service ACTIVE
[0;34m[INFO][0m Waiting for port 5432 bind…
[0;32m[OK][0m Port bound
[0;34m[INFO][0m Waiting pg_isready (socket)…
[0;32m[OK][0m Readiness via socket OK
[0;34m[INFO][0m Waiting pg_isready (TCP db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com:5432)…
[0;32m[OK][0m Startup sequence complete
[0;34m[INFO][0m Configuring synchronous replication on primary worker-01...
[0;34m[INFO][0m Current synchronous_standby_names: ''
[0;34m[INFO][0m Initializing synchronous_standby_names with first standby
[0;34m[INFO][0m New synchronous_standby_names: 'ANY 1 (worker_01_standby_01)'
[2026-01-18 23:43:26 UTC] USER=www-data EUID=0 PID=4063362 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_commit = on;
ALTER SYSTEM
[2026-01-18 23:43:26 UTC] USER=www-data EUID=0 PID=4063386 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -v ON_ERROR_STOP=1 -c ALTER SYSTEM SET synchronous_standby_names = 'ANY 1 (worker_01_standby_01)';
ALTER SYSTEM
[2026-01-18 23:43:26 UTC] USER=www-data EUID=0 PID=4063419 ACTION=passthru ARGS=sudo -u postgres psql -U postgres -h /var/run/postgresql-user-sau-main-dev-worker-01 -p 5432 -d postgres --no-psqlrc -c SELECT pg_reload_conf();
pg_reload_conf
----------------
t
(1 row)
[0;32m[OK][0m ✅ Synchronous replication configured on primary
[0;32m[OK][0m Setting: ANY 1 (worker_01_standby_01)
[0;34m[INFO][0m Validating core security GUCs (via local socket)…
[0;32m[OK][0m Security GUCs verified (ssl, min TLS, SCRAM, audit logs)
[0;34m[INFO][0m Skipping database/role provisioning on standby node (read-only)
[0;34m[INFO][0m Database/roles will be replicated from primary: worker-01
[0;34m[INFO][0m Applying connection and memory optimizations...
[0;34m[INFO][0m Standby will use primary's max_connections: 100
[0;34m[INFO][0m Current settings: max_connections=100, work_mem=8MB
[0;34m[INFO][0m Target settings (standby): max_connections=100, work_mem=8MB
[0;32m[OK][0m Connection settings already optimized
[0;34m[INFO][0m Skipping password setting - this is a standby (read-only)
[0;34m[INFO][0m Use primary's postgres password to connect to this standby
[0;34m[INFO][0m Updating /etc/hosts with PostgreSQL hostname mappings...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m CONFIGURING POSTGRESQL NETWORK & DNS
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Environment: user-sau-main-dev
[0;34m[INFO][0m Identifier: worker-01-standby-01
[0;34m[INFO][0m PostgreSQL IP: 10.100.1.233
[0;34m[INFO][0m Primary hostname: db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
[0;34m[INFO][0m Adding /etc/hosts entry for worker-01-standby-01...
[0;34m[INFO][0m db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com → 10.100.1.233
[0;34m[INFO][0m ✅ db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com already exists with correct IP
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;32m✅[0m ✅ Network & DNS configuration complete
[0;32m✅[0m ═══════════════════════════════════════════════════════════════
[0;34m[INFO][0m Verifying /etc/hosts entries:
10.100.1.233 db-user-sau-main-dev-postgresql-worker-01-standby-01.fastorder.com
[0;32m[OK][0m PostgreSQL 'user-sau-main-dev' is up with TLS/SCRAM/logging.
Superuser TCP mode: cert
Connect (mTLS):
psql "sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/worker-01-standby-01/ca.crt \
sslcert=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.crt \
sslkey=/home/postgres/ssl/.postgresql/user-sau-main-dev/worker-01-standby-01/postgres.key \
host=db-user-sau-main-dev-postgresql-worker-01-standby-01 port=5432 dbname=postgres user=postgres"
File been compeleted perfectly: 02-setup-pg-instance
[0;34m[INFO][0m Registering PostgreSQL node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: PostgreSQL
[INFO] Identifier: user-sau-main-dev-postgresql-worker-01-standby-01
[INFO] Identifier Parent: worker-01
[INFO] IP: 10.100.1.233
[INFO] Port: 5432
[INFO] FQDN: db-user-sau-main-dev-postgresql-worker-01-standby-01
[INFO] Status: running
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: 6b53354f-af0c-46ce-9112-1ad9eae0ff4a
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[0;32m[OK][0m PostgreSQL node registered to observability API
[DEBUG] Tracking substep start: steps/01-install/steps/03-role (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 📦 03 role...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-18 23:43:32 UTC] USER=www-data EUID=0 PID=4063919 ACTION=fsop ARGS=test -f /data/postgresql/17/user-sau-main-dev/worker-01-standby-01/standby.signal
⚠ This is a PostgreSQL STANDBY (read-only replica)
⚠ Skipping role creation - standby gets roles from primary via replication
⚠ Use the PRIMARY's credentials to connect to this standby
[DEBUG] Tracking substep start: steps/01-install/steps/05-setup-service (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 📦 05 setup service...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
ℹ️ Service-specific setup (user) is handled by parent script
✅ Step 5 completed (service setup delegated to 01-install/run.sh)
🔍 DEBUG_CHECKPOINT_01: Starting service-specific steps for SERVICE=user
🔍 DEBUG_CHECKPOINT_02: Checking for service-specific run.sh: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/user/run.sh
🔍 DEBUG_CHECKPOINT_03: No specific folder for user, using default
[DEBUG] Tracking substep start: steps/01-install/steps/default (RUN_UUID=63f840b8-2f06-4cde-90af-9e45d4a13e61)
[INFO] 🔸 Service: user (using default contracts schema)
🔍 DEBUG_CHECKPOINT_04: Executing default: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/01-install/steps/default/run.sh
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🟢 Starting default contracts schema provisioning for SERVICE=user
[INFO] Environment: user-sau-main-dev
[INFO] Schema: user (contracts tables)
[INFO] Identifier: worker-01-standby-01
[INFO] VM IP: 142.93.238.16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Skipping Schema Setup on worker-01-standby-01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ℹ️ Schema setup only runs on coordinator
ℹ️ This is a worker-01-standby-01 node - schemas replicate automatically
✅ Nothing to do on this node
[0;32m✓[0m ✅ Standby worker-01-standby-01 setup completed
[0;32m✓[0m ✅ PostgreSQL installation completed
[0;34m[INFO][0m Discovering additional setup steps...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 02-pg-bouncer.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up PgBouncer connection pooling...
[2026-01-18 23:43:37 UTC] USER=www-data EUID=0 PID=4064063 ACTION=fsop ARGS=rm -f /tmp/pgbouncer-ip.service /tmp/pgbouncer.service
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[0;32m✓ [SECRETS][0m Centralized Secrets Manager library loaded (Purpose-Engine Pattern)
[0;34m[SECRETS][0m Functions: PostgreSQL (build_pg_secret_name, get/set_pg_credentials_to_vault, rotate_pg_password)
[0;34m[SECRETS][0m Search (build_es_secret_name, get/set_es_credentials_to_vault)
[0;34m[SECRETS][0m Backups (build_backup_path)
[0;34m[SECRETS][0m Docs: /var/www/html/skeleton.dev.fastorder.com/docs/FASTCTL_USAGE_GUIDE.md
[0;34m[INFO][0m Checking for existing PgBouncer application environment in topology …
[0;32m[OK][0m Using existing PgBouncer environment:
[0;34m[INFO][0m IP: 10.100.1.184
[0;34m[INFO][0m FQDN: db-user-sau-main-dev-postgresql-bouncer.fastorder.com
[0;34m[INFO][0m Domain: db-user-sau-main-dev-postgresql-bouncer.fastorder.com
[0;34m[INFO][0m Ensuring /etc/hosts entry for db-user-sau-main-dev-postgresql-bouncer.fastorder.com …
[0;34m[INFO][0m Adding db-user-sau-main-dev-postgresql-bouncer.fastorder.com to /etc/hosts
[2026-01-18 23:43:38 UTC] USER=www-data EUID=0 PID=4064130 ACTION=fsop ARGS=sed -i /\sdb-user-sau-main-dev-postgresql-bouncer.fastorder.com\(\s\|$\)/d /etc/hosts
[0;32m[OK][0m Added db-user-sau-main-dev-postgresql-bouncer.fastorder.com -> 10.100.1.184 to /etc/hosts
[1;33m[WARN][0m IP 10.100.1.184 is assigned to multiple interfaces:
inet 10.100.1.217/32 scope global lo
valid_lft forever preferred_lft forever
inet 10.100.1.184/32 scope global lo
--
inet 10.100.1.219/32 scope global eth0:219
valid_lft forever preferred_lft forever
inet 10.100.1.184/32 scope global eth0
[1;33m[WARN][0m This may cause routing issues
[0;34m[INFO][0m Final verification of /etc/hosts entry for db-user-sau-main-dev-postgresql-bouncer.fastorder.com …
[0;32m[OK][0m /etc/hosts correctly maps db-user-sau-main-dev-postgresql-bouncer.fastorder.com to 10.100.1.184
[1;33m[WARN][0m IP 10.100.1.184 is already bound to other interface(s):
inet 10.100.1.184/32 scope global lo
inet 10.100.1.184/32 scope global eth0
[0;34m[INFO][0m Attempting to also bind 10.100.1.184 to lo:pgbouncer ...
[2026-01-18 23:43:38 UTC] USER=www-data EUID=0 PID=4064168 ACTION=passthru ARGS=ip addr add 10.100.1.184/32 dev lo label lo:pgbouncer
RTNETLINK answers: File exists
[0;32m[OK][0m IP 10.100.1.184 is already bound to lo (may have different label)
[2026-01-18 23:43:38 UTC] USER=www-data EUID=0 PID=4064188 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:43:38 UTC] USER=www-data EUID=0 PID=4064301 ACTION=passthru ARGS=systemctl restart pgbouncer-ip@user-sau-main-dev.service
Job for pgbouncer-ip@user-sau-main-dev.service failed because the control process exited with error code.
See "systemctl status pgbouncer-ip@user-sau-main-dev.service" and "journalctl -xeu pgbouncer-ip@user-sau-main-dev.service" for details.
[2026-01-18 23:43:39 UTC] USER=www-data EUID=0 PID=4064311 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer-ip@user-sau-main-dev.service
[1;33m[WARN][0m pgbouncer-ip@user-sau-main-dev.service is not active
[1;33m[WARN][0m Check status: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@user-sau-main-dev.service
[2026-01-18 23:43:39 UTC] USER=www-data EUID=0 PID=4064335 ACTION=fsop ARGS=mkdir -p /etc/pgbouncer/user-sau-main-dev
[2026-01-18 23:43:39 UTC] USER=www-data EUID=0 PID=4064346 ACTION=fsop ARGS=mkdir -p /run/pgbouncer/user-sau-main-dev
[2026-01-18 23:43:39 UTC] USER=www-data EUID=0 PID=4064355 ACTION=fsop ARGS=mkdir -p /var/log/pgbouncer/user-sau-main-dev
[2026-01-18 23:43:39 UTC] USER=www-data EUID=0 PID=4064372 ACTION=fsop ARGS=chmod 750 /etc/pgbouncer/user-sau-main-dev
[2026-01-18 23:43:39 UTC] USER=www-data EUID=0 PID=4064381 ACTION=fsop ARGS=chmod 750 /run/pgbouncer/user-sau-main-dev
[2026-01-18 23:43:39 UTC] USER=www-data EUID=0 PID=4064390 ACTION=fsop ARGS=chmod 750 /var/log/pgbouncer/user-sau-main-dev
[2026-01-18 23:43:39 UTC] USER=www-data EUID=0 PID=4064399 ACTION=fsop ARGS=chown root:postgres /etc/pgbouncer/user-sau-main-dev
[2026-01-18 23:43:39 UTC] USER=www-data EUID=0 PID=4064408 ACTION=fsop ARGS=chown postgres:postgres /run/pgbouncer/user-sau-main-dev
[2026-01-18 23:43:39 UTC] USER=www-data EUID=0 PID=4064417 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbouncer/user-sau-main-dev
[0;34m[INFO][0m Generating pgbouncer_admin client certificates...
[0;34m[INFO][0m ⏳ This may take 30-60 seconds...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
Environment: user-sau-main-dev
Username: pgbouncer_admin
Identifier: pgbouncer
📦 Start executing client cert generation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Service: user
Zone: sau
Branch: main
Env: dev
Node: pgbouncer
User (CN): pgbouncer_admin
Hostname: db-user-sau-main-dev-postgresql-bouncer.fastorder.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:43:39 UTC] USER=www-data EUID=0 PID=4064452 ACTION=fsop ARGS=chmod 755 /tmp/pg-client-pgbouncer-pgbouncer_admin
[2026-01-18 23:43:39 UTC] USER=www-data EUID=0 PID=4064461 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.crt /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-18 23:43:39 UTC] USER=www-data EUID=0 PID=4064470 ACTION=fsop ARGS=cp /opt/fastorder/ssl/ca/fastorder_ra_root/ra_root.key /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
[2026-01-18 23:43:39 UTC] USER=www-data EUID=0 PID=4064479 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt
[2026-01-18 23:43:39 UTC] USER=www-data EUID=0 PID=4064488 ACTION=fsop ARGS=chmod 644 /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.key
🔑 Generating private key (PKCS#1 format)...
🔑 Converting to PKCS#8 PEM (for pgjdbc/debezium)...
🔑 (optional) Exporting DER as well...
📝 Generating CSR...
🔐 Signing with CA...
Certificate request self-signature ok
subject=CN = pgbouncer_admin
📂 Installing to canonical location → /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064505 ACTION=fsop ARGS=mkdir -p /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064515 ACTION=fsop ARGS=chmod 750 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064524 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064533 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064542 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/ra_root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064551 ACTION=fsop ARGS=ln -sf root.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064560 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin.key.pkcs1 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064569 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_der.key /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064578 ACTION=fsop ARGS=cp -f /tmp/pg-client-pgbouncer-pgbouncer_admin/pgbouncer_admin_pk8.der /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064587 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064596 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064605 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064614 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064623 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064632 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064643 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064652 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064661 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064670 ACTION=fsop ARGS=chown root:sslusers /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064679 ACTION=fsop ARGS=chown root:root /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064688 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt
✅ Canonical installation complete
📂 Creating symlinks for ab → /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064714 ACTION=fsop ARGS=mkdir -p /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064723 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064732 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064741 ACTION=fsop ARGS=chown ab:ab /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064750 ACTION=fsop ARGS=chmod 700 /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064759 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064768 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064777 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064786 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064795 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064804 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064813 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064825 ACTION=fsop ARGS=chown -h ab:ssl-cert /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for ab in /home/ab/ssl/.postgresql/user-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
📂 Creating symlinks for www-data → /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064835 ACTION=fsop ARGS=mkdir -p /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064844 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064853 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064862 ACTION=fsop ARGS=chown www-data:www-data /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064871 ACTION=fsop ARGS=chmod 700 /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064880 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064889 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064898 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064907 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064916 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064925 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064934 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-18 23:43:40 UTC] USER=www-data EUID=0 PID=4064946 ACTION=fsop ARGS=chown -h www-data:ssl-cert /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for www-data in /home/www-data/ssl/.postgresql/user-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
📂 Creating symlinks for postgres → /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4064956 ACTION=fsop ARGS=mkdir -p /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4064965 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4064974 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4064983 ACTION=fsop ARGS=chown postgres:postgres /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4064992 ACTION=fsop ARGS=chmod 700 /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065001 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065010 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065019 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065028 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065037 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065046 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065055 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065065 ACTION=fsop ARGS=chown -h postgres:ssl-cert /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for postgres in /home/postgres/ssl/.postgresql/user-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
📂 Creating symlinks for kafka → /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065075 ACTION=fsop ARGS=mkdir -p /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065084 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065093 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065102 ACTION=fsop ARGS=chown kafka:kafka /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065111 ACTION=fsop ARGS=chmod 700 /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065120 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065129 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065138 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065147 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065156 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1 /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key.pkcs1
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065166 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_der.key
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065175 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin_pk8.der
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065186 ACTION=fsop ARGS=chown -h kafka:ssl-cert /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/ca.crt
✅ Symlinks created for kafka in /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer → /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer
🎉 All requested users processed.
📋 Creating Kafka SSL certificate symlinks for www-data...
Source: /opt/kafka/secrets/user-sau-main-dev/coordinator/pem
Destination: /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065197 ACTION=fsop ARGS=mkdir -p /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065206 ACTION=fsop ARGS=chmod 750 /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065215 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /var/www/ssl/kafka/user-sau-main-dev/ca.pem
✅ Symlinked ca.pem
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065224 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
✅ Symlinked client-cert.pem
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065233 ACTION=fsop ARGS=ln -sf /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Symlinked client-key.pem
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065244 ACTION=fsop ARGS=chown -R www-data:www-data /var/www/ssl/kafka/user-sau-main-dev
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065253 ACTION=fsop ARGS=chmod 640 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065262 ACTION=fsop ARGS=chown kafka:sslusers /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-key.pem
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065271 ACTION=fsop ARGS=chmod 644 /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/ca.pem /opt/kafka/secrets/user-sau-main-dev/coordinator/pem/client-cert.pem
✅ Kafka certificate symlinks ready for www-data
PHP Kafka consumers can now use:
- ssl.ca.location: /var/www/ssl/kafka/user-sau-main-dev/ca.pem
- ssl.certificate.location: /var/www/ssl/kafka/user-sau-main-dev/client-cert.pem
- ssl.key.location: /var/www/ssl/kafka/user-sau-main-dev/client-key.pem
✅ Client certificate generated successfully!
Environment: user-sau-main-dev
User: pgbouncer_admin
Node: pgbouncer
FQDN: db-user-sau-main-dev-postgresql-bouncer.fastorder.com
Next steps for Kafka Connect (Debezium → Postgres):
- Point connector to PEM key files:
database.sslcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
database.sslkey: /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key # PKCS#8 PEM
database.sslrootcert: /home/kafka/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt
- If Connect runs in a container, bind-mount /home/kafka/ssl/.postgresql inside the container
and use the container path in connector config.
For local testing:
export PGSSLCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt"
export PGSSLKEY="/home/$USER/ssl/.postgresql/user-sau-main-dev/pgbouncer/pgbouncer_admin.key"
export PGSSLROOTCERT="/home/$USER/ssl/.postgresql/user-sau-main-dev/pgbouncer/root.crt"
export PGSSLMODE="verify-full"
psql -h db-user-sau-main-dev-postgresql-bouncer.fastorder.com -U pgbouncer_admin -d postgres
[0;32m[OK][0m mTLS client certificate present: /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[0;34m[INFO][0m Creating symlinks to canonical certificates in /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend...
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065287 ACTION=fsop ARGS=mkdir -p /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065296 ACTION=fsop ARGS=mkdir -p /etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065305 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065314 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key /etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065323 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/root.crt
[0;34m[INFO][0m Creating coordinator CA symlink for PostgreSQL server verification...
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065332 ACTION=fsop ARGS=ln -sf /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m Verifying canonical certificate permissions...
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065341 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065350 ACTION=fsop ARGS=chmod 640 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065359 ACTION=fsop ARGS=chmod 644 /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065368 ACTION=fsop ARGS=chown root:www-data /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key
[0;32m[OK][0m Backend certificate symlinks created in /etc/ssl
[0;32m[OK][0m Coordinator CA symlink created for server verification
[0;32m[OK][0m Certificates already in canonical location - no symlinks needed
[2026-01-18 23:43:41 UTC] USER=www-data EUID=0 PID=4065379 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/server.crt
[2026-01-18 23:43:42 UTC] USER=www-data EUID=0 PID=4065388 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/server.key
[2026-01-18 23:43:42 UTC] USER=www-data EUID=0 PID=4065397 ACTION=fsop ARGS=test -r /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt
[2026-01-18 23:43:42 UTC] USER=www-data EUID=0 PID=4065406 ACTION=fsop ARGS=test -r /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m PgBouncer will use PostgreSQL coordinator CA: /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;32m[OK][0m PostgreSQL coordinator at db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432 is reachable
[0;34m[INFO][0m Dumping SCRAM secrets from coordinator for PgBouncer auth_file …
[2026-01-18 23:43:42 UTC] USER=www-data EUID=0 PID=4065425 ACTION=fsop ARGS=cp /tmp/tmp.ym76rfjZuo /etc/pgbouncer/user-sau-main-dev/userlist.txt
[2026-01-18 23:43:42 UTC] USER=www-data EUID=0 PID=4065436 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/user-sau-main-dev/userlist.txt
[2026-01-18 23:43:42 UTC] USER=www-data EUID=0 PID=4065445 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/user-sau-main-dev/userlist.txt
[0;32m[OK][0m Auth file written: /etc/pgbouncer/user-sau-main-dev/userlist.txt
[0;34m[INFO][0m Retrieved password from vault for pgbouncer_admin
[0;34m[INFO][0m Ensuring PgBouncer admin role 'pgbouncer_admin' exists in Postgres (coordinator) …
[0;32m[OK][0m Role pgbouncer_admin created/updated successfully
[0;34m[SECRETS][0m Setting credentials in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/pgbouncer_admin
[0;32m✓ [SECRETS][0m Credentials updated in vault: fastorder/db/user/sau/main/dev/postgresql/coordinator/pgbouncer_admin
[0;34m[INFO][0m ✅ PgBouncer admin password stored in centralized secrets vault
[0;34m[INFO][0m Re-fetching SCRAM secrets after role creation to ensure pgbouncer_admin is included …
[2026-01-18 23:43:47 UTC] USER=www-data EUID=0 PID=4065622 ACTION=fsop ARGS=cp /tmp/tmp.EqzpCLh1Xs /etc/pgbouncer/user-sau-main-dev/userlist.txt
[2026-01-18 23:43:47 UTC] USER=www-data EUID=0 PID=4065637 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/user-sau-main-dev/userlist.txt
[2026-01-18 23:43:47 UTC] USER=www-data EUID=0 PID=4065657 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/user-sau-main-dev/userlist.txt
[0;32m[OK][0m Auth file updated with pgbouncer_admin SCRAM hash
[0;34m[INFO][0m Auth file contains [2026-01-18 23:43:47 UTC] USER=www-data EUID=0 PID=4065668 ACTION=passthru ARGS=bash -c wc -l < '/etc/pgbouncer/user-sau-main-dev/userlist.txt'
4 user(s)
[0;32m[OK][0m Admin 'pgbouncer_admin' password generated and saved
[0;34m[INFO][0m Configuring PostgreSQL to prevent Citus metadata sync hangs...
ALTER ROLE
[0;32m[OK][0m Disabled Citus metadata sync for pgbouncer_admin
[0;34m[INFO][0m Verifying application database fastorder_user_sau_main_dev_db exists...
[0;32m[OK][0m ✓ Database fastorder_user_sau_main_dev_db exists
[0;34m[INFO][0m Granting permissions to pgbouncer_admin on fastorder_user_sau_main_dev_db...
GRANT
[0;32m[OK][0m ✓ Granted CONNECT on fastorder_user_sau_main_dev_db to pgbouncer_admin
GRANT
[0;32m[OK][0m ✓ Granted USAGE on schema public to pgbouncer_admin
GRANT
[0;32m[OK][0m ✓ Granted SELECT on all tables to pgbouncer_admin
ALTER DATABASE
[0;32m[OK][0m Set synchronous_commit=local for fastorder_user_sau_main_dev_db
[0;34m[INFO][0m Ensuring pg_hba.conf entry for pgbouncer_admin …
[0;34m[INFO][0m Adding pg_hba.conf entries for pgbouncer_admin with cert auth …
[2026-01-18 23:43:47 UTC] USER=unknown EUID=33 PID=4065704 ACTION=-u ARGS=postgres bash
ERROR: Invalid or unauthorized action: -u
[0;32m[OK][0m pg_hba.conf updated and PostgreSQL configuration reloaded
[1;33m[WARN][0m pg_hba.conf entry may not have loaded correctly
[0;34m[INFO][0m Writing /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini …
[2026-01-18 23:43:49 UTC] USER=www-data EUID=0 PID=4065809 ACTION=fsop ARGS=cp /tmp/tmp.7ej7MTKe20 /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini
[2026-01-18 23:43:49 UTC] USER=www-data EUID=0 PID=4065818 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini
[2026-01-18 23:43:49 UTC] USER=www-data EUID=0 PID=4065827 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini
[2026-01-18 23:43:49 UTC] USER=www-data EUID=0 PID=4065836 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbouncer/user-sau-main-dev /run/pgbouncer/user-sau-main-dev /var/log/pgbouncer/user-sau-main-dev
[2026-01-18 23:43:49 UTC] USER=www-data EUID=0 PID=4065845 ACTION=fsop ARGS=chmod 640 /etc/pgbouncer/user-sau-main-dev/userlist.txt
[0;32m[OK][0m pgbouncer.ini ready
[0;34m[INFO][0m Verifying TLS settings in pgbouncer.ini:
[2026-01-18 23:43:49 UTC] USER=www-data EUID=0 PID=4065855 ACTION=fsop ARGS=grep -E (client_tls_sslmode|server_tls) /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini
client_tls_sslmode = verify-full
server_tls_sslmode = verify-full
server_tls_ca_file = /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
server_tls_cert_file = /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
server_tls_key_file = /etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;34m[INFO][0m Verifying PgBouncer server certificate files:
[2026-01-18 23:43:49 UTC] USER=www-data EUID=0 PID=4065864 ACTION=fsop ARGS=test -r /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[0;32m[OK][0m Server cert readable by postgres: /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt
[2026-01-18 23:43:49 UTC] USER=www-data EUID=0 PID=4065873 ACTION=fsop ARGS=test -r /etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;32m[OK][0m Server key readable by postgres: /etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key
[0;34m[INFO][0m Verifying coordinator CA certificate:
[2026-01-18 23:43:49 UTC] USER=www-data EUID=0 PID=4065882 ACTION=fsop ARGS=test -r /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;32m[OK][0m Coordinator CA readable by postgres: /etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/coordinator-ca.crt
[0;34m[INFO][0m Preflight: stopping any conflicting PgBouncer on 6432 …
[2026-01-18 23:43:49 UTC] USER=www-data EUID=0 PID=4065891 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer.service
[2026-01-18 23:43:49 UTC] USER=www-data EUID=0 PID=4065900 ACTION=passthru ARGS=systemctl stop pgbouncer@user-sau-main-dev.service
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json?all=1": dial unix /var/run/docker.sock: connect: permission denied
[1;33m[WARN][0m Killing existing pgbouncer processes: 3817175
[2026-01-18 23:45:19 UTC] USER=www-data EUID=0 PID=4073482 ACTION=passthru ARGS=bash -c kill -9 3817175
[2026-01-18 23:45:21 UTC] USER=www-data EUID=0 PID=4073588 ACTION=passthru ARGS=systemctl daemon-reload
[0;32m[OK][0m systemd unit installed: pgbouncer@user-sau-main-dev.service
[0;34m[INFO][0m Running pre-flight IP conflict check for 10.100.1.184:6432 …
[1;33m[WARN][0m IP conflict checker not found at /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/lib/check-ip-conflicts.sh
[1;33m[WARN][0m Skipping pre-flight check - conflicts may occur
[0;34m[INFO][0m Starting PgBouncer (user-sau-main-dev) …
[2026-01-18 23:45:22 UTC] USER=www-data EUID=0 PID=4073681 ACTION=passthru ARGS=systemctl restart pgbouncer@user-sau-main-dev.service
[2026-01-18 23:45:22 UTC] USER=www-data EUID=0 PID=4073693 ACTION=passthru ARGS=systemctl is-active --quiet pgbouncer@user-sau-main-dev.service
[0;32m[OK][0m Service ACTIVE
[0;34m[INFO][0m Verifying auth_file before probing …
[0;34m[INFO][0m Auth file contains 4 user(s)
[1;33m[WARN][0m Auth file does NOT contain pgbouncer_admin entry - authentication will fail
[0;34m[INFO][0m Probing admin console via SSL (psql to database 'pgbouncer') …
[0;34m[INFO][0m Retrieved password from vault for admin console probe
[1;33m[WARN][0m Admin console probe failed (see error below)
psql: error: connection to server at "10.100.1.184", port 6432 failed: server certificate for "db-user-sau-main-dev-postgresql-bouncer.fastorder.com" (and 6 other names) does not match host name "10.100.1.184"
[1;33m[WARN][0m Troubleshooting:
[1;33m[WARN][0m 1. Check auth_file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/user-sau-main-dev/userlist.txt
[1;33m[WARN][0m 2. Test with: PGPASSWORD='yvonAdiGcvLlur+JNgqyr7ru' psql -h 10.100.1.184 -p 6432 -U pgbouncer_admin -d pgbouncer
[1;33m[WARN][0m 3. Check logs: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@user-sau-main-dev.service -n 50
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Running Comprehensive PgBouncer Verification Tests
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Password extracted: yvonAdiGcv... (using postgres user certificates)
[0;34m[INFO][0m Test 1/7: Admin Console - SHOW POOLS
database | user | cl_active | cl_waiting | cl_active_cancel_req | cl_waiting_cancel_req | sv_active | sv_active_cancel | sv_being_canceled | sv_idle | sv_used | sv_tested | sv_login | maxwait | maxwait_us | pool_mode | load_balance_hosts
-----------+-----------+-----------+------------+----------------------+-----------------------+-----------+------------------+-------------------+---------+---------+-----------+----------+---------+------------+-----------+--------------------
pgbouncer | pgbouncer | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | statement |
(1 row)
[0;32m[OK][0m ✓ SHOW POOLS: SUCCESS
[0;34m[INFO][0m Test 2/7: Admin Console - SHOW VERSION
[0;32m[OK][0m ✓ SHOW VERSION: PgBouncer 1.24.1
[0;34m[INFO][0m Test 3/7: Admin Console - SHOW STATS
database | total_server_assignment_count | total_xact_count | total_query_count | total_received | total_sent | total_xact_time | total_query_time | total_wait_time | total_client_parse_count | total_server_parse_count | total_bind_count | avg_server_assignment_count | avg_xact_count | avg_query_count | avg_recv | avg_sent | avg_xact_time | avg_query_time | avg_wait_time | avg_client_parse_count | avg_server_parse_count | avg_bind_count
-----------+-------------------------------+------------------+-------------------+----------------+------------+-----------------+------------------+-----------------+--------------------------+--------------------------+------------------+-----------------------------+----------------+-----------------+----------+----------+---------------+----------------+---------------+------------------------+------------------------+----------------
pgbouncer | 0 | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
(1 row)
[0;32m[OK][0m ✓ SHOW STATS: SUCCESS
[0;34m[INFO][0m Test 4/7: Admin Console - SHOW DATABASES
name | host | port | database | force_user | pool_size | min_pool_size | reserve_pool_size | server_lifetime | pool_mode | load_balance_hosts | max_connections | current_connections | max_client_connections | current_client_connections | paused | disabled
--------------------------------+-----------------------------------------------------------+------+--------------------------------+------------+-----------+---------------+-------------------+-----------------+-----------+--------------------+-----------------+---------------------+------------------------+----------------------------+--------+----------
fastorder_user_sau_main_dev_db | db-user-sau-main-dev-postgresql-coordinator.fastorder.com | 5432 | fastorder_user_sau_main_dev_db | | 100 | 0 | 20 | 3600 | | | 0 | 0 | 0 | 0 | 0 | 0
pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | 0 | 3600 | statement | | 0 | 0 | 0 | 2 | 0 | 0
(2 rows)
[0;32m[OK][0m ✓ SHOW DATABASES: SUCCESS
[0;34m[INFO][0m Test 5/7: Admin Console - SHOW CONFIG
[0;32m[OK][0m ✓ SHOW CONFIG: SUCCESS
[0;34m[INFO][0m Key settings:
[0;34m[INFO][0m client_tls_sslmode = verify-full|disable|yes
[0;34m[INFO][0m max_client_conn = 2048|100|yes
[0;34m[INFO][0m pool_mode = transaction|session|yes
[0;34m[INFO][0m server_tls_sslmode = verify-full|prefer|yes
psql "host=db-user-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_user_sau_main_dev_db user=pgbouncer_admin password=yvonAdiGcvLlur+JNgqyr7ru connect_timeout=5 sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/root.crt sslcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.crt sslkey=/etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/pgbouncer_admin.key" --no-psqlrc -Atc 'SELECT version();'
[0;34m[INFO][0m Test 6/7: Application Database - SELECT version()
[1;33m[WARN][0m ✗ Application database query: FAILED (timeout or connection issue)
[1;33m[WARN][0m If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[0;34m[INFO][0m Test 7/8: Application Database - Connection Details
[1;33m[WARN][0m ✗ Connection details: FAILED (timeout or connection issue)
[1;33m[WARN][0m If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[0;34m[INFO][0m Test 8/8: End-to-End Application Routing - Pool Verification
[0;34m[INFO][0m Running actual queries through PgBouncer to verify routing and pooling...
[1;33m[WARN][0m ✗ End-to-end routing verification: FAILED - All 3 queries failed
[1;33m[WARN][0m If Citus is not set up yet, run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[1;33m[WARN][0m Otherwise check if database fastorder_user_sau_main_dev_db exists and user pgbouncer_admin has permissions
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Verification Complete - Tests 1-5 PASSED (Admin console verified)
[1;33m[WARN][0m Tests 6-8 FAILED - Application database not accessible
[1;33m[WARN][0m This is expected if Citus is not set up yet
[1;33m[WARN][0m Run: ./setup/05-db/engine/postgresql/steps/03-citus-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m[OK][0m PgBouncer is up for user-sau-main-dev
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Connection Examples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Password stored in: AWS Secrets Manager (fastorder/db/web/ksa/main/dev/postgresqluser/sau/main/dev/coordinator-pgbouncer_admin)
Current password: yvonAdiGcvLlur+JNgqyr7ru
1. Admin Console (using IP address to avoid DNS/SSL issues):
psql "host=10.100.1.184 port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=yvonAdiGcvLlur+JNgqyr7ru sslmode=verify-full sslrootcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"
2. Admin Console (using hostname):
psql "host=db-user-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=pgbouncer sslkey=/etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=yvonAdiGcvLlur+JNgqyr7ru sslmode=verify-full sslrootcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"
3. Application Database:
psql "host=db-user-sau-main-dev-postgresql-bouncer.fastorder.com port=6432 dbname=fastorder_user_sau_main_dev_db sslkey=/etc/ssl/private/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.key sslcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/pgbouncer_admin.crt user=pgbouncer_admin password=yvonAdiGcvLlur+JNgqyr7ru sslmode=verify-full sslrootcert=/etc/ssl/certs/user-sau-main-dev/pg/pgbouncer-backend/root.crt" --no-psqlrc -v ON_ERROR_STOP=1 -c "SHOW POOLS;"
4. Using .pgpass file:
echo "db-user-sau-main-dev-postgresql-bouncer.fastorder.com:6432:*:pgbouncer_admin:yvonAdiGcvLlur+JNgqyr7ru" >> ~/.pgpass
chmod 600 ~/.pgpass
psql -h db-user-sau-main-dev-postgresql-bouncer.fastorder.com -p 6432 -U pgbouncer_admin -d fastorder_user_sau_main_dev_db
5. Retrieve password from vault:
source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
PGPASSWORD="$(get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password')" \
psql -h 10.100.1.184 -p 6432 -U pgbouncer_admin -d pgbouncer -c "SHOW POOLS;"
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Architecture
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
• Default db 'fastorder_user_sau_main_dev_db' → Citus coordinator (db-user-sau-main-dev-postgresql-coordinator.fastorder.com)
• Worker access: 'fastorder_user_sau_main_dev_db_worker_1', 'fastorder_user_sau_main_dev_db_worker_2', … (if exist)
• Client TLS: require (password auth) / verify-full (mTLS with certs)
• Server TLS: verify-full (PgBouncer validates PostgreSQL certs)
• Auth: SCRAM-SHA-256 via /etc/pgbouncer/user-sau-main-dev/userlist.txt
• Pool mode: transaction (stateless connections)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Management
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Service Status:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer@user-sau-main-dev.service
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl status pgbouncer-ip@user-sau-main-dev.service
Logs:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru journalctl -u pgbouncer@user-sau-main-dev.service -f
/usr/local/bin/fastorder-provisioning-wrapper.sh tail -f /var/log/pgbouncer/user-sau-main-dev/pgbouncer.log
Reload Config:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@user-sau-main-dev.service
Restart:
command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl restart pgbouncer@user-sau-main-dev.service
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Files
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Config: /etc/pgbouncer/user-sau-main-dev/pgbouncer.ini
Auth file: /etc/pgbouncer/user-sau-main-dev/userlist.txt
Server cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/server.crt
Server key: /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/server.key
CA cert: /etc/fastorder/postgresql/certs/user-sau-main-dev/pgbouncer/ca.crt
PG CA: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
Logs: /var/log/pgbouncer/user-sau-main-dev/pgbouncer.log
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Troubleshooting
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
If "SASL authentication failed":
1. Check auth file: /usr/local/bin/fastorder-provisioning-wrapper.sh cat /etc/pgbouncer/user-sau-main-dev/userlist.txt
2. Verify pgbouncer_admin is present with SCRAM hash
3. Get password from vault:
source /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
get_pg_credentials_from_vault 'coordinator-pgbouncer_admin' 'password'
4. Reload PgBouncer: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@user-sau-main-dev.service
If "no pg_hba.conf entry":
1. Check pg_hba.conf on coordinator
2. Add rule: hostssl all pgbouncer_admin 10.100.1.184/32 cert clientcert=verify-full
3. Reload PostgreSQL
To add users to PgBouncer:
1. Create user in PostgreSQL with password
2. Re-run SCRAM dump:
psql "host=db-user-sau-main-dev-postgresql-coordinator.fastorder.com port=5432 dbname=postgres user=postgres \
sslmode=verify-full sslrootcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/root.crt \
sslcert=/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.crt sslkey=/etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/postgres.key" \
-Atc "SELECT '\"' || rolname || '\" \"' || rolpassword || '\"' \
FROM pg_authid WHERE rolpassword LIKE 'SCRAM-SHA-256%' \
AND rolcanlogin ORDER BY rolname;" | command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh fsop tee /etc/pgbouncer/user-sau-main-dev/userlist.txt
3. Reload: command sudo -n /usr/local/bin/fastorder-provisioning-wrapper.sh passthru systemctl reload pgbouncer@user-sau-main-dev.service
[0;34m[INFO][0m Registering PgBouncer node to observability API...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: PgBouncer
[INFO] Identifier: user-sau-main-dev-pgbouncer
[INFO] Identifier Parent: postgresql
[INFO] IP: 10.100.1.184
[INFO] Port: 6432
[INFO] FQDN: db-user-sau-main-dev-postgresql-bouncer.fastorder.com
[INFO] Status: running
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[SUCCESS] =========================================
[SUCCESS] Node registered successfully via API!
[SUCCESS] =========================================
[SUCCESS] Node UUID: c866fe26-0c2d-4619-a98d-8cd82c922b78
[SUCCESS] Environment UUID: a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[SUCCESS] Dashboard: https:\/\/skeleton.dev.fastorder.com\/dashboard\/monitoring\/environment\/a4fdf095-4188-4b8d-b14b-0256f3d06f0b
[0;32m[OK][0m PgBouncer node registered to observability API
[0;32m✓[0m ✅ PgBouncer setup completed
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 03-citus-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m CITUS DISTRIBUTED CLUSTER SETUP
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Phase 1: Installing Citus extension on workers...
[0;34m[INFO][0m Phase 2: Setting up coordinator and registering workers...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m 📦 PHASE 1: Installing Citus extension on 1 worker(s)...
[0;34m[INFO][0m → Worker 1/1: Installing Citus on worker-01...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS CLUSTER SETUP
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m 🔧 Setting up Citus Worker...
[0;34m[INFO][0m Temporarily disabling synchronous replication for extension installation...
t
[0;34m[INFO][0m Installing Citus extension on worker...
[0;32m[OK][0m Citus extension installed on worker
[0;34m[INFO][0m Restoring synchronous replication settings...
t
[0;34m[INFO][0m Worker Citus extension installed - registration will happen when coordinator setup runs
[0;32m[OK][0m Citus setup complete for worker-01
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m✓[0m ✅ Citus extension installed on worker-01
[0;32m✓[0m ✅ Phase 1 Complete: All 1 workers have Citus extension installed
[0;34m[INFO][0m 🔧 PHASE 2: Setting up Citus coordinator and registering workers...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS CLUSTER SETUP
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m 🔧 Setting up Citus Coordinator...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m DIAGNOSTIC: Configuration Variables
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m PG_WORKERS_NUM: 1
[0;34m[INFO][0m ENV_ID: user-sau-main-dev
[0;34m[INFO][0m DOMAIN: fastorder.com
[0;34m[INFO][0m PORT: 5432
[0;34m[INFO][0m SOCKET_DIR: /var/run/postgresql-user-sau-main-dev-coordinator
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Ensuring postgres client certificates exist for coordinator...
[0;32m[OK][0m Postgres client certificates already exist for coordinator
[0;34m[INFO][0m Adding citus_cert_map to coordinator pg_ident.conf...
[0;32m[OK][0m pg_ident.conf updated for coordinator
[0;34m[INFO][0m Installing Citus extension on coordinator...
[0;32m[OK][0m Citus extension installed on coordinator (postgres database)
[0;34m[INFO][0m Installing Citus extension on application database: fastorder_user_sau_main_dev_db...
[0;32m[OK][0m Citus extension installed on application database: fastorder_user_sau_main_dev_db
[0;34m[INFO][0m Configuring Citus SSL connection parameters...
[2026-01-18 23:45:46 UTC] USER=www-data EUID=0 PID=4074498 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-coordinator.service
[0;32m[OK][0m ✅ Citus SSL connection parameters configured: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator
[1;33m[WARN][0m Node not identified as coordinator, initializing...
[0;34m[INFO][0m Checking coordinator configuration...
[0;34m[INFO][0m Persisting citus.local_hostname to postgresql.conf...
[2026-01-18 23:45:49 UTC] USER=www-data EUID=0 PID=4074573 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/user-sau-main-dev/coordinator/postgresql.conf
[2026-01-18 23:45:49 UTC] USER=www-data EUID=0 PID=4074612 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-coordinator.service
[0;32m[OK][0m ✅ citus.local_hostname persisted to config and reloaded
[0;34m[INFO][0m Configuring coordinator hostname in postgres database: db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432
[0;32m[OK][0m ✅ Coordinator hostname set to db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432 in postgres database
[0;34m[INFO][0m Checking coordinator configuration in application database: fastorder_user_sau_main_dev_db...
[1;33m[WARN][0m ⚠️ Coordinator registered as 'localhost' in application database, fixing...
[0;34m[INFO][0m Configuring coordinator hostname in application database: db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432
[0;32m[OK][0m ✅ Coordinator hostname set to db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432 in application database
[0;34m[INFO][0m Validating coordinator configuration before worker registration...
[0;32m[OK][0m ✅ Coordinator hostname validated: db-user-sau-main-dev-postgresql-coordinator.fastorder.com
[0;32m[OK][0m ✅ citus_tables view is accessible
[0;34m[INFO][0m Checking coordinator self-registration...
[0;32m[OK][0m ✅ Coordinator is already self-registered
[0;34m[INFO][0m Configuring coordinator shard placement policy...
[0;32m[OK][0m ✅ Coordinator already configured in postgres database (shouldhaveshards = false)
[1;33m[WARN][0m ⚠️ Coordinator has 66 shards in fastorder_user_sau_main_dev_db - cannot set shouldhaveshards=false
[1;33m[WARN][0m You must rebalance shards to workers first, then run this setup again
[1;33m[WARN][0m Skipping shouldhaveshards configuration for application database
[0;34m[INFO][0m Registering 1 worker(s) to Citus cluster...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m PRE-FLIGHT: Checking worker availability...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Checking worker worker-01...
[0;34m[INFO][0m FQDN: db-user-sau-main-dev-postgresql-worker-01.fastorder.com
[0;32m[OK][0m ✅ Worker worker-01 is reachable via SSL
[0;32m[OK][0m All workers are reachable - proceeding with registration
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Adding Citus worker: db-user-sau-main-dev-postgresql-worker-01.fastorder.com:5432
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Adding citus_cert_map to worker-01 pg_ident.conf...
[0;32m[OK][0m pg_ident.conf updated for worker-01
[0;34m[INFO][0m Configuring worker worker-01 HBA for coordinator (10.100.1.231) access...
[0;32m[OK][0m Worker worker-01 HBA configured for coordinator (10.100.1.231)
[0;34m[INFO][0m Adding replication rules for 3 standby(s)...
[0;32m[OK][0m Replication rules already exist for worker-01
[0;34m[INFO][0m Reloading worker worker-01 to apply HBA changes...
[2026-01-18 23:45:52 UTC] USER=www-data EUID=0 PID=4074781 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-worker-01.service
[0;34m[INFO][0m Configuring coordinator HBA for worker worker-01 (10.100.1.232) access...
[0;32m[OK][0m Coordinator HBA configured for worker worker-01 (10.100.1.232)
[0;34m[INFO][0m Reloading coordinator to apply HBA changes...
[2026-01-18 23:45:52 UTC] USER=www-data EUID=0 PID=4074811 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-coordinator.service
[0;34m[INFO][0m Ensuring postgres client certificates exist for worker-01...
[0;32m[OK][0m Postgres client certificates already exist for worker-01
[0;34m[INFO][0m Configuring citus.node_conninfo on worker-01...
[2026-01-18 23:45:52 UTC] USER=www-data EUID=0 PID=4074828 ACTION=passthru ARGS=systemctl reload postgresql@user-sau-main-dev-worker-01.service
[0;32m[OK][0m citus.node_conninfo configured on worker-01
[0;34m[INFO][0m Temporarily relaxing sync-rep on worker worker-01...
t
[0;32m[OK][0m Worker worker-01 sync-rep relaxed (was: sync_commit=on)
[0;34m[INFO][0m Ensuring Citus extension on worker databases...
CREATE EXTENSION
CREATE EXTENSION
[0;34m[INFO][0m Running citus_add_node with 180s timeout...
NOTICE: shards are still on the coordinator after adding the new node
HINT: Use SELECT rebalance_table_shards(); to balance shards data between workers and coordinator or SELECT citus_drain_node('db-user-sau-main-dev-postgresql-coordinator.fastorder.com',5432); to permanently move shards away from the coordinator.
2
[0;34m[INFO][0m Restoring worker worker-01 sync-rep settings...
t
[0;32m[OK][0m Worker worker-01 sync-rep restored
[0;32m[OK][0m ✅ Worker db-user-sau-main-dev-postgresql-worker-01.fastorder.com successfully added to Citus cluster
[0;34m[INFO][0m Node ID: 2
[0;34m[INFO][0m Registered in: postgres, fastorder_user_sau_main_dev_db
[0;32m[OK][0m Worker worker-01 registration successful
[0;34m[INFO][0m Configuring worker worker-01 shard placement policy...
[0;32m[OK][0m ✅ Worker worker-01 configured to hold shards in all databases
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m POST-REGISTRATION: Verifying cluster state...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Expected workers: 1
[0;34m[INFO][0m Registered workers: 1
[0;32m[OK][0m ✅ All 1 workers successfully registered!
[0;34m[INFO][0m Citus cluster configuration:
db-user-sau-main-dev-postgresql-coordinator.fastorder.com 5432 0 t primary f
db-user-sau-main-dev-postgresql-worker-01.fastorder.com 5432 1 t primary t
[0;34m[INFO][0m Note: groupid=0 is the coordinator, groupid>0 are workers
[0;34m[INFO][0m shouldhaveshards: false=query router only, true=holds data shards
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m FINAL VALIDATION: Verifying configuration persistence...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:45:55 UTC] USER=www-data EUID=0 PID=4075126 ACTION=passthru ARGS=sudo -u postgres grep -q ^citus.local_hostname /data/postgresql/17/user-sau-main-dev/coordinator/postgresql.conf
[0;32m[OK][0m ✅ citus.local_hostname persisted in postgresql.conf
[0;32m[OK][0m ✅ All 1 worker(s) successfully registered and verified
[0;32m[OK][0m ✅ All validation checks passed
[0;32m[OK][0m Citus coordinator setup complete
[0;32m[OK][0m Citus setup complete for coordinator
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m✓[0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✓[0m ✅ CITUS CLUSTER SETUP COMPLETED SUCCESSFULLY
[0;32m✓[0m Coordinator: Ready and accepting connections
[0;32m✓[0m Workers registered: 1
[0;32m✓[0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 05-backup-setup.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up coordinator backup...
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for user-sau-main-dev...
[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO] Version: pgBackRest 2.56.0
[INFO] 2️⃣ Creating backup directories...
[2026-01-18 23:45:57 UTC] USER=www-data EUID=0 PID=4075221 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/user-sau-main-dev
[2026-01-18 23:45:57 UTC] USER=www-data EUID=0 PID=4075230 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/backup/user-sau-main-dev
[2026-01-18 23:45:57 UTC] USER=www-data EUID=0 PID=4075239 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-01-18 23:45:57 UTC] USER=www-data EUID=0 PID=4075248 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-18 23:45:57 UTC] USER=www-data EUID=0 PID=4075257 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-18 23:45:57 UTC] USER=www-data EUID=0 PID=4075266 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-18 23:46:02 UTC] USER=www-data EUID=0 PID=4075442 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-18 23:46:02 UTC] USER=www-data EUID=0 PID=4075451 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-18 23:46:02 UTC] USER=www-data EUID=0 PID=4075460 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-01-18 23:46:02 UTC] USER=www-data EUID=0 PID=4075469 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/archive/user-sau-main-dev
[2026-01-18 23:46:02 UTC] USER=www-data EUID=0 PID=4075478 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/user-sau-main-dev
[INFO] ✅ Backup directories created
[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-user-sau-main-dev
[2026-01-18 23:46:02 UTC] USER=www-data EUID=0 PID=4075499 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-18 23:46:02 UTC] USER=www-data EUID=0 PID=4075508 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key
[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-01-18 23:46:02 UTC] USER=www-data EUID=0 PID=4075517 ACTION=fsop ARGS=find /data/postgresql/17/user-sau-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-01-18 23:46:02 UTC] USER=www-data EUID=0 PID=4075526 ACTION=fsop ARGS=chown -R postgres:postgres /data/postgresql/17/user-sau-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed
[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-18 23:46:02 UTC] USER=www-data EUID=0 PID=4075535 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-01-18 23:46:02 UTC] USER=www-data EUID=0 PID=4075544 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-01-18 23:46:02 UTC] USER=www-data EUID=0 PID=4075553 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created
[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-18 23:46:02 UTC] USER=www-data EUID=0 PID=4075562 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/user-sau-main-dev/coordinator/PG_VERSION
[2026-01-18 23:46:02 UTC] USER=www-data EUID=0 PID=4075572 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running
[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza user-sau-main-dev-coordinator already initialized and verified
[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
pg_reload_conf
----------------
t
(1 row)
[INFO] ✅ WAL archiving configured for coordinator
[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-18 23:46:02 UTC] USER=www-data EUID=0 PID=4075626 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-18 23:46:04 UTC] USER=www-data EUID=0 PID=4075651 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-coordinator.service
[2026-01-18 23:46:08 UTC] USER=www-data EUID=0 PID=4075719 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled
[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-18 23:46:08 UTC] USER=www-data EUID=0 PID=4075743 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=user-sau-main-dev-coordinator --log-level-console=info check
2026-01-18 23:46:08.689 P00 INFO: check command begin 2.56.0: --exec-id=4075751-6a4cdf1a --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/user-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-user-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/user-sau-main-dev --stanza=user-sau-main-dev-coordinator
2026-01-18 23:46:08.705 P00 INFO: check repo1 configuration (primary)
2026-01-18 23:46:08.716 P00 ERROR: [028]: backup and archive info files exist but do not match the database
HINT: is this the correct stanza?
HINT: did an error occur during stanza-upgrade?
2026-01-18 23:46:08.716 P00 INFO: check command end: aborted with exception [028]
[WARN] ⚠️ Stanza verification failed - this may be normal if WAL archiving hasn't started yet
[WARN] The backup system is configured and will work once WAL segments are generated
[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-18 23:46:08 UTC] USER=www-data EUID=0 PID=4075764 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|user-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-user-sau-main-dev.sh
[2026-01-18 23:46:08 UTC] USER=www-data EUID=0 PID=4075773 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-user-sau-main-dev.sh
[2026-01-18 23:46:08 UTC] USER=www-data EUID=0 PID=4075791 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|user-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-user-sau-main-dev.sh
[2026-01-18 23:46:08 UTC] USER=www-data EUID=0 PID=4075800 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-user-sau-main-dev.sh
[INFO] ✅ Backup scripts created
[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-18 23:46:08 UTC] USER=www-data EUID=0 PID=4075818 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-user-sau-main-dev
[INFO] ✅ Cron jobs configured
[INFO] Schedule:
[INFO] - Full backup: Sundays at 2:00 AM
[INFO] - Differential backup: Mon-Sat at 2:00 AM
[INFO] 🔟 Creating restore documentation...
[2026-01-18 23:46:08 UTC] USER=www-data EUID=0 PID=4075836 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|user-sau-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-18 23:46:08 UTC] USER=www-data EUID=0 PID=4075846 ACTION=fsop ARGS=sed -i s|__ENV_ID__|user-sau-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-18 23:46:08 UTC] USER=www-data EUID=0 PID=4075863 ACTION=fsop ARGS=sed -i s|__DATA_DIR__|/data/postgresql/17/user-sau-main-dev/coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-18 23:46:08 UTC] USER=www-data EUID=0 PID=4075877 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-18 23:46:09 UTC] USER=www-data EUID=0 PID=4075886 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-18 23:46:09.117 P00 INFO: start command begin 2.56.0: --exec-id=4075931-e21144ee --log-level-console=info --log-level-file=debug --stanza=user-sau-main-dev-coordinator
2026-01-18 23:46:09.120 P00 WARN: stop file does not exist for stanza user-sau-main-dev-coordinator
2026-01-18 23:46:09.121 P00 INFO: start command end: completed successfully (9ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-18 23:46:09.169 P00 INFO: stanza-upgrade command begin 2.56.0: --exec-id=4075954-87754f67 --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/user-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-user-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/user-sau-main-dev --stanza=user-sau-main-dev-coordinator
2026-01-18 23:46:09.172 P00 INFO: stanza-upgrade for stanza 'user-sau-main-dev-coordinator' on repo1
2026-01-18 23:46:09.182 P00 INFO: stanza-upgrade command end: completed successfully (18ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-18 23:46:09 UTC] USER=www-data EUID=0 PID=4075958 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260118-234609.log
[2026-01-18 23:46:09 UTC] USER=www-data EUID=0 PID=4075967 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260118-234609.log
[2026-01-18 23:46:09 UTC] USER=www-data EUID=0 PID=4075976 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260118-234609.log
[INFO] Running backup (timeout: 10 minutes)...
[INFO] ✅ Initial full backup completed successfully
[INFO] Log: /var/log/pgbackrest/initial-backup-20260118-234609.log
2026-01-18 23:46:17.864 P00 INFO: repo1: remove expired backup 20260118-213103F
2026-01-18 23:46:17.896 P00 INFO: repo1: 17-21 remove archive, start = 000000010000000000000003, stop = 000000010000000000000005
2026-01-18 23:46:17.897 P00 INFO: repo1: 17-22 remove archive, start = 000000010000000000000001, stop = 000000010000000000000001
2026-01-18 23:46:17.897 P00 INFO: repo1: 17-23 remove archive, start = 000000010000000000000002, stop = 000000010000000000000002
2026-01-18 23:46:17.897 P00 INFO: expire command end: completed successfully (43ms)
[INFO] Current backups:
[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️ No worker identifier provided - skipping worker backup setup
[INFO] (Run with 'worker-01', 'worker-02', etc. to configure worker backups)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Completed steps:
[INFO] 1. pgBackRest installed and configured
[INFO] 2. WAL archiving enabled (archive_mode=on)
[INFO] 3. PostgreSQL restarted with new settings
[INFO] 4. pgBackRest stanza initialized and verified
[INFO] 5. Initial full backup completed
[INFO] 6. Automated backup cron jobs configured
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO] Coordinator:
[INFO] Stanza: user-sau-main-dev-coordinator
[INFO] Schedule: Full: Sun 2AM, Diff: Mon-Sat 2AM
[INFO] Common:
[INFO] Backup dir: /var/lib/pgbackrest/backup/user-sau-main-dev
[INFO] Archive dir: /var/lib/pgbackrest/archive/user-sau-main-dev
[INFO] Config: /etc/pgbackrest/pgbackrest.conf
[INFO] Restore guide: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[INFO] Retention:
[INFO] Full backups: 4 (keep last 4 full backups)
[INFO] Differential: 4 (keep last 4 diff per full)
[INFO] Archive WAL: Auto-managed by pgBackRest
[INFO] Manual commands:
[INFO] Coordinator: sudo -u postgres pgbackrest --stanza=user-sau-main-dev-coordinator backup
[INFO] List all backups: sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Setting up worker backups for 1 worker(s)...
[0;34m[INFO][0m Setting up backup for: worker-01
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[INFO] 🔍 Configuring backups for user-sau-main-dev...
[INFO] 1️⃣ Installing pgBackRest...
[INFO] ✅ pgBackRest already installed
[INFO] Version: pgBackRest 2.56.0
[INFO] 2️⃣ Creating backup directories...
[2026-01-18 23:46:18 UTC] USER=www-data EUID=0 PID=4076303 ACTION=fsop ARGS=mkdir -p /var/lib/pgbackrest/archive/user-sau-main-dev
[2026-01-18 23:46:18 UTC] USER=www-data EUID=0 PID=4076324 ACTION=fsop ARGS=mkdir -p /var/log/pgbackrest
[2026-01-18 23:46:18 UTC] USER=www-data EUID=0 PID=4076333 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest
[2026-01-18 23:46:18 UTC] USER=www-data EUID=0 PID=4076342 ACTION=fsop ARGS=mkdir -p /etc/pgbackrest/conf.d
[2026-01-18 23:46:18 UTC] USER=www-data EUID=0 PID=4076351 ACTION=fsop ARGS=chown -R postgres:postgres /var/lib/pgbackrest
[2026-01-18 23:46:19 UTC] USER=www-data EUID=0 PID=4076371 ACTION=fsop ARGS=chown -R postgres:postgres /var/log/pgbackrest
[2026-01-18 23:46:20 UTC] USER=www-data EUID=0 PID=4076380 ACTION=fsop ARGS=chown -R postgres:postgres /etc/pgbackrest
[2026-01-18 23:46:20 UTC] USER=www-data EUID=0 PID=4076389 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest
[2026-01-18 23:46:20 UTC] USER=www-data EUID=0 PID=4076409 ACTION=fsop ARGS=chmod 750 /var/lib/pgbackrest/backup/user-sau-main-dev
[INFO] ✅ Backup directories created
[INFO] 3️⃣ Configuring pgBackRest for coordinator...
[INFO] Using existing cipher key from /etc/pgbackrest/.cipher-key-user-sau-main-dev
[2026-01-18 23:46:20 UTC] USER=www-data EUID=0 PID=4076430 ACTION=fsop ARGS=chmod 640 /etc/pgbackrest/pgbackrest.conf
[2026-01-18 23:46:20 UTC] USER=www-data EUID=0 PID=4076439 ACTION=fsop ARGS=chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
[INFO] ✅ pgBackRest configuration created with shared cipher key
[INFO] 3️⃣.5️⃣ Cleaning up data directory...
[INFO] Removing old .backup.* files...
[2026-01-18 23:46:20 UTC] USER=www-data EUID=0 PID=4076448 ACTION=fsop ARGS=find /data/postgresql/17/user-sau-main-dev/coordinator -name *.backup.* -type f -delete
[INFO] Ensuring correct ownership...
[2026-01-18 23:46:20 UTC] USER=www-data EUID=0 PID=4076457 ACTION=fsop ARGS=chown -R postgres:postgres /data/postgresql/17/user-sau-main-dev/coordinator
[INFO] ✅ Data directory cleaned and permissions fixed
[INFO] 4️⃣ Creating pgBackRest spool directory...
[2026-01-18 23:46:20 UTC] USER=www-data EUID=0 PID=4076466 ACTION=fsop ARGS=mkdir -p /var/spool/pgbackrest
[2026-01-18 23:46:20 UTC] USER=www-data EUID=0 PID=4076475 ACTION=fsop ARGS=chown postgres:postgres /var/spool/pgbackrest
[2026-01-18 23:46:20 UTC] USER=www-data EUID=0 PID=4076484 ACTION=fsop ARGS=chmod 750 /var/spool/pgbackrest
[INFO] ✅ Spool directory created
[INFO] 4️⃣.5️⃣ Ensuring PostgreSQL coordinator is running...
[2026-01-18 23:46:20 UTC] USER=www-data EUID=0 PID=4076493 ACTION=passthru ARGS=sudo -u postgres test -f /data/postgresql/17/user-sau-main-dev/coordinator/PG_VERSION
[2026-01-18 23:46:20 UTC] USER=www-data EUID=0 PID=4076503 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[INFO] ✅ Coordinator is already running
[INFO] 5️⃣ Initializing pgBackRest stanza...
[INFO] Stanza exists - verifying system-id consistency...
[INFO] ✅ Coordinator stanza user-sau-main-dev-coordinator already initialized and verified
[INFO] 6️⃣ Configuring WAL archiving in PostgreSQL...
[INFO] Updating coordinator postgresql.conf...
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
pg_reload_conf
----------------
t
(1 row)
[INFO] ✅ WAL archiving configured for coordinator
[INFO] 7️⃣ Restarting PostgreSQL to enable archive_mode...
[INFO] Stopping PostgreSQL...
[2026-01-18 23:46:21 UTC] USER=www-data EUID=0 PID=4076570 ACTION=passthru ARGS=systemctl stop postgresql@user-sau-main-dev-coordinator.service
[INFO] Starting PostgreSQL with archive_mode enabled...
[2026-01-18 23:46:23 UTC] USER=www-data EUID=0 PID=4076628 ACTION=passthru ARGS=systemctl start postgresql@user-sau-main-dev-coordinator.service
[2026-01-18 23:46:26 UTC] USER=www-data EUID=0 PID=4076755 ACTION=passthru ARGS=systemctl is-active --quiet postgresql@user-sau-main-dev-coordinator.service
[INFO] ✅ PostgreSQL restarted successfully
[INFO] ✅ archive_mode is now enabled
[INFO] Verifying pgBackRest stanza with archive_mode enabled...
[2026-01-18 23:46:27 UTC] USER=www-data EUID=0 PID=4076777 ACTION=passthru ARGS=sudo -u postgres pgbackrest --stanza=user-sau-main-dev-coordinator --log-level-console=info check
2026-01-18 23:46:27.203 P00 INFO: check command begin 2.56.0: --exec-id=4076784-f62673ac --log-level-console=info --log-level-file=debug --pg1-path=/data/postgresql/17/user-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-user-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/user-sau-main-dev --stanza=user-sau-main-dev-coordinator
2026-01-18 23:46:27.227 P00 INFO: check repo1 configuration (primary)
2026-01-18 23:46:27.273 P00 INFO: check repo1 archive for WAL (primary)
2026-01-18 23:46:27.574 P00 INFO: WAL segment 000000010000000000000005 successfully archived to '/var/lib/pgbackrest/backup/user-sau-main-dev/archive/user-sau-main-dev-coordinator/17-23/0000000100000000/000000010000000000000005-f878e33e4e3bac95e0cdce927872e18b9077f95c.lz4' on repo1
2026-01-18 23:46:27.574 P00 INFO: check command end: completed successfully (375ms)
[INFO] ✅ Stanza verification passed
[INFO] 8️⃣ Creating backup automation scripts...
[2026-01-18 23:46:27 UTC] USER=www-data EUID=0 PID=4076806 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|user-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-full-backup-user-sau-main-dev.sh
[2026-01-18 23:46:27 UTC] USER=www-data EUID=0 PID=4076815 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-full-backup-user-sau-main-dev.sh
[2026-01-18 23:46:27 UTC] USER=www-data EUID=0 PID=4076833 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|user-sau-main-dev-coordinator|g /usr/local/bin/pgbackrest-diff-backup-user-sau-main-dev.sh
[2026-01-18 23:46:27 UTC] USER=www-data EUID=0 PID=4076856 ACTION=fsop ARGS=chmod 755 /usr/local/bin/pgbackrest-diff-backup-user-sau-main-dev.sh
[INFO] ✅ Backup scripts created
[INFO] 9️⃣ Setting up cron jobs for automated backups...
[2026-01-18 23:46:27 UTC] USER=www-data EUID=0 PID=4076874 ACTION=fsop ARGS=chmod 644 /etc/cron.d/pgbackrest-user-sau-main-dev
[INFO] ✅ Cron jobs configured
[INFO] Schedule:
[INFO] - Full backup: Sundays at 2:00 AM
[INFO] - Differential backup: Mon-Sat at 2:00 AM
[INFO] 🔟 Creating restore documentation...
[2026-01-18 23:46:27 UTC] USER=www-data EUID=0 PID=4076892 ACTION=fsop ARGS=sed -i s|__STANZA_NAME__|user-sau-main-dev-coordinator|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-18 23:46:28 UTC] USER=www-data EUID=0 PID=4076901 ACTION=fsop ARGS=sed -i s|__ENV_ID__|user-sau-main-dev|g /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-18 23:46:28 UTC] USER=www-data EUID=0 PID=4076927 ACTION=fsop ARGS=chmod 644 /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[2026-01-18 23:46:28 UTC] USER=www-data EUID=0 PID=4076936 ACTION=fsop ARGS=chown postgres:postgres /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[INFO] ✅ Restore documentation created at: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[INFO] 1️⃣1️⃣ Taking first full backup...
[INFO] Verifying PostgreSQL coordinator service is active...
[INFO] Waiting for PostgreSQL to be ready...
[INFO] PostgreSQL coordinator is ready
[INFO] Initializing pgbackrest stanza...
2026-01-18 23:46:28.366 P00 INFO: start command begin 2.56.0: --exec-id=4076968-6b8dae68 --log-level-console=info --log-level-file=debug --stanza=user-sau-main-dev-coordinator
2026-01-18 23:46:28.366 P00 WARN: stop file does not exist for stanza user-sau-main-dev-coordinator
2026-01-18 23:46:28.366 P00 INFO: start command end: completed successfully (3ms)
[INFO] Upgrading stanza to match current PostgreSQL system-id...
2026-01-18 23:46:28.442 P00 INFO: stanza-upgrade command begin 2.56.0: --exec-id=4076985-3229c106 --log-level-console=info --log-level-file=debug --no-online --pg1-path=/data/postgresql/17/user-sau-main-dev/coordinator --pg1-port=5432 --pg1-socket-path=/var/run/postgresql-user-sau-main-dev-coordinator --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/var/lib/pgbackrest/backup/user-sau-main-dev --stanza=user-sau-main-dev-coordinator
2026-01-18 23:46:28.447 P00 INFO: stanza-upgrade for stanza 'user-sau-main-dev-coordinator' on repo1
2026-01-18 23:46:28.448 P00 INFO: stanza 'user-sau-main-dev-coordinator' on repo1 is already up to date
2026-01-18 23:46:28.448 P00 INFO: stanza-upgrade command end: completed successfully (9ms)
[INFO] This may take a few minutes depending on database size...
[2026-01-18 23:46:28 UTC] USER=www-data EUID=0 PID=4076989 ACTION=fsop ARGS=touch /var/log/pgbackrest/initial-backup-20260118-234628.log
[2026-01-18 23:46:28 UTC] USER=www-data EUID=0 PID=4076998 ACTION=fsop ARGS=chown postgres:postgres /var/log/pgbackrest/initial-backup-20260118-234628.log
[2026-01-18 23:46:28 UTC] USER=www-data EUID=0 PID=4077008 ACTION=fsop ARGS=chmod 644 /var/log/pgbackrest/initial-backup-20260118-234628.log
[INFO] Running backup (timeout: 10 minutes)...
[2026-01-18 23:46:34 UTC] USER=www-data EUID=0 PID=4077163 ACTION=fsop ARGS=cp /tmp/pgbackrest-backup-4076269.log /var/log/pgbackrest/initial-backup-20260118-234628.log
[INFO] ✅ Initial full backup completed successfully
[INFO] Log: /var/log/pgbackrest/initial-backup-20260118-234628.log
2026-01-18 23:46:34.647 P00 INFO: repo1: remove expired backup 20260118-213122F
2026-01-18 23:46:34.684 P00 INFO: repo1: remove archive path /var/lib/pgbackrest/backup/user-sau-main-dev/archive/user-sau-main-dev-coordinator/17-21
2026-01-18 23:46:34.686 P00 INFO: repo1: 17-22 no archive to remove
2026-01-18 23:46:34.686 P00 INFO: repo1: 17-23 no archive to remove
2026-01-18 23:46:34.686 P00 INFO: expire command end: completed successfully (43ms)
[INFO] Current backups:
stanza: user-sau-main-dev-coordinator
status: ok
cipher: aes-256-cbc
db (prior)
wal archive min/max (17): 000000010000000000000003/0000000100000000000000AE
full backup: 20260118-220155F
timestamp start/stop: 2026-01-18 22:01:55+00 / 2026-01-18 22:02:04+00
wal start/stop: 000000010000000000000003 / 000000010000000000000003
database size: 37.5MB, database backup size: 37.5MB
repo1: backup set size: 5.6MB, backup size: 5.6MB
full backup: 20260118-220214F
timestamp start/stop: 2026-01-18 22:02:14+00 / 2026-01-18 22:02:16+00
wal start/stop: 000000010000000000000006 / 000000010000000000000006
database size: 37.5MB, database backup size: 37.5MB
repo1: backup set size: 5.6MB, backup size: 5.6MB
db (current)
wal archive min/max (17): 000000010000000000000003/000000010000000000000006
full backup: 20260118-234609F
timestamp start/stop: 2026-01-18 23:46:09+00 / 2026-01-18 23:46:17+00
wal start/stop: 000000010000000000000003 / 000000010000000000000003
database size: 37.5MB, database backup size: 37.5MB
repo1: backup set size: 5.6MB, backup size: 5.6MB
full backup: 20260118-234628F
timestamp start/stop: 2026-01-18 23:46:28+00 / 2026-01-18 23:46:34+00
wal start/stop: 000000010000000000000006 / 000000010000000000000006
database size: 37.5MB, database backup size: 37.5MB
repo1: backup set size: 5.6MB, backup size: 5.6MB
[INFO] 🔟 Checking for worker configurations...
[INFO] ℹ️ No worker identifier provided - skipping worker backup setup
[INFO] (Run with 'worker-01', 'worker-02', etc. to configure worker backups)
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Backup setup complete!
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] ✅ Completed steps:
[INFO] 1. pgBackRest installed and configured
[INFO] 2. WAL archiving enabled (archive_mode=on)
[INFO] 3. PostgreSQL restarted with new settings
[INFO] 4. pgBackRest stanza initialized and verified
[INFO] 5. Initial full backup completed
[INFO] 6. Automated backup cron jobs configured
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[INFO] Configuration Details:
[INFO] Coordinator:
[INFO] Stanza: user-sau-main-dev-coordinator
[INFO] Schedule: Full: Sun 2AM, Diff: Mon-Sat 2AM
[INFO] Common:
[INFO] Backup dir: /var/lib/pgbackrest/backup/user-sau-main-dev
[INFO] Archive dir: /var/lib/pgbackrest/archive/user-sau-main-dev
[INFO] Config: /etc/pgbackrest/pgbackrest.conf
[INFO] Restore guide: /var/lib/pgbackrest/RESTORE_INSTRUCTIONS_user-sau-main-dev.md
[INFO] Retention:
[INFO] Full backups: 4 (keep last 4 full backups)
[INFO] Differential: 4 (keep last 4 diff per full)
[INFO] Archive WAL: Auto-managed by pgBackRest
[INFO] Manual commands:
[INFO] Coordinator: sudo -u postgres pgbackrest --stanza=user-sau-main-dev-coordinator backup
[INFO] List all backups: sudo -u postgres pgbackrest info
[INFO] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;32m✓[0m ✅ Backup setup completed for coordinator and all workers
[0;34m[INFO][0m Skipping 06-distribute-tables-canary.sh (test script - set RUN_TESTS=true to enable)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 07-distribute-tables.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:46:36 UTC] USER=unknown EUID=33 PID=4077225 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/metrics
[2026-01-18 23:46:36 UTC] USER=unknown EUID=33 PID=4077232 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/metrics
[2026-01-18 23:46:36 UTC] USER=unknown EUID=33 PID=4077239 ACTION=fsop ARGS=mkdir -p /var/log/fastorder/audit
[2026-01-18 23:46:36 UTC] USER=unknown EUID=33 PID=4077246 ACTION=fsop ARGS=chmod 777 /var/log/fastorder/audit
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m CITUS TABLE DISTRIBUTION
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m 🔐 Secure connection established
[0;34m[INFO][0m Host: db-user-sau-main-dev-postgresql-coordinator.fastorder.com:5432
[0;34m[INFO][0m Database: fastorder_user_sau_main_dev_db
[0;34m[INFO][0m SSL: verify-full (TLS 1.2+)
[0;34m[INFO][0m Timeouts: statement=120s, idle_tx=300s
[0;34m[INFO][0m 🔍 Running preflight checks...
[0;34m[INFO][0m Testing database connectivity...
[0;32m[OK][0m ✅ Database connection successful
[0;32m[OK][0m ✅ Connected to correct database: fastorder_user_sau_main_dev_db
[0;34m[INFO][0m Checking Citus extension in database fastorder_user_sau_main_dev_db...
[0;32m[OK][0m Citus version: 13.2-1
[0;34m[INFO][0m Checking worker registration...
[0;32m[OK][0m Registered workers: 1
[0;34m[INFO][0m Worker nodes:
[0;34m[INFO][0m nodename | nodeport | isactive | noderole
[0;34m[INFO][0m ---------------------------------------------------------+----------+----------+----------
[0;34m[INFO][0m db-user-sau-main-dev-postgresql-worker-01.fastorder.com | 5432 | t | primary
[0;34m[INFO][0m (1 row)
[0;34m[INFO][0m
[0;34m[INFO][0m 📊 Starting table distribution...
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Distributing: auth.login_account
[0;34m[INFO][0m Description: User authentication table - distributed by region for tenant isolation
[0;34m[INFO][0m Shard key: region_hint
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ⏭️ Table does not exist, skipping
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;32m[OK][0m ✅ All tables distributed successfully!
[0;34m[INFO][0m ═══════════════════════════════════════════════════════════════════════════════
[0;34m[INFO][0m 📊 Citus Cluster Summary:
[0;34m[INFO][0m Distributed tables:
[0;34m[INFO][0m table | type | shard_key | shards | size
[0;34m[INFO][0m ---------------------------+-------------+-----------+--------+---------
[0;34m[INFO][0m "user".contract_key | reference | <none> | 1 | 16 kB
[0;34m[INFO][0m "user".contract_type | reference | <none> | 1 | 16 kB
[0;34m[INFO][0m "user".contract_term_json | distributed | id | 32 | 512 kB
[0;34m[INFO][0m "user".contract_term_vars | distributed | id | 32 | 1792 kB
[0;34m[INFO][0m (4 rows)
[0;34m[INFO][0m
[0;34m[INFO][0m Worker capacity:
[0;34m[INFO][0m worker | total_shards | total_size
[0;34m[INFO][0m --------+--------------+------------
[0;34m[INFO][0m (0 rows)
[0;34m[INFO][0m
[0;32m[OK][0m Citus table distribution complete
[0;34m[INFO][0m Skipping 08-distribute-tables-rollback.sh (rollback script - run manually only)
[0;34m[INFO][0m Skipping 09-distribute-tables-test.sh (test script - set RUN_TESTS=true to enable)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Executing step: 10-setup-cdc.sh
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m CDC PIPELINE SETUP (Debezium + Elasticsearch Sink)
[0;34m[INFO][0m ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[0;34m[INFO][0m Log file: /var/log/fastorder/cdc/10-setup-cdc-*.log
[0;34m[INFO][0m Running CDC setup for identifier: coordinator
[2026-01-18 23:46:41] ==========================================
[2026-01-18 23:46:41] CDC SETUP SCRIPT STARTED
[2026-01-18 23:46:41] Log file: /var/log/fastorder/cdc/10-setup-cdc-20260118_234641.log
[2026-01-18 23:46:41] ==========================================
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
[2026-01-18 23:46:41] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:46:41] CDC Pipeline Setup (Debezium + ES Sink)
[2026-01-18 23:46:41] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:46:41] Environment: user-sau-main-dev
[2026-01-18 23:46:41] Identifier: coordinator
[2026-01-18 23:46:41] Service: user
[2026-01-18 23:46:41] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:46:41] 📂 CDC_BASE_DIR exists: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc
[2026-01-18 23:46:41] Looking for service folder: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user
[2026-01-18 23:46:41]
[2026-01-18 23:46:41] 📂 Found CDC configuration for service: user
[2026-01-18 23:46:41] Scanning for subservice directories in: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user
[2026-01-18 23:46:41] Found subservice: contracts, checking for steps at: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps
[2026-01-18 23:46:41]
[2026-01-18 23:46:41] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:46:41] Setting up CDC for: user/contracts
[2026-01-18 23:46:41] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:46:41] Found 8 step script(s) in /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps
[2026-01-18 23:46:41]
[2026-01-18 23:46:41] 🔧 Running: 00-create-eav-tables.sh
[2026-01-18 23:46:41] Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/00-create-eav-tables.sh
[2026-01-18 23:46:41] Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Create EAV Tables for CDC Pipeline
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Identifier: coordinator
Tables: user.contracts_int, user.contracts_json
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📝 Step 1: Creating EAV tables...
📥 Executing SQL...
BEGIN
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE FUNCTION
psql:/tmp/create-eav-tables-user-sau-main-dev.sql:61: NOTICE: trigger "trg_contracts_int_updated_at" for relation "user.contracts_int" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
ALTER TABLE
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE FUNCTION
psql:/tmp/create-eav-tables-user-sau-main-dev.sql:120: NOTICE: trigger "trg_contracts_json_updated_at" for relation "user.contracts_json" does not exist, skipping
DROP TRIGGER
CREATE TRIGGER
ALTER TABLE
COMMIT
psql:/tmp/create-eav-tables-user-sau-main-dev.sql:161: NOTICE: Created publication with all tables
DO
pubname | schemaname | tablename
------------------------+------------+----------------
cdc_pub_user_contracts | user | contracts
cdc_pub_user_contracts | user | contracts_int
cdc_pub_user_contracts | user | contracts_json
(3 rows)
✅ EAV tables created
🔍 Step 2: Verifying tables...
📊 Table: user.contracts_int
Table "user.contracts_int"
Column | Type | Collation | Nullable | Default
-------------+--------------------------+-----------+----------+--------------------------
id | uuid | | not null | utils.uuid_generate_v7()
tenant_id | character varying(100) | | not null |
contract_id | character(36) | | not null |
key | character varying(100) | | not null |
value_int | integer | | not null |
created_at | timestamp with time zone | | not null | now()
updated_at | timestamp with time zone | | not null | now()
Indexes:
"contracts_int_pkey" PRIMARY KEY, btree (id)
"idx_contracts_int_contract_id" btree (contract_id)
"idx_contracts_int_key" btree (key)
"idx_contracts_int_tenant_contract" btree (tenant_id, contract_id)
"uq_contracts_int_contract_key" UNIQUE CONSTRAINT, btree (contract_id, key)
Foreign-key constraints:
"fk_contracts_int_contract" FOREIGN KEY (contract_id) REFERENCES "user".contracts(id) ON DELETE CASCADE
Publications:
"cdc_pub_user_contracts"
📊 Table: user.contracts_json
Table "user.contracts_json"
Column | Type | Collation | Nullable | Default
-------------+--------------------------+-----------+----------+--------------------------
id | uuid | | not null | utils.uuid_generate_v7()
tenant_id | character varying(100) | | not null |
contract_id | character(36) | | not null |
key | character varying(100) | | not null |
value_json | jsonb | | not null |
created_at | timestamp with time zone | | not null | now()
updated_at | timestamp with time zone | | not null | now()
Indexes:
"contracts_json_pkey" PRIMARY KEY, btree (id)
"idx_contracts_json_contract_id" btree (contract_id)
"idx_contracts_json_key" btree (key)
"idx_contracts_json_tenant_contract" btree (tenant_id, contract_id)
"idx_contracts_json_value_gin" gin (value_json)
"uq_contracts_json_contract_key" UNIQUE CONSTRAINT, btree (contract_id, key)
Foreign-key constraints:
"fk_contracts_json_contract" FOREIGN KEY (contract_id) REFERENCES "user".contracts(id) ON DELETE CASCADE
Publications:
📊 Publication Tables:
user.contracts
user.contracts_int
user.contracts_json
📝 Step 3: Sample data commands (for testing)...
-- Insert sample INT attributes (tenant_id must match parent contract)
INSERT INTO "user".contracts_int (tenant_id, contract_id, "key", value_int)
VALUES
('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'max_users', 100),
('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'credit_limit', 50000),
('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'tier_level', 2)
ON CONFLICT (contract_id, "key")
DO UPDATE SET value_int = EXCLUDED.value_int, updated_at = NOW();
-- Insert sample JSON attributes (tenant_id must match parent contract)
INSERT INTO "user".contracts_json (tenant_id, contract_id, "key", value_json)
VALUES
('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'metadata', '{"lang":"en","tier":"gold"}'::jsonb),
('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'settings', '{"notifications":true,"theme":"dark"}'::jsonb),
('YOUR_TENANT_ID', 'YOUR_CONTRACT_ID', 'permissions', '{"admin":true,"export":true}'::jsonb)
ON CONFLICT (contract_id, "key")
DO UPDATE SET value_json = EXCLUDED.value_json, updated_at = NOW();
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
EAV Tables Created Successfully
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Tables:
- user.contracts_int
- user.contracts_json
Publication: cdc_pub_user_contracts
Next Steps:
1. Update Debezium connector table.include.list
2. Setup ksqlDB pipeline (05-setup-ksqldb-pipeline.sh)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:46:42] ✅ Completed: 00-create-eav-tables.sh
[2026-01-18 23:46:42]
[2026-01-18 23:46:42] 🔧 Running: 00b-migrate-tenant-id.sh
[2026-01-18 23:46:42] Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/00b-migrate-tenant-id.sh
[2026-01-18 23:46:42] Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Migration: Add tenant_id to EAV Tables
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Identifier: coordinator
Tables: user.contracts_int, user.contracts_json
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔍 Pre-flight: Checking current state...
✅ tenant_id column already exists in both tables
✅ tenant_id is already NOT NULL - migration complete
[2026-01-18 23:46:42] ✅ Completed: 00b-migrate-tenant-id.sh
[2026-01-18 23:46:42]
[2026-01-18 23:46:42] 🔧 Running: 01-setup-debezium-user-contracts.sh
[2026-01-18 23:46:42] Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/01-setup-debezium-user-contracts.sh
[2026-01-18 23:46:42] Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
✓ Centralized Secrets Manager library loaded
Location: /opt/fastorder/bash/infra_core/secrets/secrets-vault.sh
Functions: PostgreSQL (build_pg_secret_name, get_pg_credentials, set_pg_credentials)
Elasticsearch (build_es_secret_name, get_es_credentials, set_es_credentials)
Provider: aws
🔑 Configuring AWS credentials...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Debezium CDC Setup (User Contracts)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Identifier: coordinator
Table: user.contracts
Privacy: Minimal user index (GDPR compliant)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔍 Verifying Kafka infrastructure...
✅ db-user-sau-main-dev-postgresql.fastorder.com resolves to 10.100.1.231
🔐 psql will use client cert for mTLS.
🔐 Retrieving credentials from secrets vault...
✅ Credentials retrieved from secrets vault
🔐 Writing Debezium credentials to FileConfigProvider secrets file...
[2026-01-18 23:46:45 UTC] USER=www-data EUID=0 PID=4078051 ACTION=passthru ARGS=sed -i s|^debezium.database.password=.*|debezium.database.password=khWchI2QiMMgBOwFUgoMdOQzv| /opt/kafka/secrets/user-sau-main-dev/coordinator/connector-secrets.properties
✅ Updated Debezium credentials in /opt/kafka/secrets/user-sau-main-dev/coordinator/connector-secrets.properties
🔐 Syncing debezium_user password in PostgreSQL...
✅ debezium_user password synchronized
🔍 Checking PostgreSQL SSL status...
✅ Server SSL is ON.
🔧 Applying schema, publication & grants over TLS…
ALTER SYSTEM
pg_reload_conf
----------------
t
(1 row)
DROP PUBLICATION
CREATE PUBLICATION
NOTICE: Added user.contracts_int to publication
NOTICE: Added user.contracts_json to publication
DO
GRANT
GRANT
GRANT
GRANT
GRANT
✅ Publication & grants done.
⏳ Waiting for Kafka Connect @ https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083/connectors…
[2026-01-18 23:46:46] 🔗 Waiting for Kafka Connect at: https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083
[2026-01-18 23:46:46] ⏳ Waiting for HTTP endpoint: https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083
[2026-01-18 23:46:46] Expected codes: 200,500, timeout: 300s
[2026-01-18 23:46:46] ✅ HTTP endpoint ready: https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083 (code: 200, took: 0s)
[2026-01-18 23:46:46] 🔄 Testing Connect worker readiness...
[2026-01-18 23:46:46] ✅ Kafka Connect worker ready
🧹 Cleaning up existing Debezium connector and slot (if any)...
Step 0a: Also resetting ES Sink connector offsets (required for coordinated reset)...
→ Deleting ES Sink connector offsets...
→ Creating temporary ES Sink placeholder for offset deletion...
{"error_code":400,"message":"Connector configuration is invalid and contains the following 2 error(s):\nCould not connect to Elasticsearch. Error message: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused\nFailed to create client to verify connection. java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused\nYou can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`"}{"error_code":404,"message":"Unknown connector pg_user_sau_main_dev_coordinator_user_contracts_es_sink"} ⚠️ ES Sink offset deletion returned HTTP 404 (may be OK if no offsets existed)
→ Deleting ES Sink connector...
{"error_code":404,"message":"Connector pg_user_sau_main_dev_coordinator_user_contracts_es_sink not found"} ✓ ES Sink connector cleanup complete
Step 0b: Clearing stale Debezium connector offsets from Kafka Connect...
→ Stopping connector pg_user_sau_main_dev_user_contracts_debezium...
→ Deleting connector offsets (forces fresh snapshot)...
✓ Connector offsets deleted successfully (HTTP 200)
Step 1: Deleting Debezium connector...
Deleting connector: pg_user_sau_main_dev_user_contracts_debezium (attempt 1/10)
✓ Connector pg_user_sau_main_dev_user_contracts_debezium confirmed deleted
Step 2: Waiting for replication slot to become inactive...
✓ Slot slot_user_sau_main_dev_user_contracts does not exist (clean state)
Step 3: Dropping replication slot...
✓ Slot slot_user_sau_main_dev_user_contracts already dropped
Step 4: Final verification...
✅ Cleanup complete - environment is clean for fresh CDC snapshot
🔐 Checking Debezium SSL certificate permissions...
🔍 Validating Debezium SSL certificates...
🔐 Connector will use mTLS to Postgres.
✓ Certificate: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user.crt
✓ Key: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/debezium_user_pk8.der
✓ Root CA: /etc/fastorder/postgresql/certs/user-sau-main-dev/coordinator/ca.crt
📤 Upserting connector: PUT https://eventbus-user-sau-main-dev-kafka-connect.fastorder.com:8083/connectors/pg_user_sau_main_dev_user_contracts_debezium/config
Attempt 1/5: Sending PUT request to Kafka Connect...
(This may take up to 60s as Connect validates the configuration)
✅ Success (HTTP 201)
🌐 HTTP Response: 201
✅ Connector upserted.
🔄 Verifying connector task startup...
✅ Debezium connector task is RUNNING
ℹ️ Source table user.contracts has 0 rows.
ℹ️ Snapshot will be metadata-only; offsets may stay empty until first change.
⏳ Waiting for Debezium initial snapshot to complete...
📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
📊 Debezium snapshot status: unknown
⏳ Snapshot in progress... (0s elapsed)
⏳ Snapshot in progress... (5s elapsed)
⏳ Snapshot in progress... (10s elapsed)
📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
📊 Debezium snapshot status: unknown
⏳ Snapshot in progress... (15s elapsed)
⏳ Snapshot in progress... (20s elapsed)
⏳ Snapshot in progress... (25s elapsed)
📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
📊 Debezium snapshot status: unknown
⏳ Snapshot in progress... (30s elapsed)
⏳ Snapshot in progress... (35s elapsed)
⏳ Snapshot in progress... (40s elapsed)
📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
📊 Debezium snapshot status: unknown
⏳ Snapshot in progress... (45s elapsed)
⏳ Snapshot in progress... (50s elapsed)
⏳ Snapshot in progress... (55s elapsed)
📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
📊 Debezium snapshot status: unknown
⏳ Snapshot in progress... (60s elapsed)
⏳ Snapshot in progress... (65s elapsed)
⏳ Snapshot in progress... (70s elapsed)
📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
📊 Debezium snapshot status: unknown
⏳ Snapshot in progress... (75s elapsed)
⏳ Snapshot in progress... (80s elapsed)
⏳ Snapshot in progress... (85s elapsed)
📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
📊 Debezium snapshot status: unknown
⏳ Snapshot in progress... (90s elapsed)
⏳ Snapshot in progress... (95s elapsed)
⏳ Snapshot in progress... (100s elapsed)
📊 Slot status: restart_lsn=0/7066388, confirmed_flush_lsn=0/70663C0
📊 Debezium snapshot status: unknown
⏳ Snapshot in progress... (105s elapsed)
⏳ Snapshot in progress... (110s elapsed)
⏳ Snapshot in progress... (115s elapsed)
⚠️ Snapshot wait timeout (120s) on EMPTY table.
Offsets are still empty, but source table has 0 rows.
Proceeding anyway – CDC health will be verified by test inserts.
✅ Debezium connector is RUNNING after snapshot
🔍 Final verification: Checking Debezium offsets are recorded...
ℹ️ Source table has 0 rows - skipping offset verification
✅ Debezium connector verified RUNNING (empty source table)
🔄 Phase 2: Updating connector to snapshot.mode=initial...
✅ Connector updated to snapshot.mode=initial (HTTP 200)
✅ Connector verified RUNNING after Phase 2 update
✅ Debezium connector configured successfully (two-phase snapshot complete)
==================================================================
MULTI-TABLE CDC Pipeline Configuration
==================================================================
Tables:
- user.contracts (main table)
- user.contracts_int (EAV integer attributes)
- user.contracts_json (EAV JSON attributes)
Topics:
- cdc.user.contracts
- cdc.user.contracts_int
- cdc.user.contracts_json
COLUMN EXCLUSION (raw PII never leaves PostgreSQL):
user.contracts.email,user.contracts.phone
CAPTURED (safe for Kafka/ES):
id (PK), tenant_id, home_region, username,
display_name, email_hash, phone_hash, country_code,
region_code, tags, segments, contract info
DATA FLOW (Multi-Table CDC with ksqlDB Join):
PostgreSQL Tables (1:N)
↓ Debezium (CDC per table)
↓ Kafka Topics (3 separate topics)
↓ ksqlDB (pivot + join → flat document)
↓ Compacted Topic (search.user.contracts.v1)
↓ ES Sink (UPSERT)
↓ Elasticsearch (flat search index)
NEXT STEPS:
1. Run 00-create-eav-tables.sh (if not done)
2. Run 05-setup-ksqldb-pipeline.sh
3. Run 06-setup-es-sink-ksqldb.sh
4. Run 07-test-multi-table-cdc.sh
==================================================================
[2026-01-18 23:49:12] ✅ Completed: 01-setup-debezium-user-contracts.sh
[2026-01-18 23:49:12]
[2026-01-18 23:49:12] 🔧 Running: 01b-install-ksqldb.sh
[2026-01-18 23:49:12] Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/01b-install-ksqldb.sh
[2026-01-18 23:49:12] Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ksqlDB Installation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Identifier: coordinator
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔧 Allocating new VM_IP for ksqlDB: 10.100.1.234
🔧 Adding VM_IP 10.100.1.234 to loopback interface...
[2026-01-18 23:49:12 UTC] USER=www-data EUID=0 PID=4082191 ACTION=fsop ARGS=tee -a /etc/hosts
VM_IP: 10.100.1.234
FQDN: eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com
📦 Step 1: Checking Confluent Platform installation...
✅ ksqlDB already installed (version: )
📁 Step 2: Creating directories...
[2026-01-18 23:49:15 UTC] USER=www-data EUID=0 PID=4082262 ACTION=fsop ARGS=mkdir -p /var/lib/ksqldb/user-sau-main-dev/coordinator
[2026-01-18 23:49:15 UTC] USER=www-data EUID=0 PID=4082283 ACTION=fsop ARGS=mkdir -p /var/log/ksqldb/user-sau-main-dev/coordinator
[2026-01-18 23:49:15 UTC] USER=www-data EUID=0 PID=4082304 ACTION=fsop ARGS=mkdir -p /etc/ksqldb/user-sau-main-dev/coordinator
[2026-01-18 23:49:15 UTC] USER=www-data EUID=0 PID=4082325 ACTION=fsop ARGS=chown -R kafka:kafka /var/lib/ksqldb/user-sau-main-dev/coordinator /var/log/ksqldb/user-sau-main-dev/coordinator /etc/ksqldb/user-sau-main-dev/coordinator
✅ Directories created
⚙️ Step 3: Generating ksqlDB configuration...
[2026-01-18 23:49:15 UTC] USER=www-data EUID=0 PID=4082356 ACTION=fsop ARGS=mv /tmp/ksql-server-user-sau-main-dev.properties /etc/ksqldb/user-sau-main-dev/coordinator/ksql-server.properties
[2026-01-18 23:49:15 UTC] USER=www-data EUID=0 PID=4082385 ACTION=fsop ARGS=chown kafka:kafka /etc/ksqldb/user-sau-main-dev/coordinator/ksql-server.properties
[2026-01-18 23:49:15 UTC] USER=www-data EUID=0 PID=4082406 ACTION=fsop ARGS=chmod 640 /etc/ksqldb/user-sau-main-dev/coordinator/ksql-server.properties
✅ Configuration generated: /etc/ksqldb/user-sau-main-dev/coordinator/ksql-server.properties
🔧 Step 4: Creating systemd service...
[2026-01-18 23:49:15 UTC] USER=www-data EUID=0 PID=4082448 ACTION=fsop ARGS=mv /tmp/ksqldb-user-sau-main-dev-coordinator.service /etc/systemd/system/ksqldb-user-sau-main-dev-coordinator.service
[2026-01-18 23:49:15 UTC] USER=www-data EUID=0 PID=4082469 ACTION=passthru ARGS=systemctl daemon-reload
[2026-01-18 23:49:16 UTC] USER=www-data EUID=0 PID=4082528 ACTION=passthru ARGS=systemctl enable ksqldb-user-sau-main-dev-coordinator.service
✅ Systemd service created: ksqldb-user-sau-main-dev-coordinator.service
🚀 Step 5: Starting ksqlDB service...
🔍 Checking Kafka broker connectivity...
✅ Kafka broker is accessible
[2026-01-18 23:49:16 UTC] USER=www-data EUID=0 PID=4082591 ACTION=passthru ARGS=systemctl start ksqldb-user-sau-main-dev-coordinator.service
✅ ksqlDB service started
⏳ Waiting for ksqlDB to be ready...
..............................
🔍 Step 6: Verifying installation...
📊 Service Status:
[2026-01-18 23:50:17 UTC] USER=www-data EUID=0 PID=4084157 ACTION=passthru ARGS=systemctl status ksqldb-user-sau-main-dev-coordinator.service --no-pager -l
● ksqldb-user-sau-main-dev-coordinator.service - ksqlDB Server (user-sau-main-dev coordinator)
Loaded: loaded (/etc/systemd/system/ksqldb-user-sau-main-dev-coordinator.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2026-01-18 23:36:11 UTC; 14min ago
Docs: https://docs.ksqldb.io/
Main PID: 4024305 (java)
Tasks: 111 (limit: 19051)
Memory: 511.4M
CPU: 1min 17.395s
CGroup: /system.slice/ksqldb-user-sau-main-dev-coordinator.service
└─4024305 java -cp "/usr/share/java/ksqldb/*:/usr/share/java/rest-utils/*:/usr/share/java/confluent-common/*:" -Xms256m -Xmx512m -server -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:NewRatio=1 -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dksql.log.dir=/var/log/ksqldb/user-sau-main-dev/coordinator -Dlog4j.configuration=file:/etc/ksqldb/log4j.properties -Dksql.server.install.dir=/usr "-Xlog:gc*:file=/var/log/ksqldb/user-sau-main-dev/coordinator/ksql-server-gc.log:time,tags:filecount=10,filesize=102400" io.confluent.ksql.rest.server.KsqlServerMain /etc/ksqldb/user-sau-main-dev/coordinator/ksql-server.properties
Jan 18 23:49:28 web-03 ksql-server-start[4024305]: [2026-01-18 23:49:28,152] INFO Reporting thread saturation 0.001656360627351647 for _confluent-ksql-user-sau-main-dev_ksqldb_coordinatorquery_CTAS_CONTRACTS_JSON_AGG_289-f4d5c8dc-cd18-4200-9918-d2dee68acf57-StreamThread-3 (io.confluent.ksql.utilization.PersistentQuerySaturationMetrics:197)
Jan 18 23:49:28 web-03 ksql-server-start[4024305]: [2026-01-18 23:49:28,152] INFO Reporting thread saturation 0.001422370726561438 for _confluent-ksql-user-sau-main-dev_ksqldb_coordinatorquery_CTAS_CONTRACTS_JSON_AGG_289-f4d5c8dc-cd18-4200-9918-d2dee68acf57-StreamThread-2 (io.confluent.ksql.utilization.PersistentQuerySaturationMetrics:197)
Jan 18 23:49:28 web-03 ksql-server-start[4024305]: [2026-01-18 23:49:28,152] INFO Reporting thread saturation 0.0011361482767260137 for _confluent-ksql-user-sau-main-dev_ksqldb_coordinatorquery_CTAS_CONTRACTS_JSON_AGG_289-f4d5c8dc-cd18-4200-9918-d2dee68acf57-StreamThread-4 (io.confluent.ksql.utilization.PersistentQuerySaturationMetrics:197)
📊 ksqlDB Info:
⚠️ ksqlDB not responding yet (may still be starting)
📡 Step 7: Registering ksqlDB to Observability API...
🔄 Registering ksqlDB node to observability dashboard...
[INFO] Detected 4-part identifier format
[INFO] Registering node via API
[INFO] Application: ksqlDB
[INFO] Identifier: user-sau-main-dev-ksqldb-coordinator
[INFO] Identifier Parent: eventbus
[INFO] IP: 10.100.1.234
[INFO] Port: 8088
[INFO] FQDN: eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com
[INFO] Status: starting
[INFO] Environment: user-sau-main-dev (service=user, zone=sau, branch=main, env=dev)
[INFO] Calling registration API: https://skeleton.dev.fastorder.com/api/obs/register
[WARN] Registration API call failed (HTTP 500), retrying (1/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[22P02]: Invalid text representation: 7 ERROR: invalid input value for enum obs.instance_status: \"starting\"\nCONTEXT: unnamed portal parameter $8 = '...'"}
[WARN] Registration API call failed (HTTP 500), retrying (2/3)...
[WARN] Response: {"success":false,"error":"Registration failed: SQLSTATE[22P02]: Invalid text representation: 7 ERROR: invalid input value for enum obs.instance_status: \"starting\"\nCONTEXT: unnamed portal parameter $8 = '...'"}
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] ❌ REGISTRATION FAILED AFTER 3 ATTEMPTS
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ERROR] HTTP Status: 500
[ERROR] Response: {"success":false,"error":"Registration failed: SQLSTATE[22P02]: Invalid text representation: 7 ERROR: invalid input value for enum obs.instance_status: \"starting\"\nCONTEXT: unnamed portal parameter $8 = '...'"}
[ERROR]
[ERROR] API endpoint: https://skeleton.dev.fastorder.com/api/obs/register
[ERROR]
[ERROR] Troubleshooting:
[ERROR] 1. Check if skeleton.dev.fastorder.com is accessible
[ERROR] 2. Verify web application is running
[ERROR] 3. Check web application logs: /var/www/html/skeleton.dev.fastorder.com/logs/
[ERROR] 4. Test API manually:
[ERROR] curl -k -X POST 'https://skeleton.dev.fastorder.com/api/obs/register' \
[ERROR] -H 'Content-Type: application/json' \
[ERROR] -H 'X-Internal-Token: $OBS_INTERNAL_API_TOKEN' \
[ERROR] -d '$PAYLOAD'
[ERROR] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
⚠️ Failed to register ksqlDB (non-fatal)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ksqlDB Installation Complete
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Service: ksqldb-user-sau-main-dev-coordinator
VM_IP: 10.100.1.234
FQDN: eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com
Port: 8088
Config: /etc/ksqldb/user-sau-main-dev/coordinator/ksql-server.properties
Data: /var/lib/ksqldb/user-sau-main-dev/coordinator
Logs: /var/log/ksqldb/user-sau-main-dev/coordinator
Dashboard:
https://skeleton.dev.fastorder.com/dashboard/monitoring/environment2/<env-id>/service/ksqldb
CLI Access (with SSL):
ksql --ssl https://eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com:8088
REST API (HTTPS):
curl -k https://eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com:8088/info
curl -k https://eventbus-user-sau-main-dev-ksqldb-coordinator.fastorder.com:8088/ksql -H 'Content-Type: application/vnd.ksql.v1+json' -d '{"ksql": "SHOW STREAMS;"}'
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2026-01-18 23:50:22] ✅ Completed: 01b-install-ksqldb.sh
[2026-01-18 23:50:22]
[2026-01-18 23:50:22] 🔧 Running: 02-setup-ksqldb-pipeline.sh
[2026-01-18 23:50:22] Full path: /opt/fastorder/bash/scripts/env_app_setup/setup/05-db/engine/postgresql/steps/10-setup-cdc/user/contracts/steps/02-setup-ksqldb-pipeline.sh
[2026-01-18 23:50:22] Executing directly (script is executable)
[INFO] Loaded environment: user-sau-main-dev (svc=user zone=sau env=dev ip=142.93.238.16)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ksqlDB CDC Pipeline Setup
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment: user-sau-main-dev
Tables: user.contracts, contracts_int, contracts_json
Output: user_sau_main_dev_user_contracts
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔍 Step 0: Checking ksqlDB availability...
⏳ Waiting for ksqlDB to be ready...
Still waiting... (5s/60s)
Still waiting... (10s/60s)
Still waiting... (15s/60s)
Still waiting... (20s/60s)
Still waiting... (25s/60s)
Still waiting... (30s/60s)
Still waiting... (35s/60s)
Still waiting... (40s/60s)
Still waiting... (45s/60s)
Still waiting... (50s/60s)
Still waiting... (55s/60s)
Still waiting... (60s/60s)
❌ ksqlDB is not running at https://10.100.1.234:8088 or http://10.100.1.234:8088
Please install ksqlDB first:
/opt/fastorder/bash/scripts/env_app_setup/setup/04-eventbus/engine/kafka/steps/20-install-ksqldb.sh
[2026-01-18 23:51:23] ❌ FAILED: 02-setup-ksqldb-pipeline.sh (exit code: 1)
[2026-01-18 23:51:23] ❌ CRITICAL: This is a required step for CDC pipeline. Aborting.
[0;31m[ERROR][0m ❌ Database infrastructure (postgresql) setup failed with exit code: 1
⏳ This step is pending and will execute after the previous steps complete successfully.
Loading logs...