Skip to content
/ terraform-aws-elasticsearch Public

Module that deploys an Elasticsearch cluster

0 stars 0 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

infrahouse/terraform-aws-elasticsearch

BranchesTags

Repository files navigation

terraform-aws-elasticsearch

The module deploys a multi-node Elasticsearch cluster.

Usage

Dependencies

The module requires several additional components that are needed to provision the Elasticsearch cluster.

  • At least two subnets to place a load balancer and autoscaling group.
  • Route53 zone - the cluster will have an HTTPS endpoint for the cluster.

Service network

The easiest way to create subnets in AWS is to use the Service Network Terraform module.

Typical configuration would include at least two public and two private subnets.

module "service-network" {
  source                = "infrahouse/service-network/aws"
  version               = "~> 2.0"
  service_name          = "elastic"
  vpc_cidr_block        = "10.1.0.0/16"
  management_cidr_block = "10.1.0.0/16"
  subnets = [
    {
      cidr                    = "10.1.0.0/24"
      availability-zone       = data.aws_availability_zones.available.names[0]
      map_public_ip_on_launch = true
      create_nat              = true
      forward_to              = null
    },
    {
      cidr                    = "10.1.1.0/24"
      availability-zone       = data.aws_availability_zones.available.names[1]
      map_public_ip_on_launch = true
      create_nat              = true
      forward_to              = null
    },
    {
      cidr                    = "10.1.2.0/24"
      availability-zone       = data.aws_availability_zones.available.names[0]
      map_public_ip_on_launch = false
      create_nat              = false
      forward_to              = "10.1.0.0/24"
    },
    {
      cidr                    = "10.1.3.0/24"
      availability-zone       = data.aws_availability_zones.available.names[1]
      map_public_ip_on_launch = false
      create_nat              = false
      forward_to              = "10.1.1.0/24"
    }
  ]
}

Route53 Zone

The module will create an A record for the cluster in a specified zone. If the cluster name (passed as var.cluster_name) is 'elastic', the client URL is going to be https://elastic.ci-cd.infrahouse.com. The zone can be created in the same Terraform module or accessed as a data source.

data "aws_route53_zone" "cicd" {
  name = "ci-cd.infrahouse.com"
}

Bootstrapping cluster

Any new cluster needs to be bootstrapped first. Let's say we want to create a three node cluster. Declare the cluster and add bootstrap_mode = true to the module inputs. The size of the autoscaling group will be not three, but one node.

module "test" {
  module "test" {
    source  = "registry.infrahouse.com/infrahouse/elasticsearch/aws"
    version = "2.1.5"
    
    providers = {
      aws     = aws
      aws.dns = aws
    }
    internet_gateway_id = module.service-network.internet_gateway_id
    key_pair_name       = aws_key_pair.test.key_name
    subnet_ids          = module.service-network.subnet_public_ids
    zone_id             = data.aws_route53_zone.cicd.zone_id
    bootstrap_mode      = true
  }
}

Provisioning remaining nodes

After the cluster is bootstrapped, disable the bootstrap mode.

diff --git a/test_data/test_module/main.tf b/test_data/test_module/main.tf
index c13df0d..33cf0d3 100644
--- a/test_data/test_module/main.tf
+++ b/test_data/test_module/main.tf
@@ -12,5 +12,5 @@ module "test" {
   subnet_ids                    = module.service-network.subnet_private_ids
   zone_id                       = data.aws_route53_zone.cicd.zone_id
-  bootstrap_mode                = true
+  bootstrap_mode                = false
 }

Accessing the cluster

The module creates three endpoints to access the cluster. All three of them are output variables of the module.

  • Master nodes: https://${var.cluster_name}-master.${data.aws_route53_zone.cluster.name} or https://${var.cluster_name}.${data.aws_route53_zone.cluster.name}
  • Data nodes: https://${var.cluster_name}-data.${data.aws_route53_zone.cluster.name}

Requirements

Name Version
terraform ~> 1.5
aws ~> 5.11
cloudinit ~> 2.3
random ~> 3.6
tls ~> 4.0

Providers

Name Version
aws ~> 5.11
aws.dns ~> 5.11
random ~> 3.6
tls ~> 4.0

Modules

Name Source Version
ca_cert_secret registry.infrahouse.com/infrahouse/secret/aws ~> 1.0
ca_key_secret registry.infrahouse.com/infrahouse/secret/aws ~> 1.0
elastic_cluster registry.infrahouse.com/infrahouse/website-pod/aws 5.0.0
elastic_cluster_data registry.infrahouse.com/infrahouse/website-pod/aws 5.0.0
elastic_data_userdata registry.infrahouse.com/infrahouse/cloud-init/aws 1.13.1
elastic_master_userdata registry.infrahouse.com/infrahouse/cloud-init/aws 1.13.1
update-dns registry.infrahouse.com/infrahouse/update-dns/aws 0.9.0
update-dns-data registry.infrahouse.com/infrahouse/update-dns/aws 0.9.0

Resources

Name Type
aws_s3_bucket.snapshots-bucket resource
aws_s3_bucket_public_access_block.public_access resource
aws_secretsmanager_secret.elastic resource
aws_secretsmanager_secret.kibana_system resource
aws_secretsmanager_secret_version.elastic resource
aws_secretsmanager_secret_version.kibana_system resource
aws_security_group.backend_extra resource
aws_vpc_security_group_ingress_rule.backend_extra_reserved resource
aws_vpc_security_group_ingress_rule.elastic_exporter resource
aws_vpc_security_group_ingress_rule.node_exporter resource
random_password.elastic resource
random_password.kibana_system resource
random_string.bucket_prefix resource
random_string.profile-suffix resource
tls_private_key.ca_key resource
tls_self_signed_cert.ca_cert resource
aws_ami.ubuntu data source
aws_availability_zones.available data source
aws_caller_identity.current data source
aws_iam_policy_document.elastic_permissions data source
aws_iam_policy_document.secrets-permission-policy data source
aws_iam_role.caller_role data source
aws_region.current data source
aws_route53_zone.cluster data source
aws_subnet.selected data source

Inputs

Name Description Type Default Required
asg_ami Image for EC2 instances string null no
asg_create_initial_lifecycle_hook Used for migration from version 1.* bool true no
asg_health_check_grace_period ASG will wait up to this number of seconds for instance to become healthy number 900 no
bootstrap_mode Set this to true if the cluster is to be bootstrapped bool true no
cluster_data_count Number of data nodes in the cluster number 3 no
cluster_master_count Number of master nodes in the cluster number 3 no
cluster_name How to name the cluster string "elastic" no
environment Name of environment. string "development" no
extra_files Additional files to create on an instance. 
list(object({
    content     = string
    path        = string
    permissions = string
  }))
 [] no
extra_repos Additional APT repositories to configure on an instance. 
map(object({
    source = string
    key    = string
  }))
 {} no
instance_type Instance type to run the elasticsearch node string "t3.medium" no
instance_type_data Instance type to run the elasticsearch data node. If null, use var.instance_type. string null no
instance_type_master Instance type to run the elasticsearch master node. If null, use var.instance_type. string null no
internet_gateway_id Not used, but AWS Internet Gateway must be present. Ensure by passing its id. string n/a yes
key_pair_name SSH keypair name to be deployed in EC2 instances string n/a yes
max_instance_lifetime_days The maximum amount of time, in _days_, that an instance can be in service, values must be either equal to 0 or between 7 and 365 days. number 0 no
monitoring_cidr_block CIDR range that is allowed to monitor elastic instances. string null no
packages List of packages to install when the instances bootstraps. list(string) [] no
puppet_debug_logging Enable debug logging if true. bool false no
puppet_hiera_config_path Path to hiera configuration file. string "{root_directory}/environments/{environment}/hiera.yaml" no
puppet_module_path Path to common puppet modules. string "{root_directory}/environments/{environment}/modules:{root_directory}/modules" no
puppet_root_directory Path where the puppet code is hosted. string "/opt/puppet-code" no
root_volume_size Root volume size in EC2 instance in Gigabytes number 30 no
secret_elastic_readers List of role ARNs that will have permissions to read elastic superuser secret. list(string) null no
smtp_credentials_secret AWS secret name with SMTP credentials. The secret must contain a JSON with user and password keys. string null no
snapshot_bucket_prefix A string prefix to a bucket name for snapshots. Random by default. string null no
snapshot_force_destroy Destroy S3 bucket with Elasticsearch snapshots even if non-empty bool false no
ssh_cidr_block CIDR range that is allowed to SSH into the elastic instances. string "0.0.0.0/0" no
subnet_ids List of subnet ids where the elasticsearch instances will be created list(string) n/a yes
ubuntu_codename Ubuntu version to use for the elasticsearch node string "jammy" no
zone_id Domain name zone ID where the website will be available string n/a yes

Outputs

Name Description
cluster_data_url HTTPS endpoint to access the cluster data nodes
cluster_master_url HTTPS endpoint to access the cluster masters
cluster_url HTTPS endpoint to access the cluster
data_instance_role_arn Data node EC2 instance profile will have this role ARN
elastic_password Password for Elasticsearch superuser elastic.
elastic_secret_id AWS secret that stores password for user elastic.
kibana_system_password A password of kibana_system user
kibana_system_secret_id AWS secret that stores password for user kibana_system
master_instance_role_arn Master node EC2 instance profile will have this role ARN
snapshots_bucket AWS S3 Bucket where Elasticsearch snapshots will be stored.

About

Module that deploys an Elasticsearch cluster

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

41 tags

Packages

No packages published

Contributors 2

  •  
  •  

Languages