Make CML 2.8 work a on AWS and Azure (#30)

* Fix CML 2.8 deployments on AWS and Azure (#29) * cml aws deploy fix * fix azure deployment * add missing comments --------- Co-authored-by: Oliver Uhlar -X (ouhlar - PANTHEON TECH SRO at Cisco) <[email protected]> * Reverting some changes - reverting the allowed_ipv4_subnet changes from previous commit - update the VMX patch script - fix some white space issues in shell scripts * update copyright to include 2025 * Fixes for CML 2.8 and more - overall, should fix #27 to allow deployments on AWS and Azure with 2.8 - changed variable names for AWS access and secret keys to start with aws_ (e.g. TF_VAR_access_key -> TF_VAR_aws_access_key), same for Azure subscription_id and tenant_id. - do product licensing using the PCL instead of the cml.sh script (fixes #28) - fix letsencrypt certificate installation for nginx and Cockpit - add i386 repository to allow installation of older libraries for IOL compatibility - whitespace and formatting of files - for Azure, use the "minimal" SKU of the Noble image (instead of "server") - slightly more comprehensive log output in cml.sh to see where time is spent when deploying * Fixes and more documentation updates - improved the license script to allow for no license to be installed - removed the unnecessary wait in the customize script - some cosmetic output fixes - documentation updates, including the CHANGELOG - fixed the .envrc.example * Yet another small doc correction * updated config.yml template/example --------- Co-authored-by: Oliver Uhlar <[email protected]> Co-authored-by: Oliver Uhlar -X (ouhlar - PANTHEON TECH SRO at Cisco) <[email protected]>
CiscoDevNet · Jan 20, 2025 · dd1212d · dd1212d
1 parent eee45ce
commit dd1212d
Show file tree

Hide file tree

Showing 65 changed files with 341 additions and 241 deletions.
diff --git a/.envrc.example b/.envrc.example
@@ -25,8 +25,8 @@
 # Azure
 #
 
-#export TF_VAR_subscription_id=""
-#export TF_VAR_tenant_id=""
+#export TF_VAR_azure_subscription_id=""
+#export TF_VAR_azure_tenant_id=""
 
 #########
 # Secrets

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,6 +2,24 @@
 
 Lists the changes for the tool releases.
 
+## Version 2.8.0
+
+- using "aws\_" and "azure\_" prefixes to provide tokens and IDs in the environment (see `.envrc.example`)
+- adapt tooling to work with 2.8.0 (move base OS from 20.04 to 24.04)
+- allow to use the `allowed_ipv4_subnets` also for Azure
+- improve network manager handling while provisioning
+- licensing now uses the PCL instead of curl and bash
+- documentation improvements and fixes
+
+## Version 2.7.2
+
+- added the AWS mini variant which does not manage any network resources, the
+  subnet and security group ID
+- change elastic IP allocation for AWS from dynamic to static to make it work
+  again
+- this is the last release to support CML 2.7 and before
+- changed the versioning to match the CML version so that it's easier to find the proper version / release of cloud-cml which works with the CML version to be used
+
 ## Version 0.3.0
 
 - allow cluster deployments on AWS.

diff --git a/LICENSE b/LICENSE
@@ -187,7 +187,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright 2019-2024 Cisco
+   Copyright 2019-2025 Cisco
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

diff --git a/README.md b/README.md
@@ -1,16 +1,16 @@
 # README
 
-Version 0.3.1, November 29 2024
+Version 2.8.0, January 10 2025
 
-With CML 2.7, you can run CML instances on Azure and AWS.  We have tested CML deployments using this tool chain in both clouds.  **The use of this tool is considered BETA**.  The tool has certain requirements and prerequisites which are described in this README and in the [documentation](documentation) directory.
+CML instances can run on Azure and AWS cloud infrastructure.  This repository provides automation tooling using Terraform to deploy and manage CML in the cloud.  We have tested CML deployments using this tool chain in both clouds.  **The use of this tool is considered BETA**.  The tool has certain requirements and prerequisites which are described in this README and in the [documentation](documentation) directory.
 
 *It is very likely that this tool chain can not be used "as-is"*.  It should be forked and adapted to specific customer requirements and environments.
 
 > [!IMPORTANT]
 >
 > **Version 2.7 vs 2.8**
 >
-> CML2 version 2.8 has been released in November 2024.  As CML 2.8 uses Ubuntu 24.04 as the base operating system, cloud-cml needs to accommodate for that during image selection when bringing up the VM on the hosting service (AWS, Azure, ...).  This means that going forward, cloud-cml will support 2.8 and not 2.7 anymore.  This release will be the last that does support CML 2.7!
+> CML2 version 2.8 has been released in November 2024.  As CML 2.8 uses Ubuntu 24.04 as the base operating system, cloud-cml needs to accommodate for that during image selection when bringing up the VM on the hosting service (AWS, Azure, ...).  This means that going forward, cloud-cml supports 2.8 and not 2.7 anymore.  If CML versions earlier than CML 2.8 should be used then please select the release with the tag `v2.7.2` that still supports CML 2.7!
 >
 > **Support:**
 >
@@ -118,17 +118,21 @@ Regardless of the secret manager in use or whether you use random passwords or n
 ##### CyberArk Conjur installation
 
 > [!IMPORTANT]
-> CyberArk Conjur is not currently in the Terraform Registry.  You must follow its [installation instructions](https://github.com/cyberark/terraform-provider-conjur?tab=readme-ov-file#terraform-provider-conjur) before running `terraform init`. 
+> CyberArk Conjur is not currently in the Terraform Registry.  You must follow its [installation instructions](https://github.com/cyberark/terraform-provider-conjur?tab=readme-ov-file#terraform-provider-conjur) before running `terraform init`.
 
 These steps are only required if using CyberArk Conjur as an external secrets manager.
+
 1. Download the [CyberArk Conjur provider](https://github.com/cyberark/terraform-provider-conjur/releases).
 2. Copy the custom provider to `~/.terraform.d/plugins/localhost/cyberark/conjur/<version>/<architecture>/terraform-provider-conjur_v<version>`
+
    ```bash
    $ mkdir -vp ~/.terraform.d/plugins/localhost/cyberark/conjur/0.6.7/darwin_arm64/
    $ unzip ~/terraform-provider-conjur_0.6.7-4_darwin_arm64.zip -d ~/.terraform.d/plugins/localhost/cyberark/conjur/0.6.7/darwin_arm64/
    $
    ```
+
 3. Create a `.terraformrc` file in the user's home:
+
    ```hcl
    provider_installation {
      filesystem_mirror {
@@ -145,19 +149,16 @@ These steps are only required if using CyberArk Conjur as an external secrets ma
 
 Terraform can be downloaded for free from [here](https://developer.hashicorp.com/terraform/downloads). This site has also instructions how to install it on various supported platforms.
 
-Deployments of CML using Terraform were tested using the versions mentioned below on Ubuntu Linux and macOS.
+Deployments of CML using Terraform were tested using the versions mentioned below on Ubuntu Linux.
 
 ```bash
 $ terraform version
-Terraform v1.8.0
-on darwin_arm64
-+ provider registry.terraform.io/ciscodevnet/cml2 v0.7.0
-+ provider registry.terraform.io/hashicorp/aws v5.45.0
-+ provider registry.terraform.io/hashicorp/azurerm v3.99.0
-+ provider registry.terraform.io/hashicorp/cloudinit v2.3.3
+Terraform v1.10.4
+on linux_amd64
++ provider registry.terraform.io/ciscodevnet/cml2 v0.8.1
++ provider registry.terraform.io/hashicorp/aws v5.83.0
++ provider registry.terraform.io/hashicorp/cloudinit v2.3.5
 + provider registry.terraform.io/hashicorp/random v3.6.1
-+ provider registry.terraform.io/hashicorp/vault v4.2.0
-+ provider localhost/cyberark/conjur v0.6.7
 $
 ```
 
@@ -179,12 +180,14 @@ There's two Terraform variables which can be defined / set to further customize
 - `cfg_file`: This variable defines the configuration file.  It defaults to `config.yml`.
 - `cfg_extra_vars`: This variable defines the name of a file with additional variable definitions.  The default is "none".
 
+Here's an example of an `.envrc` file to set environment variable.  Note the last two lines which define the configuration file to use and the extra shell file which defines additional environment variables.
+
 ```bash
-export TF_VAR_access_key="aws-something"
-export TF_VAR_secret_key="aws-somethingelse"
+export TF_VAR_aws_access_key="aws-something"
+export TF_VAR_aws_secret_key="aws-somethingelse"
 
-# export TF_VAR_subscription_id="azure-something"
-# export TF_VAR_tenant_id="azure-something-else"
+# export TF_VAR_azure_subscription_id="azure-something"
+# export TF_VAR_azure_tenant_id="azure-something-else"
 
 export TF_VAR_cfg_file="config-custom.yml"
 export TF_VAR_cfg_extra_vars="extras.sh"
@@ -199,9 +202,9 @@ CFG_HN="domainname"
 CFG_EMAIL="[email protected]"
 ```
 
-In this example, four additional variables are defined which can be used in customization scripts during deployment to provide data (usernames, passwords, ...) for specific services like configuring DNS.
+In this example, four additional variables are defined which can be used in customization scripts during deployment to provide data (usernames, passwords, ...) for specific services like configuring DNS.  See the `03-letsencrypt.sh` file which installs a valid certificate into CML, using LetsEncrypt and DynDNS for domain name services.
 
-See the AWS specific document for additional information how to define variables in the environment using tools like `direnv` ("Terraform variable definition").
+See the AWS specific document for additional information how to define variables in the environment using tools like `direnv`  or `mise`.
 
 ## Additional customization scripts
 
@@ -224,6 +227,6 @@ Extra variable definitions and additional scripts will all be stored in the user
 
 All scripts are copied as they are including all comments which will require even more space.
 
-A potential solution to the data limit is to provide the scripts in storage by bundling them up into a tar file or similar, store the tar file in S3 and then only reference this file in the user-data.  However, this hasn't been implemented, yet.
+Cloud-cml currently uses the cloud-init Terraform provider which allows compressed storage of this data.  This allows to store more scripts and configuration due to the compression.  The 16KB limit is still in place for the compressed data, though.
 
 EOF
diff --git a/config.yml b/config.yml
@@ -1,6 +1,6 @@
 #
 # This file is part of Cisco Modeling Labs
-# Copyright (c) 2019-2024, Cisco Systems, Inc.
+# Copyright (c) 2019-2025, Cisco Systems, Inc.
 # All rights reserved.
 #
 
@@ -129,7 +129,7 @@ app:
   # **No longer used, see the secret manager section above**
   #user: admin
   #pass: your-secret-password
-  software: cml2_2.7.2-26_amd64-29.pkg
+  software: cml2_2.8.0-6_amd64-32.pkg
   # The list must have at least ONE element, this is what the dummy is for in
   # case 00- and 01- are commented out!
   customize:
@@ -148,50 +148,49 @@ license:
 # be available prior to starting an instance. Ensure that each definition has
 # also a corresponding image! A smaller selection: less copying, faster bring-up
 # time!
+#
+# NOTE: the list below should be adapted to your needs and also to the exact
+# names of the files which have been copied to your cloud storage.  Ensure
+# that those files are there!
 refplat:
   definitions:
     - alpine
-    # - alpine-trex
-    # - alpine-wanem
-    # - asav
-    # - cat8000v
-    # - cat9000v-q200
-    # - cat9000v-uadp
-    # - cat-sdwan-edge
-    # - cat-sdwan-controller
-    # - cat-sdwan-manager
-    # - cat-sdwan-validator
-    # - cat-sdwan-vedge
-    # - csr1000v
-    # - desktop
+    - alpine-trex
+    - alpine-wanem
+    - asav
+    - cat8000v
+    - cat9000v-q200
+    - cat9000v-uadp
+    - cat-sdwan-edge
+    - cat-sdwan-controller
+    - cat-sdwan-manager
+    - cat-sdwan-validator
+    - cat-sdwan-vedge
+    - csr1000v
+    - desktop
     - iol-xe
     - ioll2-xe
     - iosv
     - iosvl2
-    # - iosxrv9000
-    # - nxosv9000
+    - iosxrv9000
+    - nxosv9000
     - server
     - ubuntu
   images:
-    - alpine-3-19-1-base
-    # - alpine-3-19-1-trex
-    # - alpine-3-19-1-wanem
-    # - asav-9-20-2
-    # - cat8000v-17-13-01a
-    # - cat9000v-q200-17-12-01prd9
-    # - cat9000v-uadp-17-12-01prd9
-    # - cat-sdwan-edge-17-13-01a
-    # - cat-sdwan-controller-20-13-1
-    # - cat-sdwan-manager-20-13-1
-    # - cat-sdwan-validator-20-13-1
-    # - cat-sdwan-vedge-20-13-1
-    # - csr1000v-17-03-068a
-    # - desktop-3-19-1-xfce
-    - iol-xe-17-12-01
-    - ioll2-xe-17-12-01
-    - iosv-159-3-m8
+    - alpine-base-3-20-3
+    - alpine-desktop-3-20-3
+    # - alpine-trex-3-20-3
+    # - alpine-wanem-3-20-3
+    # - asav-9-22-1-1
+    # - cat8000v-17-15-01a
+    # - cat9000v-q200-17-15-01
+    # - cat9000v-uadp-17-15-01
+    # - csr1000v-17-03-08a
+    - iol-xe-17-15-01
+    - ioll2-xe-17-15-01
+    - iosv-159-3-m9
     - iosvl2-2020
-    # - iosxrv9000-7-11-1
-    # - nxosv9300-10-4-2-f
-    - server-tcl-14-1
-    - ubuntu-22-04-20240126
+    # - iosxrv9000-24-3-1
+    # - nxosv9300-10-5-1-f
+    - server-tcl-15-0
+    - ubuntu-24-04-20241004
diff --git a/documentation/AWS.md b/documentation/AWS.md
@@ -1,11 +1,9 @@
 # AWS
 
-Version 0.3.1, November 2024
-
 This document contains specific configuration steps to deploy a CML instance in AWS. Some sections from the top level document are repeated here with additional detail regarding AWS.
 
 > [!IMPORTANT]
-> The repository includes an alternative deployment method for AWS (aws-mini) which does not create any network resources.  It therefore relies on these resources to be available at the time of deploying CML.  See the "Mini vs regular deployments" section below!
+> The repository includes an alternative deployment method for AWS (aws-mini) which does not create any network resources.  It therefore relies on these resources to be available at the time of deploying CML.  See the ["Mini vs regular deployments"](#mini-vs-regular-deployments) section below!
 
 ## General requirements
 
@@ -44,7 +42,7 @@ As mentioned at the top, there's an `aws-mini` deployment option as an alternati
 
 The mini flavor is useful in case the AWS networking infrastructure is already in place and can not or should not be modified, cloud-cml should simply create a CML instance that uses the existing networking infrastructure by providing the subnet ID and the security group ID that should be used to attach the CML VM to.
 
-*If no Elastic IP should be used and the server should use a private IP from the configured subnet instead then this is configurable in the .tf file.  See the comment for the `resource "aws_eip" "server_eip"` inside of `main.tf` for the mini variant. 
+If no Elastic IP should be used and the server should use a private IP from the configured subnet instead then this is configurable in the .tf file.  See the comment for the `resource "aws_eip" "server_eip"` inside of `main.tf` for the mini variant.
 
 #### How to enable the mini variant
 
@@ -303,7 +301,7 @@ The final step is about creating access credentials that can be used with Terraf
 - click on "Create access key"
 - make a note of the access key and the secret key (copy them into an editor so that they can be later used when editing the `config.yml` of the deployment tool)
 
-This access key and the associated secret key must be provided to the AWS Terraform provider via the the variables `access_key` and `secret_key`, ideally via environment variables or a vault. See the Variables section below.
+This access key and the associated secret key must be provided to the AWS Terraform provider via the variables `aws_access_key` and `aws_secret_key`, ideally via environment variables or a vault. See the [Variables section](#terraform-variable-definition) below.
 
 #### Example
 
@@ -367,7 +365,7 @@ AWS CLI configurations are stored in `$HOME/.aws`.
 
 If everything was configured correct then you should be able to list instances (remember that we permitted EC2 access for the deployment users):
 
-```
+```bash
 $ aws ec2 describe-instances
 {
     "Reservations": []
@@ -379,7 +377,7 @@ As there are no instances running in this case, the output is empty. The importa
 
 ### Configuration file
 
-CML specific settings are specified in the configuration file `config.yml`.  See also [VPC support](#vpc-support) and [Cluster support](#cluster-suport) sections further down in the document.
+CML specific settings are specified in the configuration file `config.yml`.  See also [VPC support](#vpc-support) and [Cluster support](#cluster-support) sections further down in the document.
 
 #### AWS section
 
@@ -515,7 +513,7 @@ Start the tool by providing the bucket name as an argument and the location of t
 
 The tool will then display a simple dialog where the images which should be copied to the bucket can be selected:
 
-![](../images/upload-refplat.png)
+![Dialog preview](../images/upload-refplat.png)
 
 After selecting OK the upload process will be started immediately. To abort the process, Ctrl-C can be used.
 
@@ -528,17 +526,19 @@ Help can be obtained via `./upload-images-to-aws.sh --help`.
 
 The `variable.tf` defines the authentication secrets needed by the Terraform AWS provider.
 
-Here's an example using a bash script that can be sourced and which defines those variables. To automate things further, a tool like [direnv](https://direnv.net/) can be used to load this environment when changing into the directory which has this file.
+Here's an example using a bash script that can be sourced and which defines those variables. To automate things further, a tool like [direnv](https://direnv.net/) or [mise-en-place](https://mise.jdx.dev/) can be used to load this environment when changing into the directory which has this file.
 
 Content of file `.envrc`:
 
 ```bash
-export TF_VAR_access_key="your-access-key-string-from-iam"
-export TF_VAR_secret_key="your-secret-key-string-from-iam"
+export TF_VAR_aws_access_key="your-access-key-string-from-iam"
+export TF_VAR_aws_secret_key="your-secret-key-string-from-iam"
 ```
 
 Alternatively, it's also possible to provide values for variables via a file called `terraform.tfvars` file. There are various ways how to define / set variables with Terraform. See the Terraform [documentation](https://developer.hashicorp.com/terraform/language/values/variables#assigning-values-to-root-module-variables) for additional details.
 
+In addition to the above methods, Terraform can also automatically retrieve authentication credentials from the AWS configuration files located in the .aws folder. This includes credentials set up by running `aws configure`, which stores your access key and secret key in the `~/.aws/credentials` file. This method allows Terraform to use the same credentials configured for the AWS CLI, [documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs).
+
 ## Lifecycle management
 
 When all requirements are met, an instance can be deployed using Terraform.

diff --git a/documentation/Azure.md b/documentation/Azure.md
@@ -1,7 +1,5 @@
 # Azure
 
-Version 0.3.0, April 17 2024
-
 This document explains specific configuration steps to deploy a Cisco Modeling Labs (CML) instance in Azure.
 
 ## General requirements
@@ -45,8 +43,8 @@ The provided subscription ID and the tenant ID need to be configured as Terrafor
 
 { read subID ; read tenantID; } <<< "$(az account list --output=json | jq -r '.[0]|.id,.tenantId')"
 
-export TF_VAR_tenant_id="$tenantID"
-export TF_VAR_subscription_id="$subID"
+export TF_VAR_azure_tenant_id="$tenantID"
+export TF_VAR_azure_subscription_id="$subID"
 ```
 
 The values can be provided directly as well (e.g. copying and pasting them into the script).

diff --git a/main.tf b/main.tf
@@ -1,6 +1,6 @@
 #
 # This file is part of Cisco Modeling Labs
-# Copyright (c) 2019-2024, Cisco Systems, Inc.
+# Copyright (c) 2019-2025, Cisco Systems, Inc.
 # All rights reserved.
 #
 
@@ -25,9 +25,11 @@ module "secrets" {
 }
 
 module "deploy" {
-  source = "./modules/deploy"
-  cfg    = local.cfg
-  extras = local.extras
+  source                = "./modules/deploy"
+  cfg                   = local.cfg
+  extras                = local.extras
+  azure_subscription_id = var.azure_subscription_id
+  azure_tenant_id       = var.azure_tenant_id
 }
 
 provider "cml2" {