This repository has been archived by the owner on Jul 10, 2021. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 404
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs(contributing): Updating AWS EKS install docs (#1797)
- Loading branch information
Showing
1 changed file
with
226 additions
and
110 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,173 +1,289 @@ | ||
| --- | ||
| layout: single | ||
| title: "Set up a K8s v2 provider for Amazon EKS" | ||
| title: "Set up a Kubernetes v2 provider for Amazon EKS" | ||
| sidebar: | ||
| nav: setup | ||
| --- | ||
|
|
||
| {% include toc %} | ||
|
|
||
| > Before you proceed further with this setup, we strongly recommend that you familiarize yourself with [Amazon EKS concepts](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html). | ||
| Also, please visit the [AWS global infrastructure region table](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/) for the most up-to-date information on Amazon EKS regional availability. | ||
| Also, visit the [AWS global infrastructure region table](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/) for the most up-to-date information on Amazon EKS regional availability. | ||
|
|
||
| These instructions assume that you have AWS CLI [installed](https://docs.aws.amazon.com/cli/latest/userguide/installing.html), | ||
| [configured](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html), and have access to each of the managed account and managing account. | ||
| These instructions assume that you have AWS CLI [installed](https://docs.aws.amazon.com/cli/latest/userguide/installing.html) and [configured](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) on an Ubuntu machine running on AWS EC2. | ||
|
|
||
| ## Set up the managing account | ||
| ## Preparing to install Spinnaker on EKS | ||
|
|
||
| In the managing account, create a two-subnet VPC, IAM roles, instance profiles, and a Security Group for EKS control-plane communications and an EKS cluster. | ||
| The following steps describes how to the tools you need to install and manage Spinnaker and EKS. | ||
|
|
||
| > This step will take around 15-20 minutes to complete | ||
| ```bash | ||
| curl -O https://d3079gxvs8ayeg.cloudfront.net/templates/managing.yaml | ||
| aws cloudformation deploy --stack-name spinnaker-managing-infrastructure-setup --template-file managing.yaml \ | ||
| --parameter-overrides UseAccessKeyForAuthentication=false EksClusterName=spinnaker-cluster --capabilities CAPABILITY_NAMED_IAM | ||
| ### 1. Install and configure kubectl | ||
|
|
||
| Install `kubectl` to manage Kubernetes and `aws-iam-authenticator` to manage cluster authentication: | ||
|
|
||
| ``` | ||
| # Download and install kubectl | ||
| `curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl | ||
| chmod +x ./kubectl | ||
| sudo mv ./kubectl /usr/local/bin/kubectl | ||
| # Verify the installation of kubectl | ||
| kubectl help | ||
| # Download and install aws-iam-authenticator | ||
| curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.13.7/2019-06-11/bin/linux/amd64/aws-iam-authenticator | ||
| chmod +x ./aws-iam-authenticator | ||
| mkdir -p $HOME/bin && cp ./aws-iam-authenticator $HOME/bin/aws-iam-authenticator && export PATH=$HOME/bin:$PATH | ||
| echo 'export PATH=$HOME/bin:$PATH' >> ~/.bashrc | ||
| #Verify the installation of aws-iam-authenticator | ||
| aws-iam-authenticator help` | ||
| ``` | ||
|
|
||
| After the stack creation succeeds, run the following: | ||
| The commands return the help information for `kubectl` and `aws-iam-authenticator` respectively. If the help for either tool does not get returned, verify that you have installed the tool. | ||
|
|
||
| ### 2. Install awscli | ||
|
|
||
| ``` | ||
| # Install the awscli | ||
| sudo apt install python-pip awscli | ||
| # Verify the installation | ||
| aws --version | ||
| ``` | ||
|
|
||
| The command returns the `awscli` version. | ||
|
|
||
| ### 3. Install eksctl | ||
|
|
||
| Install `eksctl` to manage EKS clusters from the command line: | ||
|
|
||
| ```bash | ||
| VPC_ID=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`VpcId`].OutputValue' --output text) | ||
| CONTROL_PLANE_SG=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`SecurityGroups`].OutputValue' --output text) | ||
| AUTH_ARN=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`AuthArn`].OutputValue' --output text) | ||
| SUBNETS=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`SubnetIds`].OutputValue' --output text) | ||
| MANAGING_ACCOUNT_ID=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`ManagingAccountId`].OutputValue' --output text) | ||
| EKS_CLUSTER_ENDPOINT=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`EksClusterEndpoint`].OutputValue' --output text) | ||
| EKS_CLUSTER_NAME=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`EksClusterName`].OutputValue' --output text) | ||
| EKS_CLUSTER_CA_DATA=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`EksClusterCA`].OutputValue' --output text) | ||
| SPINNAKER_INSTANCE_PROFILE_ARN=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`SpinnakerInstanceProfileArn`].OutputValue' --output text) | ||
| ``` | ||
| # Download and configure eksctl | ||
| curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp | ||
| ## Set up the managed account | ||
| sudo mv /tmp/eksctl /usr/local/bin | ||
| In each of managed accounts, create a IAM role that can be assumed by Spinnaker: | ||
| # Verify the installation | ||
| eksctl help | ||
| ``` | ||
|
|
||
| The command returns the help for `eksctl`. | ||
|
|
||
| > This needs to be executed in managing account as well. | ||
| ### 4. Install Halyard | ||
|
|
||
| ```bash | ||
| curl -O https://d3079gxvs8ayeg.cloudfront.net/templates/managed.yaml | ||
| Install Halyard, which is used to install and manage Spinnaker: | ||
|
|
||
| aws cloudformation deploy --stack-name spinnaker-managed-infrastructure-setup --template-file managed.yaml \ | ||
| --parameter-overrides AuthArn=$AUTH_ARN ManagingAccountId=$MANAGING_ACCOUNT_ID --capabilities CAPABILITY_NAMED_IAM | ||
| ``` | ||
| # Download and configure Halyard | ||
| curl -O https://raw.githubusercontent.com/spinnaker/halyard/master/install/debian/InstallHalyard.sh | ||
| ## `kubectl` and `heptio authenticator` configurations | ||
| sudo useradd halyard | ||
| 1. Install and configure [kubectl and aws-iam-authenticator](https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html) on the workstation/instance where you are running Halyard from. Halyard version must be >=1.5.0. | ||
| sudo bash InstallHalyard.sh | ||
| Also, when an Amazon EKS cluster is created, the IAM entity (user or role) that creates the cluster is added to the Kubernetes RBAC authorization table as the administrator. Initially, only that IAM user can make calls to the Kubernetes API server using `kubectl`. | ||
| sudo update-halyard | ||
| If you use the console to create the cluster, you must ensure that the same IAM user credentials are in the AWS SDK credential chain when you are running `kubectl` commands on your cluster. | ||
| # Verify the installation | ||
| hal -v | ||
| ``` | ||
|
|
||
| In the setup as done above, we used AWS CLI, hence you must ensure that the server/workstation from where you are running the `kubectl` commands in step-2 below have the same AWS credentials. | ||
| The command returns the Halyard version. | ||
|
|
||
| {:start="2"} | ||
| ### 5. Create the Amazon EKS cluster for Spinnaker | ||
|
|
||
| 2. Create default [kubectl configuration file](https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html) | ||
| ``` | ||
| eksctl create cluster --name=eks-spinnaker --nodes=2 --region=us-west-2 --write-kubeconfig=false | ||
| ``` | ||
|
|
||
| Paste the following to your `kubeconfig` file, replace `<endpoint-url>`, `<base64-encoded-ca-cert>` and `<cluster-name>` with values of `$EKS_CLUSTER_ENDPOINT`, `$EKS_CLUSTER_CA_DATA` and `$EKS_CLUSTER_NAME` | ||
| as noted above: | ||
| ## Install and configure Spinnaker | ||
|
|
||
| ```yaml | ||
| This section walks you through the process of installing and configuring Spinnaker for use with Amazon EKS. | ||
|
|
||
| apiVersion: v1 | ||
| clusters: | ||
| - cluster: | ||
| server: <endpoint-url> | ||
| certificate-authority-data: <base64-encoded-ca-cert> | ||
| name: kubernetes | ||
| contexts: | ||
| - context: | ||
| cluster: kubernetes | ||
| user: aws | ||
| name: aws | ||
| current-context: aws | ||
| kind: Config | ||
| preferences: {} | ||
| users: | ||
| - name: aws | ||
| user: | ||
| exec: | ||
| apiVersion: client.authentication.k8s.io/v1alpha1 | ||
| command: aws-iam-authenticator | ||
| args: | ||
| - "token" | ||
| - "-i" | ||
| - "<cluster-name>" | ||
| # - "-r" | ||
| # - "<role-arn>" | ||
| # env: | ||
| # - name: AWS_PROFILE | ||
| # value: "<aws-profile>" | ||
| ### 1. Retrieve Amazon EKS cluster kubectl contexts | ||
|
|
||
| ``` | ||
| aws eks update-kubeconfig --name eks-spinnaker --region us-west-2 --alias eks-spinnaker | ||
| ``` | ||
|
|
||
| (Optional) To have the Heptio authenticator assume a role to perform cluster operations (instead of the default AWS credential provider chain), uncomment the `-r` and `<role-arn>` lines and substitute an IAM role ARN to use with your user. | ||
| ### 2. Check Halyard version | ||
|
|
||
| (Optional) To have the Heptio authenticator always use a specific named AWS credential profile (instead of the default AWS credential provider chain), uncomment the env lines and substitute `<aws-profile>` with the profile name to use. | ||
| More recent versions of Spinnaker require a more recent version of Halyard. For example, Spinnaker 1.19.x requires Halyard 1.32.0 or later. | ||
|
|
||
| {:start="3"} | ||
| Verify your Halyard version: | ||
|
|
||
| 3. [Create the necessary service accounts and cluster role bindings](/setup/install/providers/kubernetes-v2/#optional-create-a-kubernetes-service-account) | ||
| ``` | ||
| hal -v | ||
| ``` | ||
|
|
||
| ### 3. Add and configure Kubernetes accounts | ||
|
|
||
| ## Enable Kubernetes Cloud provider using Halyard | ||
| Enable the Kubernetes provider for Spinnaker: | ||
|
|
||
| ```bash | ||
| ``` | ||
| # Enable the Kubernetes provider | ||
| hal config provider kubernetes enable | ||
| hal config provider kubernetes account add ${MY_K8_ACCOUNT} --provider-version v2 --context $(kubectl config current-context) | ||
| # Set the current kubectl context to the cluster for Spinnaker | ||
| kubectl config use-context eks-spinnaker | ||
| ``` | ||
|
|
||
| A context element in a kubeconfig file is used to group access parameters under a convenient name. Each context has three parameters: cluster, namespace, and user. By default, `kubectl` uses parameters from the current context to communicate with the cluster. | ||
|
|
||
| ``` | ||
| # Assign the Kubernetes context to CONTEXT | ||
| CONTEXT=$(kubectl config current-context) | ||
| ``` | ||
|
|
||
| Next, create a service account for the Amazon EKS cluster: | ||
|
|
||
| ``` | ||
| kubectl apply --context $CONTEXT -f https://spinnaker.io/downloads/kubernetes/service-account.yml | ||
| ``` | ||
|
|
||
| See the [Kubernetes documentation for more details on service accounts](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/). | ||
|
|
||
| Extract the secret token of the `spinnaker-service-account`: | ||
|
|
||
| ``` | ||
| TOKEN=$(kubectl get secret --context $CONTEXT \ | ||
| $(kubectl get serviceaccount spinnaker-service-account \ | ||
| --context $CONTEXT \ | ||
| -n spinnaker \ | ||
| -o jsonpath='{.secrets[0].name}') \ | ||
| -n spinnaker \ | ||
| -o jsonpath='{.data.token}' | base64 --decode) | ||
| ``` | ||
|
|
||
| Set the user entry in `kubeconfig`: | ||
|
|
||
| ``` | ||
| kubectl config set-credentials ${CONTEXT}-token-user --token $TOKEN | ||
| kubectl config set-context $CONTEXT --user ${CONTEXT}-token-user | ||
| ``` | ||
|
|
||
| Add `eks-spinnaker` cluster as a Kubernetes provider: | ||
|
|
||
| ``` | ||
| hal config provider kubernetes account add eks-spinnaker --provider-version v2 --context $CONTEXT | ||
| ``` | ||
|
|
||
| ### 4. Enable artifact support | ||
|
|
||
| ``` | ||
| hal config features edit --artifacts true | ||
| ``` | ||
|
|
||
| ### 5. Configure Spinnaker to install in Kubernetes | ||
|
|
||
| For our environment, we will use a distributed Spinnaker installation onto the Kubernetes cluster. This installation model has Halyard deploy each of the Spinnaker microservices separately. A distributed installation helps to limit update-related downtime. | ||
|
|
||
| ``` | ||
| hal config deploy edit --type distributed --account-name eks-spinnaker | ||
| ``` | ||
|
|
||
| ### 6. Configure Spinnaker to use AWS S3 | ||
|
|
||
| Finally, enable [artifact support](/reference/artifacts-with-artifactsrewrite//#enabling-artifact-support). | ||
| You will need your AWS account access key and secret access key. | ||
|
|
||
| ## Launch and Configure Amazon EKS Worker Nodes | ||
| ``` | ||
| export`` YOUR_ACCESS_KEY_ID``=<``access``-``key``>` | ||
| `hal config storage s3 edit ``--``access``-``key``-``id $YOUR_ACCESS_KEY_ID \` | ||
| ` ``--``secret``-``access``-``key ``--``region us``-``west``-``2 | ||
| ``` | ||
|
|
||
| Worker nodes launched using the below commands are standard Amazon EC2 instances and use [EKS optimized AMIs](https://docs.aws.amazon.com/eks/latest/userguide/worker.html). | ||
| Enter your AWS account secret access key at the prompt. | ||
|
|
||
| ```bash | ||
| curl -O https://d3079gxvs8ayeg.cloudfront.net/templates/amazon-eks-nodegroup.yaml | ||
| aws cloudformation deploy --stack-name spinnaker-eks-nodes --template-file amazon-eks-nodegroup.yaml \ | ||
| --parameter-overrides NodeInstanceProfile=$SPINNAKER_INSTANCE_PROFILE_ARN \ | ||
| NodeInstanceType=t2.large ClusterName=$EKS_CLUSTER_NAME NodeGroupName=spinnaker-cluster-nodes ClusterControlPlaneSecurityGroup=$CONTROL_PLANE_SG \ | ||
| Subnets=$SUBNETS VpcId=$VPC_ID --capabilities CAPABILITY_NAMED_IAM | ||
| Then, set the storage source to S3: | ||
|
|
||
| ``` | ||
| hal config storage edit --type s3 | ||
| ``` | ||
|
|
||
| ## Join the nodes with the Spinnaker EKS cluster | ||
| ### 7. Choose the Spinnaker version | ||
|
|
||
| Replace `<spinnaker-role-arn>` with `$AUTH_ARN` and save it as `aws-auth-cm.yaml` | ||
| To identify the latest version of Spinnaker to install, run the following command to get a list of available versions: | ||
|
|
||
| {% raw %} | ||
| ```yaml | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: aws-auth | ||
| namespace: kube-system | ||
| data: | ||
| mapRoles: | | ||
| - rolearn: <spinnaker-role-arn> | ||
| username: system:node:{{EC2PrivateDNSName}} | ||
| groups: | ||
| - system:bootstrappers | ||
| - system:nodes | ||
| ``` | ||
| hal version list | ||
| ``` | ||
|
|
||
| At the time of writing, 1.19.2 is the latest Spinnaker version. Configure Halyard to deploy Spinnaker 1.19.2: | ||
|
|
||
| ``` | ||
| export VERSION=1.19.2 | ||
| hal config version edit --version $VERSION | ||
| ``` | ||
| {% endraw %} | ||
|
|
||
| Join the nodes with the cluster: | ||
| Now, we are finally ready to install Spinnaker on the `eks-spinnaker` Amazon EKS cluster: | ||
|
|
||
| ```bash | ||
| kubectl apply -f aws-auth-cm.yaml | ||
| ``` | ||
| hal deploy apply | ||
| ``` | ||
|
|
||
| Watch the status of your nodes and wait for them to reach the `Ready` status: | ||
| ### 8. Verify the Spinnaker installation | ||
|
|
||
| ```bash | ||
| kubectl get nodes --watch | ||
| ``` | ||
| kubectl -n spinnaker get svc | ||
| ``` | ||
|
|
||
| The command returns the Spinnaker services that are in the `spinnaker` namespace. | ||
|
|
||
| ### 9. Expose Spinnaker using Elastic Load Balancer | ||
|
|
||
| Expose the Spinnaker API (Gate) and the Spinnaker UI (Deck) using Load Balancers by running the following commands to create the `spin-gate-public` and `spin-deck-public services`: | ||
|
|
||
| ``` | ||
| export NAMESPACE=spinnaker | ||
| # Expose Gate and Deck | ||
| kubectl -n ${NAMESPACE} expose service spin-gate --type LoadBalancer \ | ||
| --port 80 --target-port 8084 --name spin-gate-public | ||
| kubectl -n ${NAMESPACE} expose service spin-deck --type LoadBalancer \ | ||
| --port 80 --target-port 9000 --name spin-deck-public | ||
| export API_URL=$(kubectl -n $NAMESPACE get svc spin-gate-public \ | ||
| -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') | ||
| export UI_URL=$(kubectl -n $NAMESPACE get svc spin-deck-public \ | ||
| -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') | ||
| # Configure the URL for Gate | ||
| hal config security api edit --override-base-url http://${API_URL} | ||
| # Configure the URL for Deck | ||
| hal config security ui edit --override-base-url http://${UI_URL} | ||
| # Apply your changes to Spinnaker | ||
| hal deploy apply | ||
| ``` | ||
|
|
||
| It can take several moments for Spinnaker to restart. | ||
|
|
||
| You can verify that the Spinnaker Pods have restarted and check their status: | ||
|
|
||
| ``` | ||
| kubectl -n spinnaker get pods | ||
| ``` | ||
|
|
||
| ### 10. Re-verify the Spinnaker installation | ||
|
|
||
| Run the following command to verify that the Spinnaker services are present in the cluster: | ||
|
|
||
| ``` | ||
| kubectl -n spinnaker get svc | ||
| ``` | ||
|
|
||
| ### 11. Log in to Spinnaker console | ||
|
|
||
| Get the URL to Deck, the UI. | ||
|
|
||
| ``` | ||
| kubectl -n $NAMESPACE get svc spin-deck-public -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' | ||
| ``` | ||
|
|
||
| ## Next steps | ||
| Navigate to the URL in a supported browser and log in. | ||
|
|
||
| Optionally, you can [set up another cloud provider](/setup/install/providers/), but otherwise you’re ready to [choose an environment](/setup/install/environment/) in which to install Spinnaker. | ||
|
|