Develop (EMCECS#372)

* Update ECS CE Docker image with ECS 3.0.0 Hotfix 2 (EMCECS#312) * Reorg base files (fix my glitch) * Fixed Dockerfile so it patches everything in-place. (cherry picked from commit 867189e) * update release files to use new image. * Duct tape for EMCECS#301 Update to ECS 3.0.0.2 (3.0.0 HF 2) (EMCECS#314) * fix a small typo * fix package install issue because EPEL is between versions again. * quick fixes for crashing HF2 to unblock clients * Docs update (EMCECS#313) * Adds FAQ page * installation troubleshooting * formatting * Addition of network troubleshooting * Addition of network troubleshooting * More Troubleshooting * sidebar implementation attempt * implements important links dropdown, disables page dropdown. * whoops * Adds migration page and some small updates. * Bunch of docs updates * bugfix-hf2 (EMCECS#315) * multitail and dstat are coming in handy right now * change fact cache location to /var/cache/emc/ecs-install * log the state of Docker at the end of bootstrapping for troubleshooting help (those hashes are good to see!) * [Ansible] Stop templating, start regex replacing props/confs * release prep 2.3.0 (EMCECS#316) * OVA prep (EMCECS#318) * Configure Jenkins pipeline to test installation process * Get repository information from Jenkins SCM config * Fix env var * Obtain TF options from Jenkins params. Moved deprovision step to post action * Allow multi-node configuration * Update checkout step in jenkinsfile * add zerofill.sh to /tools (EMCECS#324) (cherry picked from commit 58f7e8eea587c5a005eff3df2c2733ac2f5e1a9c) * Add slack notifications to jenkins pipeline * [WiP] Configure Jenkins server to build PR and provide feedback (EMCECS#328) Configure Jenkins server to build PR and provide feedback * Docs pass 2 (EMCECS#326) * Removal of deprecated procedures * Templates * OVA install guide added * Fixed broken links * preflight remove all bootstrap packages if installed (EMCECS#330) * Non-PR Jenkins jobs do not provide URL and commit author (EMCECS#331) Fix <null> variables in Slack notifications * put yum actions in retry loop with timeout (EMCECS#332) Put yum actions in retry loop with timeout * Implements EMCECS#205 Installer must have public key initial auth capability (EMCECS#270) * ECS-CommunityEdition-205 Installer must have public key initial auth capability (cherry picked from commit 6eea10b5db3985f960d7b313d2e705a0f913ba55) * More sausage for the initial ssh key auth (cherry picked from commit 8535ccb5430e89b79d253ea1e74390a39b8b20f3) * more sausage (cherry picked from commit edf961e0765cd9a06ccea1a6d1e2406816533f46) * deploy.yml change ideas (cherry picked from commit ef48e2cc57fa6d0a57aa30bc62ba816fc167aed9) * bootstrap.sh modifications (cherry picked from commit d0b3c630f0a2004fe23534aca7e4a95986dce383) * bootstrap.sh modifications (cherry picked from commit 86f897af9395a57af5b33c2162cd85422b4e6ded) * move generic help to generic_help.j2.yml file from config.yml * include shipit.lib.sh * build install paths early add copy action for ssh PKI material * fix a couple gitopt bugs * add create_install_tree() to plugin-defaults.sh * copy ssh keys in bootstrap.sh * more longopts adjustments * add loop delay in retry_with_timeout() * stop trying to autoremove curl, it'll always error. * key_vals need basename not full path set 0700 bits on ssh/ssl stores * remove optarg debugging * more ssh pubkey sausage * update reference.deploy.yml to include feature * jenkins changes * jenkins changes * jenkins changes * jenkins changes * jenkins changes * bump versions and move OVA download links. (EMCECS#335) * open-vm-tools now has a cross dependency (EMCECS#337) with open-vm-tools-desktop and yum fails to install open-vm-tools on remote nodes when open-vm-tools-desktop is not installed. * ECS-CommunityEdition-317 Make `ecsconfig ping -cx` loop when dtquery fails (EMCECS#344) (cherry picked from commit f5c7810) * Change the way ecs-install is pushed to repo (EMCECS#346) * invoke zerofill via bash rather than expecting exec bit (EMCECS#343) * Remove Ansible verbosity flag from Jenkinsfile (EMCECS#350) * OVA QoL Improvements (EMCECS#351) * add `ova-step1` and `ova-step2` macros * add `ecsdeploy noop` for some ova macros to look better * make videploy more intelligent and play nice with update_deploy * Implement Ansible global OVA flag fact (EMCECS#349) * implements Ansible global ova flag fact - custom fact in /etc/ansible/facts.d/ova.fact - ova conditional flags in playbooks * misaligned `when` * skip rebooting when using the OVA. * Upgrade Ansible to 2.3 (EMCECS#347) * install ansible package from @edge_main for 2.3 * Ansible changes for Ansible 2.3 * ECS-CommunityEdition-235 Bump Ansible version to 2.3 * refactor Ansible task `when:` clauses to Ansible 2.3 spec * refactor node reboot actions for Ansible 2.3 Also resolve EMCECS#342 * remove unused json_file plugin * must ignore_errors: True `needs-restarting -r` * refactor port-check `when`s to Ansible 2.3 spec * cleanup * add loop_control to path permissions entries * add loop_control labels to many iterators * add loop_control labels to many iterators * break out one directive per line * add loop_control labels to many iterators * incorrect `when` * speling * Switch to Alpine 3.6 release (EMCECS#359) * Switch to Alpine 3.6 release Install Python 2 from APK * Changes to Rockerfile for python:2-alpine parity * Split steps out from Ansible to get realtime console logging (EMCECS#358) * Split steps out from Ansible to get realtime console logging * use /tmp? * template out a script to run command on install node via IP * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * log environment info to file log only, never to console. * Add CentOS 7.4 support (EMCECS#360) (cherry picked from commit 1a96087) * [WiP] Misc. 2.5.0 bugfixes (EMCECS#352) * update reference deploy version * bugfix typo in ed25519 private key filename * fix ova flag implementation * fix ova flag implementation round 2 * fix ova flag implementation round 3 * update entrypoint.sh * bump version to 2.5.0b1 (EMCECS#364) * [WiP] Update ecs-install Python requirements (EMCECS#356) * update python requirements * pin python requirements to major versions rather than patches. * add python2-dev to temporary build environment * need cryptography>=1.9 * [WiP] ECS 3.1.0.0 Reduced GA and CE Support (EMCECS#353) * make 3.0.0.2 to use 100% regex (cherry picked from commit 0689ce5) * prep 3.1.0.0 RC3 (cherry picked from commit 2c3d5b9) * prep 3.1.0.0 RC3 (cherry picked from commit ea12c29) * ECS 3.1 templates * local facts must be fully qualified? * interface roles should be defined in deploy.yml * actually use a comma in the jinja joiner() func * use ansible_fqdn for agent strings not ansible_hostname * ECS 3.1.0.0 RC4 * joiner() needs to be the prefix not the suffix * the infamous missing comma * no trailing comma * remove redundant spaces * set host: field in testing * make object-main_network.json.j2 VDC-aware + formatting * Set georeceiver initialBufferNumOnHeap to 10 * Mount /usr instead of /usr/local to capture new install path * [WiP] ECS 3.1 Full GA and CE support (EMCECS#367) * Use nodeId instead of the node IP to create data store * Fix errors getting node ID * fix 3.1 patch again * migrate cm.object.properties/'MustHaveEnoughResources=false' into Dockerfile * Run cf_client in container for new low partition count vars * Run cf_client in container for new low partition count vars * migrate cf_client variable settings into Dockerfile * update comments in Dockerfile for 3.1.0.0 * release-2.5.0-prep (EMCECS#370) * Update ECS-Installation.md (cherry picked from commit f8be70f) * Update ECS-Installation.md (cherry picked from commit b479b07) * bump versions
nikhil-vr · Oct 31, 2017 · 09e1186 · 09e1186
1 parent b479b07
commit 09e1186
Show file tree

Hide file tree

Showing 62 changed files with 837 additions and 581 deletions.
diff --git a/Jenkinsfile b/Jenkinsfile
@@ -62,19 +62,54 @@ pipeline {
                 sh 'terraform output -json > output.json'
             }
         }
-        stage('Deploy ECS'){
+        stage('Setup install node'){
             steps {
-                  sh './tests/tf_to_hosts output.json hosts.ini'
+                  sh 'chmod +x ./tests/tf_to_hosts.py'
+                  sh 'chmod +x ./tests/tf_to_ssh.py'
+                  sh './tests/tf_to_hosts.py output.json hosts.ini'
+                  sh "./tests/tf_to_ssh.py output.json ./ssh.sh $SSH_USR"
+                  sh 'chmod +x ./ssh.sh'
+                  sh 'cat output.json'
+                  sh 'cat hosts.ini'
+                  sh 'cat ./ssh.sh'
                   ansiblePlaybook \
-                      playbook: 'tests/ansible/install_node.yml',
+                      playbook: 'tests/ansible/install_node_setup.yml',
                       inventory: 'hosts.ini',
                       extraVars: [
                           ansible_ssh_user: "$SSH_USR",
                           ansible_ssh_pass: "$SSH_PSW",
                           ansible_become_pass: "$SSH_PSW",
                           current_directory: "$WORKSPACE"
-                      ],
-                      extras: '-vvv'
+                      ]
+              }
+        }
+        stage('Bootstrap install node'){
+            steps {
+                  sh './ssh.sh curl http://10.1.83.5/registry.crt -o /tmp/registry.crt'
+                  sh './ssh.sh /root/ecs/bootstrap.sh -n -v --build-from http://10.1.83.5/alpine --vm-tools --proxy-cert /root/ecs/contrib/sslproxycert/emc_ssl.pem --proxy-endpoint 10.1.83.5:3128 -c /root/ecs/deploy.yml --centos-mirror 10.1.83.5 --registry-cert /tmp/registry.crt --registry-endpoint cache.gotham.local:5000 --override-dns 10.1.83.19'
+              }
+        }
+        stage('Reboot install node'){
+            steps {
+                  ansiblePlaybook \
+                      playbook: 'tests/ansible/install_node_reboot.yml',
+                      inventory: 'hosts.ini',
+                      extraVars: [
+                          ansible_ssh_user: "$SSH_USR",
+                          ansible_ssh_pass: "$SSH_PSW",
+                          ansible_become_pass: "$SSH_PSW",
+                          current_directory: "$WORKSPACE"
+                      ]
+              }
+        }
+        stage('Deploy ECS'){
+            steps {
+                  sh './ssh.sh step1'
+              }
+        }
+        stage('Configure ECS'){
+            steps {
+                  sh './ssh.sh step2'
               }
         }
     }

diff --git a/bootstrap.sh b/bootstrap.sh
@@ -251,7 +251,7 @@ while true; do
         export mirror_val="${2}"
         shift 2
         ;;
-    -o|--override-dhcp-dns)
+    -o|--override-dns)
         export dhcpdns_flag=true
         export dhcpdns_val="${2}"
         shift 2
@@ -781,7 +781,7 @@ if get_os_needs_restarting; then
 fi
 
 if ${zerofill_flag}; then
-    sudo "${INSTALL_ROOT}/tools/zerofill.sh"
+    sudo bash "${INSTALL_ROOT}/tools/zerofill.sh"
 fi
 
 

diff --git a/bootstrap_plugins/centos74.plugin.sh b/bootstrap_plugins/centos74.plugin.sh
@@ -0,0 +1,200 @@
+#@IgnoreInspection BashAddShebang
+
+# Copyright (c) 2015 EMC Corporation
+# All Rights Reserved
+#
+# This software contains the intellectual property of EMC Corporation
+# or is licensed to EMC Corporation from third parties.  Use of this
+# software and the intellectual property contained therein is expressly
+# limited to the terms and conditions of the License Agreement under which
+# it is provided by or on behalf of EMC.
+
+# OS Support library for CentOS 7.3
+
+os_supported=true
+
+# Docker binary
+docker_binary='/bin/docker'
+
+# packages to clean up during preflight
+# Don't `yum autoremove curl`.  Yum is a dependency and it will throw errors.
+list_preflight_packages="git nfs-client nfs-tools rsync wget ntp docker vim pigz gdisk aria2 htop iotop iftop multitail dstat jq python-docker-py dkms qemu-guest-agent open-vm-tools open-vm-tools-desktop docker"
+
+# Do any OS-specific tasks that must be done prior to bootstrap
+do_preflight() {
+    rm_repo_pkg "$list_preflight_packages"
+}
+
+# packages to install before others
+list_prefix_packages='wget curl epel-release yum-utils'
+
+# script to run for installing prefix_packages
+in_prefix_packages() {
+    in_repo_pkg "$list_prefix_packages"
+}
+
+# packages to install
+# list_general_packages='yum-utils git python-pip python-docker-py'
+list_general_packages='git ntp docker vim rsync pigz gdisk aria2'
+
+# script to run for installing general_packages
+in_general_packages() {
+    in_repo_pkg "$list_general_packages"
+#    if ! docker version; then
+#        curl -fsSL https://get.docker.com/ | sudo sh
+#    fi
+    sudo systemctl enable docker
+    sudo systemctl start docker
+    sudo usermod -aG docker $(whoami)
+}
+
+# packages to install after others
+list_suffix_packages='htop iotop iftop multitail dstat jq python-docker-py'
+# list_suffix_packages='htop jq pigz gdisk aria2 python-docker-py'
+
+# script to run for installing suffix_packages
+in_suffix_packages() {
+    in_repo_pkg "$list_suffix_packages"
+
+    # Install Rocker
+    curl -fsSL ${rocker_artifact_url} \
+    | sudo tar -xzC /usr/local/bin && sudo chmod +x /usr/local/bin/rocker
+}
+
+# packages to install if a VM
+list_vm_packages='dkms qemu-guest-agent open-vm-tools open-vm-tools-desktop'
+
+# command to run for installing vm_packages
+in_vm_packages() {
+    in_repo_pkg "$list_vm_packages"
+    # return 0
+}
+
+# command to install one or more os package manager package
+in_repo_pkg() {
+    retry_with_timeout 10 300 sudo yum -y install $*
+}
+
+rm_repo_pkg() {
+    retry_with_timeout 10 300 sudo yum -y autoremove $*
+}
+
+# command to update all packages in the os package manager
+up_repo_pkg_all() {
+    retry_with_timeout 10 300 sudo yum -y update
+}
+
+# command to rebuild the os package manager's database
+up_repo_db() {
+    retry_with_timeout 10 300 sudo yum -y makecache
+}
+
+# command to set os package manager proxy
+set_repo_proxy_conf() {
+    sudo sed -i -e '/^proxy=/d' /etc/yum.conf
+    echo "proxy=${http_proxy}" \
+        | append /etc/yum.conf
+}
+
+# command to set os package manager to keep its cache
+set_repo_keepcache_conf() {
+    sudo sed -i -e '/^keepcache=/d' /etc/yum.conf
+    echo "keepcache=1" \
+        | append /etc/yum.conf
+}
+
+# idempotent config script to fixup repos to properly use proxycaches
+set_repo_cacheable_idempotent() {
+    sudo sed -i -e 's/^#baseurl=/baseurl=/' /etc/yum.repos.d/*
+    sudo sed -i -e 's/^mirrorlist=/#mirrorlist=/' /etc/yum.repos.d/*
+}
+
+set_repo_mirror_idempotent() {
+    # sudo sed -i -e "s#http:///centos#http://${mirror_val}/centos#g" /etc/yum.repos.d/*
+    sudo sed -i -e "s#http://.*/centos#http://${mirror_val}/centos#g" /etc/yum.repos.d/*
+}
+
+# command to set the proxy for the whole OS
+set_os_proxy() {
+    sudo sed -i -e '/_proxy/d' /etc/environment
+    echo -n "http_proxy=${http_proxy}\nhttps_proxy=${http_proxy}\nftp_proxy=${http_proxy}\n" \
+        | append /etc/environment
+    if $mirror_flag; then
+        echo -n "no_proxy=${mirror_val}\n" | append /etc/environment
+    fi
+}
+
+# command to determine if the OS needs restarting after package updates
+get_os_needs_restarting() {
+    if ! [ -z "$(sudo /usr/bin/needs-restarting)" ]; then
+        return 0
+    else
+        return 1
+    fi
+}
+
+# command to reboot the system
+do_reboot() {
+    sudo reboot
+}
+
+# Command to configure docker's proxy under centos flavored systemd
+set_docker_proxy() {
+    local tmpconf="/etc/systemd/system/docker.service.d/http-proxy.conf"
+    if ! [ -d "$(dirname $tmpconf)" ]; then
+        sudo mkdir "$(dirname $tmpconf)"
+    fi
+    log "sed error is OK here if the proxy config file does not yet exist."
+    sudo sed -i -e '/HTTP_PROXY/d' "$tmpconf"
+    echo "Environment=\"HTTP_PROXY=${http_proxy}\" \"NO_PROXY=localhost,127.0.0.1,$(hostname),$(hostname -f)\"" \
+        | append "$tmpconf"
+    sudo systemctl daemon-reload
+    sudo systemctl restart docker
+    sudo systemctl status docker
+}
+
+# command to add mitm cert to docker trust store
+set_docker_reg_cert() {
+    local registry="${1}"
+    local cert="${2}"
+    if ! [ -d "/etc/docker/certs.d/${registry}" ]; then
+        sudo mkdir -p "/etc/docker/certs.d/${registry}"
+        sudo cp "${cert}" "/etc/docker/certs.d/${registry}/ca.crt"
+    else
+        if [ -f "/etc/docker/certs.d/${registry}/ca.crt" ]; then
+            echo "Reusing existing /etc/docker/certs.d/${registry}/ca.crt"
+        else
+            sudo cp "${cert}" "/etc/docker/certs.d/${registry}/ca.crt"
+        fi
+    fi
+    set_mitm_cert "${cert}"
+    sudo systemctl daemon-reload
+    sudo systemctl restart docker
+    sudo systemctl status docker
+}
+
+# command to add mitm cert to local trust store
+set_mitm_cert() {
+    sudo cp "${1}" "/etc/pki/ca-trust/source/anchors/$(basename ${1}).crt"
+    sudo update-ca-trust extract
+}
+
+do_post_install() {
+    # Disable postfix since we don't need an MTA
+    sudo systemctl disable --now postfix
+}
+
+override_dhcp_dns() {
+    nameserver_list="${1}"
+    sudo sed -i -e 's/PEERDNS="yes"/PEERDNS="no"/' /etc/sysconfig/network-scripts/ifcfg-*
+    sudo sed -i -e '/DNS[0-9]=/d' /etc/sysconfig/network-scripts/ifcfg-*
+    sudo sed -i -e '/nameserver/d' /etc/resolv.conf
+    nsnumber=1
+    for nameserver in $(echo ${nameserver_list} | tr ',' ' '); do
+        echo "nameserver ${nameserver}" | append /etc/resolv.conf
+        for script in /etc/sysconfig/network-scripts/ifcfg-*; do
+            echo "DNS${nsnumber}=${nameserver}" | append "${script}"
+        done
+        nsnumber=$((nsnumber++))
+    done
+}
diff --git a/bootstrap_plugins/os-router.plugin.sh b/bootstrap_plugins/os-router.plugin.sh
@@ -61,10 +61,12 @@ route_os() {
         centos\ linux\ release\ 7.2*)
             source ${plugins}/centos72.plugin.sh
             ;;
-
         centos\ linux\ release\ 7.3*)
             source ${plugins}/centos73.plugin.sh
             ;;
+        centos\ linux\ release\ 7.4*)
+            source ${plugins}/centos74.plugin.sh
+            ;;
 
 #        dockerized\ centos\ linux\ release\ 7.2*)
 #            source ${plugins}/centos72-docker.plugin.sh

diff --git a/docs/design/reference.deploy.yml b/docs/design/reference.deploy.yml
@@ -1,4 +1,4 @@
-# deploy.yml reference implementation v2.2.0
+# deploy.yml reference implementation v2.5.0
 
 # [Optional]
 # By changing the license_accepted boolean value to "true" you are

diff --git a/patches/3.0.0.2/Dockerfile b/patches/3.0.0.2/Dockerfile
@@ -1,4 +1,4 @@
-# Fixes to the default 3.0 HF2 image.
+# Fixes to the default 3.0 HF2 reduced image.
 FROM emcvipr/object:3.0.0.0-86889.0a0ee19-reduced
 
 # Increase memory for transformsvc
@@ -7,7 +7,8 @@ RUN sed -i s/-Xmx128m/-Xmx512m/ /opt/storageos/bin/transformsvc
 # Fix disk partitioning script
 RUN sed -i '/VMware/ s/$/ \&\& [ ! -e \/data\/is_community_edition ]/' /opt/storageos/bin/storageserver-partition-config.sh
 
-COPY vnest-common-conf-template.xml /opt/storageos/conf/vnest-common-conf-template.xml
+# COPY vnest-common-conf-template.xml /opt/storageos/conf/vnest-common-conf-template.xml
+RUN f=/opt/storageos/conf/vnest-common-conf-template.xml; grep -q "object.UseSeparateThreadPools" $f || sed -i '/properties id="serviceProperties"/a \ \ \ \ \ \ \ \ <prop key="object.UseSeparateThreadPools">true</prop>' $f
 
 # Make vnest use separate thread pools to prevent deadlock
 RUN printf "\n# Use separate thread pools to prevent deadlock in vnest init\nobject.UseSeparateThreadPools=true\n" >> /opt/storageos/conf/vnest.object.properties

diff --git a/patches/3.1.0.0/Dockerfile b/patches/3.1.0.0/Dockerfile
@@ -1,14 +1,23 @@
-# Fixes to the default 3.0HF image.
-FROM emcvipr/object:3.1.0.0-93256.00e3410-reduced
+# Fixes to the default 3.1.0.0 reduced image.
+
+# Build on RC4 object image (GA release)
+FROM emcvipr/object:3.1.0.0-95266.ab2753a-reduced
 
 # Increase memory for transformsvc
-ADD transformsvc /opt/storageos/bin/
+RUN sed -i s/-Xmx128m/-Xmx512m/ /opt/storageos/bin/transformsvc
 
 # Fix disk partitioning script
-ADD storageserver-partition-config.sh /opt/storageos/bin/
+RUN sed -i '/VMware/ s/$/ \&\& [ ! -e \/data\/is_community_edition ]/' /opt/storageos/bin/storageserver-partition-config.sh
 RUN /usr/bin/chmod +x /opt/storageos/bin/storageserver-partition-config.sh
 
-# Make vnest use separate thread pools to prevent deadlock
-ADD vnest.object.properties /opt/storageos/conf/
-
+# Set VNets useSeperateThreadPools to True
 RUN f=/opt/storageos/conf/vnest-common-conf-template.xml; grep -q "object.UseSeparateThreadPools" $f || sed -i '/properties id="serviceProperties"/a \ \ \ \ \ \ \ \ <prop key="object.UseSeparateThreadPools">true</prop>' $f
+
+# Set georeceiver's initialBufferNumOnHeap to something smaller for CE
+RUN f=/opt/storageos/conf/georeceiver-conf.xml; grep -q 'name="initialBufferNumOnHeap" value="10"' $f ||  sed -i 's/name="initialBufferNumOnHeap" value="60"/name="initialBufferNumOnHeap" value="10"/' $f
+
+# Configure CM Object properties: Disable minimum storage device count
+RUN f=/opt/storageos/conf/cm.object.properties; grep -q 'MustHaveEnoughResources=false' $f ||  sed -i 's/MustHaveEnoughResources=true/MustHaveEnoughResources=false/' $f
+
+# Allow allocation of different blocks of a chunk to be stored on the same partition
+RUN sed -i 's#<config:boolean name="allowAllocationOnIgnoredPartitions" value="false" description="If set to true, different blocks in one chunk may be allocated on the same partition"/>#<config:boolean name="allowAllocationOnIgnoredPartitions" value="true" description="If set to true, different blocks in one chunk may be allocated on the same partition"/>#g' /opt/storageos/conf/ssm-cf-conf.xml