Release 2.3.0 (#319)

* Update ECS CE Docker image with ECS 3.0.0 Hotfix 2 (#312) * Reorg base files (fix my glitch) * Fixed Dockerfile so it patches everything in-place. (cherry picked from commit 867189e) * update release files to use new image. * Duct tape for #301 Update to ECS 3.0.0.2 (3.0.0 HF 2) (#314) * fix a small typo * fix package install issue because EPEL is between versions again. * quick fixes for crashing HF2 to unblock clients * Docs update (#313) * Adds FAQ page * installation troubleshooting * formatting * Addition of network troubleshooting * Addition of network troubleshooting * More Troubleshooting * sidebar implementation attempt * implements important links dropdown, disables page dropdown. * whoops * Adds migration page and some small updates. * Bunch of docs updates * bugfix-hf2 (#315) * multitail and dstat are coming in handy right now * change fact cache location to /var/cache/emc/ecs-install * log the state of Docker at the end of bootstrapping for troubleshooting help (those hashes are good to see!) * [Ansible] Stop templating, start regex replacing props/confs * release prep 2.3.0 (#316) * OVA prep (#318)
EMCECS · Jul 21, 2017 · 6b98d17 · 6b98d17
1 parent 25ec584
commit 6b98d17
Show file tree

Hide file tree

Showing 36 changed files with 1,051 additions and 802 deletions.
diff --git a/README.md b/README.md
@@ -102,8 +102,7 @@ The OVA is shipped as a bootstrapped Install Node.  It must be cloned multiple t
 
 ##### OVA Download Links
 
-* [dellemc-ecsce-3.0.0.2-install-node-2.2.2-vm0.ova](http://130852476153187606.public.ecstestdrive.com/public/dellemc-ecsce-3.0.0.2-install-node-2.2.2-vm0.ova)
-* [dellemc-ecsce-3.0.0.2-install-node-2.2.2-vm0.ova.xz](http://130852476153187606.public.ecstestdrive.com/public/dellemc-ecsce-3.0.0.2-install-node-2.2.2-vm0.ova.xz)
+* [dellemc-ecsce-3.0.0.2-install-node-2.3.0-vm0.ova](http://130852476153187606.public.ecstestdrive.com/public/dellemc-ecsce-3.0.0.2-install-node-2.3.0-vm0.ova)
 
 #### [ECS Single-Node Deployment with Install Node (recommended)](docs/source/installation/ECS-Installation.md)
 Using an Install Node for isolated environments, deploy a stand-alone instance of ECS to a single hardware or virtual machine.

diff --git a/README.rst b/README.rst
@@ -144,8 +144,7 @@ If you prefer to download a prefab Install Node as an OVF/OVA, follow
 one of the links below. Please note that OVAs are produced upon each
 release and do not necessarily have the most current software.
 
--  `dellemc-ecsce-2.0.2-beta.ova <http://130852476153187606.public.ecstestdrive.com/public/dellemc-ecsce-2.0.2-beta.ova>`__
--  `dellemc-ecsce-2.0.2-beta.ova.xz <http://130852476153187606.public.ecstestdrive.com/public/dellemc-ecsce-2.0.2-beta.ova.xz>`__
+-  `dellemc-ecsce-3.0.0.2-install-node-2.3.0-vm0.ova <http://130852476153187606.public.ecstestdrive.com/public/dellemc-ecsce-3.0.0.2-install-node-2.3.0-vm0.ova>`__
 
 `ECS Multi-Node Deployment with Install Node (recommended, most reusable, full-featured) <http://ecsce.readthedocs.io/en/latest/installation/ECS-Installation.html>`__
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

diff --git a/bootstrap.sh b/bootstrap.sh
@@ -584,6 +584,13 @@ else
 fi
 ping_sudo
 
+### Log Docker Inventory
+v "Logging Docker Inventory"
+p Logging Docker Inventory
+sudo docker images 2>&1 | log
+sudo docker ps -a 2>&1 | log
+ping_sudo
+
 ### Next steps
 p ''
 q 'All done bootstrapping your install node.'

diff --git a/bootstrap_plugins/centos72.plugin.sh b/bootstrap_plugins/centos72.plugin.sh
@@ -35,7 +35,7 @@ in_prefix_packages() {
 
 # packages to install
 # list_general_packages='yum-utils git python-pip python-docker-py'
-list_general_packages='git ntp docker'
+list_general_packages='git ntp docker vim rsync pigz gdisk aria2'
 
 # script to run for installing general_packages
 in_general_packages() {
@@ -49,7 +49,7 @@ in_general_packages() {
 }
 
 # packages to install after others
-list_suffix_packages='vim htop iotop iftop jq rsync pigz gdisk aria2 python-docker-py'
+list_suffix_packages='htop iotop iftop multitail dstat jq python-docker-py'
 # list_suffix_packages='htop jq pigz gdisk aria2 python-docker-py'
 
 # script to run for installing suffix_packages

diff --git a/bootstrap_plugins/centos73.plugin.sh b/bootstrap_plugins/centos73.plugin.sh
@@ -34,7 +34,7 @@ in_prefix_packages() {
 
 # packages to install
 # list_general_packages='yum-utils git python-pip python-docker-py'
-list_general_packages='git ntp docker'
+list_general_packages='git ntp docker vim rsync pigz gdisk aria2'
 
 # script to run for installing general_packages
 in_general_packages() {
@@ -48,7 +48,7 @@ in_general_packages() {
 }
 
 # packages to install after others
-list_suffix_packages='vim htop iotop iftop jq rsync pigz gdisk aria2 python-docker-py'
+list_suffix_packages='htop iotop iftop multitail dstat jq python-docker-py'
 # list_suffix_packages='htop jq pigz gdisk aria2 python-docker-py'
 
 # script to run for installing suffix_packages

diff --git a/docs/requirements.txt b/docs/requirements.txt
@@ -0,0 +1 @@
+sphinxcontrib-fulltoc
diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -30,7 +30,7 @@
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = []
+extensions = ['sphinxcontrib.fulltoc']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -89,13 +89,19 @@
 # further.  For a list of options available for each theme, see the
 # documentation.
 #
-#html_theme_options = {
+html_theme_options = {
 #    'navbar_links': [
 #    ("Google", "http://google.com", True)
 #]
-#}
+    'navbar_pagenav': False,
+}
 html_theme_path = sphinx_bootstrap_theme.get_html_theme_path()
 
+# Add sidebar to display toc
+html_sidebars = {
+    '**': ['localtoc.html']
+}
+
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".

diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -13,7 +13,9 @@ Welcome to ECS Community Edition's documentation!
 
    installation/ECS-Installation.rst
    use/ECS-UI-Web-Interface.rst
+   use/Migration.rst
    troubleshooting/ECS-Troubleshooting.rst
+   troubleshooting/FAQ.rst
 
 
 .. include:: ../../README.rst

diff --git a/docs/source/troubleshooting/ECS-Troubleshooting.md b/docs/source/troubleshooting/ECS-Troubleshooting.md
@@ -1,15 +1,28 @@
 # ECS Software 3.x - Troubleshooting Tips
 
-
-## Troubleshooting Tips
 This is a list of troubleshooting tips and nuggets that will help with issues. If you still have problems, please use the support section. 
+
+## Installation
+
+### If you change deploy.yml after running step1, you must run `update_deploy` before running step1 again. Otherwise you will likely get the following error:
+
+```
+{"failed": true, "msg": "An unhandled exception occurred while running the lookup plugin 'file'.
+ Error was a <class 'ansible.errors.AnsibleFileNotFound'>, original message: the file_name 
+ '/opt/ssh/id_ed25519.pub' does not exist, or is not readable"}
+
+```
+
+### `A block device configured in deploy.yml for data nodes is already partitioned.`
+
+This error often shows up after a failed installation attempt. In order to clean up the block devices to start over run `ecsremove purge-nodes`.
 
 
-### Provisioning of ECS 
+## Provisioning of ECS 
 
-It takes roughly 30 minutes to get the system provisioned for Step 2 (step2_object_provisioning.py).   ECS creates Storage Pools, Replication Groups with the attached disks. If Step 2 is successful, you should see something along these lines.
+It takes roughly 30 minutes to get the system provisioned for Step2.   ECS creates Storage Pools, Replication Groups with the attached disks. If Step2 is successful, you should see something along these lines.
 
-#### Adding a Secret Key for a user
+### Adding a Secret Key for a user
 
 Set the user and the key that needs to be used and execute the command. For example:
 
@@ -36,17 +49,14 @@ If you want to see if system is making progress:
 
 `curl -X GET "http://<YourIPAddress>:9101/stats/dt/DTInitStat”`
 
+## ECS Services
+
 
 ### Docker Container immediately exits on startup
 
 If your docker instance immediately exits when started, please ensure that the entries in `/etc/hosts` on the host system and `network.json` in the install directory are correct (the latter should reflect the host's public IP and the corresponding network adapter).
 
 
-### For those operating behind EMC firewall
-
-To install ECS Community Edition under these conditions, please view the readme file under **/emc-ssl-cert** for further instructions in installing the necessary CA certificate.
-
-
 ### Restoring ECS after host shutdown/restart
 
 In the case that the ECS Community Edition container does not automatically start on boot, you can bring it up manually by ensuring that docker is running (`service docker start`) and issuing a start command for the container (`docker start <container-id>`, where the container-ID is `ecsstandalone` or `ecsmultinode`, viewable via the command `sudo docker ps -a`).
@@ -63,6 +73,131 @@ For multiple-node installations, the `/etc/hosts` file on the host VM should inc
 
 If attempting to authenticate results in a response of "Connection Refused", review the below section and ensure all necessary ports are open on all ECS nodes in the cluster. 
 
+## NFS
+
+### Necessary NFS Ports
+The following ports must be opened for NFS to function properly
+
+Port Number |
+|---|
+| 111 |
+| 2049 |
+
+
+### NFS Volume Refuses to Mount
+
+ECS does support the NFS file system. However, troubles can occur when ECS is installed on the full version, or "Everything" version, of CentOS 7. ***Note that the following solution is not necessary on CentOS 7 Minimal.***
+
+#### The Problem
+CentOS 7 Everything starts with NFS/RPC/Portmap components running in the root scope. This is a problem as the ECS-CE Docker container runs its own version of rpcbind. This is the instance of rpcbind that ECS is intended to communicate with. When CentOS is running rpcbind in root scope in addition to the ECS Docker container, a conflict is created and a NFS volume cannot be mounted.
+
+This can be seen by `# rpcinfo -p` returning no NFS services.
+
+#### The Solution
+The conflict can be resolved by simply running `systemctl disable rpcbind`. This command will shut down the rpc service running on the host OS while leaving the Docker instance untouched.
+
+To confirm the CentOS service is gone, run `rpcinfo -p` in the CentOS shell. This should return an error: `rpcinfo: can't contact portmapper: RPC: Remote system error - No such file or directory`
+
+The same command, `rpcinfo-p`, can be run in the Docker container, which should return something similar to:
+```
+   program vers proto   port  service
+    100000    4   tcp    111  portmapper
+    100000    3   tcp    111  portmapper
+    100000    2   tcp    111  portmapper
+    100000    4   udp    111  portmapper
+    100000    3   udp    111  portmapper
+    100000    2   udp    111  portmapper
+    100005    3   tcp   2049  mountd
+    100005    3   udp   2049  mountd
+    100003    3   tcp   2049  nfs
+    100024    1   tcp   2049  status
+    100021    4   tcp  10000  nlockmgr
+    100021    4   udp  10000  nlockmgr
+```
+
+NFS should now function correctly. 
+
+## IBM Tivoli Monitoring
+
+### Issue
+ECS Community edition will fail to completely initialize the storage pool on machines that have the IBM Tivoli Monitoring agent installed.  The storage pool will forever stick in the "Initializing" state and attempts to create a VDC will result in HTTP 400 errors.
+
+### Analysis
+Doing a `ps -ef` inside the container will show that dataheadsvc and metering are restarting frequently.  Looking at `/opt/storageos/logs/metering.log` will show a bind exception on port 10110.  This port is already bound by Tivoli's `k10agent` process.
+
+### Workaround
+1. Uninstall Tivoli Monitoring
+or
+2. Change the port on impacted nodes.
+
+#### Changing the port on ECS
+On _all_ nodes, you will need to edit `/opt/storageos/conf/mt-var.xml` to change the bind port from 10110 to 10109.  Edit the file and change the line:
+
+```
+<property name="serviceUrl" value="service:jmx:rmi://127.0.0.1:10110/jndi/rmi://127.0.0.1:10111/sos" />
+```
+
+to:
+
+```
+<property name="serviceUrl" value="service:jmx:rmi://127.0.0.1:10109/jndi/rmi://127.0.0.1:10111/sos" />
+```
+
+Then restart the metering service:
+
+```
+kill `pidof metering`
+```
+
+## Network Troubleshooting
+
+### For those operating behind EMC firewall
+
+To install ECS Community Edition under these conditions, please view the readme file under **/emc-ssl-cert** for further instructions in installing the necessary CA certificate.
+
+### Disabling IPv6
+
+ECS Community Edition does not yet support IPv6. The following procedure can be used to disable IPv6 in CentOS 7. 
+
+### To disable IPv6 on startup:
+
+Add the following to /etc/sysctl.conf
+
+```
+net.ipv6.conf.all.disable_ipv6 = 1
+net.ipv6.conf.default.disable_ipv6 = 1
+```
+
+### To disable IPv6 running:
+
+```
+echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
+echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6
+```
+or
+
+```
+sysctl -w net.ipv6.conf.all.disable_ipv6=1
+sysctl -w net.ipv6.conf.default.disable_ipv6=1
+```
+
+### Get correct interface name
+
+CentOS 7 does not assign network interface names as eth0, eth1, etc, but rather assigns "predictable" names to each interface that generally look like `ens32` or similar. There are many benefits to this that can be read about [here](https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/).
+
+This can be disabled as documented in the above link, however, these names can otherwise be simply found and used in the ECS-Community installer without issue. To find the names for each device enter the following command: `ip a`. This command will output a list of network devices. Simply find the corresponding device and substitute it for eth0 in the stage1 installation script.
+
+### Port Conflicts
+
+It is possible that on multinode installations ECS may run into a port conflict. So far there exists a port conflict with the following:
+
+*   ScaleIO - Ports: 9011, 9099
+
+In these instances the user can attempt to: 
+
+1. Enter the container
+2. Change all instances of the conflicting ports to unused ports in `/opt/storageos/conf` 
+3. Reboot the nodes after altering the conf file.
 
 ### List of open ports required on each ECS data node
 
@@ -74,7 +209,7 @@ followed by `firewall-cmd --reload` for each host.
 
 `fwd_settings.sh` in the main directory will invoke the `firewalld` service and permanently open necessary ports. In the case of a failure in this setup referencing `iptables`, please ensure that your docker network bridge is running and installed using `yum install bridge-utils`.
 
-In the case of a multiple node configuration, you may
+
 
 |Port Name-Usage=Port Number|
 |---------------------------|