Intermittent problems with EPEL 6 repository

[villasv]

Once in a while yum makecache fails, apparently because EPEL screws up (epel: Check uncompressed DB failed) and expected the wrong payload size ([Errno 14] HTTP Error 416 - Requested Range Not Satisfiable for all its mirrors).

Perhaps I can make this problem go from "uncommon" to "rare" if I configure yum to ignore unavailable repositories (yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true), therefore encouraging it to ignore the cache.

Shouldn't be very impactful, the only use of EPEL6 at the moment is the supervisor package.

[villasv]

I believe this is going to be an open issue forever unless I stop using supervisor from EPEL. Depending on the progress of #34, this will be fixed by only by switching to ubuntu/debian.

[villasv]

Allright, no running away from Amazon Linux AMI for now, CloudInit is too convenient.

I'll move it from Blocked to Todo because although I can't hope to fix the EPEL6 instability, there should be a way to work around it and its consequence should be only slower-but-well-logged deployments.

[villasv]

$ cat /var/log/cfn-init.log
2018-11-13 12:01:12,907 [INFO] -----------------------Starting build-----------------------
2018-11-13 12:01:12,909 [INFO] Running configSets: default
2018-11-13 12:01:12,910 [INFO] Running configSet default
2018-11-13 12:01:12,912 [INFO] Running config filesystem
2018-11-13 12:01:40,984 [ERROR] Yum makecache failed. Output: Loaded plugins: priorities, update-motd, upgrade-helper
Existing lock /var/run/yum.pid: another copy is running as pid 2777.
Another app is currently holding the yum lock; waiting for it to exit...
  The other application is: yum
    Memory : 100 M RSS (346 MB VSZ)
    Started: Tue Nov 13 12:00:20 2018 - 00:53 ago
    State  : Running, pid: 2777
http://mirror.es.its.nyu.edu/epel/6/x86_64/repodata/1c3758034bd3d8080554d35fd733b6b0ecc6b0a6b0fa117e3a5057fddc935514-primary.xml.gz: [Errno 14] HTTP Error 416 - Requested Range Not Satisfiable
Trying other mirror.

... <<many other mirrors>> ...

[Errno 2] No such file or directory: '/var/cache/yum/x86_64/latest/epel/af866f0fa28368b9c495a472afb343840d0f84c47d064c94c69613f587676b1f-filelists.xml.gz'

2018-11-13 12:01:40,984 [ERROR] Error encountered during build of filesystem: Could not create yum cache (return code 1)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 229, in build
    changes['packages'][manager] = CloudFormationCarpenter._packageTools[manager]().apply(packages, self._auth_config)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/rpm_tools.py", line 51, in apply
    raise ToolError("Could not create yum cache", cache_result.returncode)
ToolError: Could not create yum cache (return code 1)
2018-11-13 12:01:40,986 [ERROR] -----------------------BUILD FAILED!------------------------
2018-11-13 12:01:40,989 [ERROR] Unhandled exception during build: Could not create yum cache (return code 1)
Traceback (most recent call last):
  File "/opt/aws/bin/cfn-init", line 171, in <module>
    worklog.build(metadata, configSets)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 129, in build
    Contractor(metadata).build(configSets, self)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 530, in build
    self.run_config(config, worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 229, in build
    changes['packages'][manager] = CloudFormationCarpenter._packageTools[manager]().apply(packages, self._auth_config)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/rpm_tools.py", line 51, in apply
    raise ToolError("Could not create yum cache", cache_result.returncode)
ToolError: Could not create yum cache (return code 1)

[villasv]

Will be solved after we're done with #53 and we're ready to ditch supervisord, the only package from EPEL being used.

https://airflow.readthedocs.io/en/stable/howto/run-with-systemd.html