New travis build system. Fix tests.

Now the travis build can be run by a simple
    rake ci:run

Because we use a lot of external software to run the tests and they
can use a lot of CPU/memory although the travis boxes are not so big.
Here is a very short description of what it does.

1. There are a bunch of different rake tasks ci:flavor:execute.
   Each `execute` task will run like in travis 4 steps:
     a. before_install (update mirrors/start stuff)
     b. install
     c. before_script (preload fixtures or change conf files)
     d. script (the actual "build" / lint / test suite)
   You can find all these tasks in the ci/ folder.
   And to execute one flavor or several flavors you can use the same
   high-level `ci:run` task if you set the `TRAVIS_FLAVOR` env variable
   to a `flavor` or a group of flavors: `flavor1,flavor2`.
   This allows the tests to run in a travis matrix and group them so
   they don't overload the box.

2. Each task will also call the `common` namespace tasks as dependencies
   for the 4 steps (install agent deps, conf files, install agent)

3. There is a run_tests rake task in `common`. When you call a flavored
   travis build it will eventually call `run_tests` to launch the test
   suite and you need to pass a `provides` array containing the deps for
   this flavor.
   For instance if I launch `webserver` this provides `[haproxy,nginx]`
   and it will launch `nosetests` with a filter so that it runs only tests
   which `requires` attributes is in the `provides` list.

Other squashed changes:
- Init set with braces is not python 2.6 compliant
See:
    d = { 'test' }
                 ^
SyntaxError: invalid syntax

- Fix nagios passive check class init
- Be less rigid to avoid flakiness on service_checks tests
  However it should be fixed, this is not the expected behavior.
  Somehow, before the flush things still get processed.

.pylintrc

@@ -40,7 +40,7 @@ load-plugins=
 # --disable=W"
 #disable=
 #
-disable=W,C,R,maybe-no-member,no-member
+disable=W,C,R,maybe-no-member,no-member,I0011
 # FIXME matt: pylint has a bug that will not let you gradually enable
 # rules in the rcfile, so this is done in our rakefile.
 # http://www.logilab.org/ticket/36584

.travis.yml

@@ -1,70 +1,38 @@
-# blacklist
-branches:
-  except:
-    - check-haproxy
-services:
-  - mysql
-  - elasticsearch
-  - memcache
-  - redis-server
-  - postgresql: "9.3"
-
 language: python
 python:
   - "2.6"
   - "2.7"
-before_install:
-  - sudo apt-get update
-  - sudo apt-get install openjdk-6-jre-headless
-  - sudo apt-get install sysstat
-  - sudo apt-get install haproxy
-  - sudo apt-get install python-mysqldb
-  - sudo apt-get install tomcat6
-  - sudo apt-get install solr-tomcat
-  - sudo apt-get install nginx
-  - sudo apt-get install apache2
-  - sudo apt-get install couchdb
-  - sudo apt-get install lighttpd
-  - sudo apt-get install gearman
-  - sudo apt-get install snmpd
-install:
-  - pip install -r requirements.txt --use-mirrors
-  - pip install . --use-mirrors
-before_script:
-  - mysql -e "create user 'dog'@'localhost' identified by 'dog'"
-  - psql -U postgres -c "create user datadog with password 'datadog'"
-  - psql -U postgres -c "grant SELECT ON pg_stat_database to datadog"
-  - psql -U postgres -c "CREATE DATABASE datadog_test" postgres
-  - psql -U postgres -c "GRANT ALL PRIVILEGES ON DATABASE datadog_test to datadog"
-  - psql -U datadog -c "CREATE TABLE Persons (PersonID int, LastName varchar(255), FirstName varchar(255), Address varchar(255), City varchar(255))" datadog_test
-  - curl -L https://raw.github.com/DataDog/dd-agent/master/tests/haproxy.cfg > /tmp/haproxy.cfg
-  - curl http://apache.mesi.com.ar/cassandra/2.0.9/apache-cassandra-2.0.9-bin.tar.gz | tar -C /tmp -xzv
-  - sudo /tmp/apache-cassandra-2.0.9/bin/cassandra
-  - sudo service haproxy restart
-  - sudo bash -c "curl -L https://raw.github.com/DataDog/dd-agent/master/tests/tomcat_cfg.xml > /etc/tomcat6/server.xml"
-  - sudo bash -c "curl -L https://raw.github.com/DataDog/dd-agent/master/tests/tomcat6 >> /etc/default/tomcat6"
-  - sudo service nginx stop
-  - sudo bash -c "curl -L https://raw.github.com/DataDog/dd-agent/master/tests/nginx.conf > /etc/nginx/conf.d/default.conf"
-  - sudo service apache2 stop
-  - sudo bash -c "curl -L https://raw.github.com/DataDog/dd-agent/master/tests/apache/ports.conf > /etc/apache2/ports.conf"
-  - sudo bash -c "curl -L https://raw.github.com/DataDog/dd-agent/master/tests/apache/apache.conf > /etc/apache2/apache.conf"
-  - sudo /etc/init.d/lighttpd stop
-  - sudo bash -c "curl -L https://raw.github.com/DataDog/dd-agent/master/tests/lighttpd/lighttpd.conf > /etc/lighttpd/lighttpd.conf"
-  - sudo mkdir -p /etc/dd-agent/
-  - sudo install -d -o "$(id -u)" /var/log/datadog
-  - sudo bash -c "curl -L https://raw.github.com/DataDog/dd-agent/master/datadog.conf.example > /etc/dd-agent/datadog.conf"
-  - sudo service snmpd stop
-  - sudo bash -c "curl -L https://raw.github.com/DataDog/dd-agent/master/tests/snmp/snmpd.conf > /etc/snmp/snmpd.conf"
-  - sudo service apache2 start
-  - sudo service nginx start
-  - sudo /etc/init.d/lighttpd start
-  - sudo service tomcat6 restart
-  - sudo service snmpd start
+
 env:
-  - DB=redis
+  global:
+    - NOSE_FILTER="not windows"
+  matrix:
+    - TRAVIS_FLAVOR=default
+    - TRAVIS_FLAVOR=cache,gearman,database
+    - TRAVIS_FLAVOR=cassandra,tomcat,jmx # JMX testing machine / need the other ones before
+    - TRAVIS_FLAVOR=elasticsearch,network,sysstat,webserver
+    - TRAVIS_FLAVOR=mongo
+
+matrix:
+  allow_failures:
+    - env: TRAVIS_FLAVOR=mongo
+
+# Override travis defaults with empty jobs
+before_install: echo "OVERRIDING TRAVIS STEPS"
+install: echo "OVERRIDING TRAVIS STEPS"
+before_script: echo "OVERRIDING TRAVIS STEPS"
+
 script:
-   - pylint --rcfile=./.pylintrc *.py */*.py */*/*.py 
-   - nosetests -v -A 'not windows' tests
+  - bundle install
+  - 'rake ci:run'
+
+after_failure:
+  - echo "Logs from installation process come here / DEBUG LOGS"
+  - cat /tmp/ci.log
+  - bash -c "if [[ \"$TRAVIS_FLAVOR\" =~ \"mongo\" ]]; then cat /tmp/mongo.log; fi"
+  - bash -c "if [[ \"$TRAVIS_FLAVOR\" =~ \"mongo\" ]]; then cat /data/mongod1/mongo.log; fi"
+  - bash -c "if [[ \"$TRAVIS_FLAVOR\" =~ \"mongo\" ]]; then cat /data/mongod2/mongo.log; fi"
+
 notifications:
-  webhooks:
-    - https://www.buildheroes.com/api/projects/1fa0bf90eb6f781bf067c82439775fd9d896c0ae/builds
+  - hipchat:
+     - secure: "SYTDsiV28Tv9bGMdtTu/MfMcpCpnKowDUJtxzK8dhOOliEKGa4F8JJT38QS5mlMhzi5TTyjs8UpXWyrAz133OOGuyt0hHEJYtm7wckNlErOwJA+A8JWRDCmOduDRn7W/A7ufx7yN3QGBtB2SS5pW3A9abi8y6mBDPUf/JCS6/aQ="

Gemfile

@@ -0,0 +1,4 @@
+source "https://rubygems.org"
+
+gem 'colorize'
+gem 'rake'

Rakefile

@@ -3,6 +3,20 @@
 
 require 'rake/clean'
 
+# Flavored Travis CI jobs
+require './ci/cache'
+require './ci/cassandra'
+require './ci/database'
+require './ci/default'
+require './ci/elasticsearch'
+require './ci/gearman'
+require './ci/jmx'
+require './ci/mongo'
+require './ci/network'
+require './ci/sysstat'
+require './ci/tomcat'
+require './ci/webserver'
+
 CLOBBER.include '**/*.pyc'
 
 desc "Run tests"
@@ -15,7 +29,7 @@ end
 desc 'Setup a development environment for the Agent'
 task "setup_env" do
    `mkdir -p venv`
-   `wget -O venv/virtualenv.py https://raw.github.com/pypa/virtualenv/1.11.X/virtualenv.py`
+   `wget -O venv/virtualenv.py https://raw.github.com/pypa/virtualenv/1.11.6/virtualenv.py`
    `python venv/virtualenv.py  --no-pip --no-setuptools venv/`
    `wget -O venv/ez_setup.py https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py`
    `venv/bin/python venv/ez_setup.py`
@@ -57,4 +71,14 @@ task "run" do
   sh("supervisord -n -c supervisord.dev.conf")
 end
 
+namespace :ci do
+  desc 'Run Travis CI flavored tests'
+  task :run, :flavor  do |t, args|
+    fail "Failing because this is supposed to run on Travis" unless ENV['TRAVIS']
+    flavor = args[:flavor] || ENV['TRAVIS_FLAVOR'] || 'default'
+    flavors = flavor.split(',')
+    flavors.each { |f| Rake::Task["ci:#{f}:execute"].invoke}
+  end
+end
+
 task :default => [:test]

checks.d/nagios.py

@@ -238,7 +238,7 @@ def __init__(self, log_path, file_template, logger, hostname, event_func,
         :param passive_checks: bool, enable or not passive checks events
         '''
         self.passive_checks = passive_checks
-        super(NagiosEventLogTailer).__init__(self, log_path, file_template,
+        super(NagiosEventLogTailer, self).__init__(log_path, file_template,
             logger, hostname, event_func, gauge_func, freq)
 
     def _parse_line(self, line):

checks.d/rabbitmq.py

@@ -235,7 +235,7 @@ def _check_aliveness(self, base_url, vhosts=None):
             vhosts = [v['name'] for v in vhosts_response]
 
         for vhost in vhosts:
-            tags = {'vhost:%s' % vhost}
+            tags = ['vhost:%s' % vhost]
             # We need to urlencode the vhost because it can be '/'.
             path = u'aliveness-test/%s' % (urllib.quote_plus(vhost))
             aliveness_url = urlparse.urljoin(base_url, path)

checks.d/vsphere.py

@@ -16,7 +16,8 @@
 
 # 3rd party
 from pyVim import connect
-from pyVmomi import vim
+# This drives travis-ci pylint crazy!
+from pyVmomi import vim # pylint: disable=E0611
 
 SOURCE_TYPE = 'vsphere'
 REAL_TIME_INTERVAL = 20 # Default vCenter sampling interval

ci/cache.rb

@@ -0,0 +1,27 @@
+require './ci/common'
+
+namespace :ci do
+  namespace :cache do
+    task :before_install => ['ci:common:before_install'] do
+      # memcache - already installed on Travis
+      sh %Q{sudo service memcached restart}
+
+      # redis-server - already installed on Travis
+      sh %Q{sudo service redis-server restart}
+    end
+
+    task :install => ['ci:common:install']
+
+    task :before_script => ['ci:common:before_script']
+
+    task :script => ['ci:common:script'] do
+      this_provides = [
+        'memcache',
+        'redis',
+      ]
+      Rake::Task['ci:common:run_tests'].invoke(this_provides)
+    end
+
+    task :execute => [:before_install, :install, :before_script, :script]
+  end
+end

ci/cassandra.rb

@@ -0,0 +1,26 @@
+require './ci/common'
+
+namespace :ci do
+  namespace :cassandra do
+    task :before_install => ['ci:common:before_install'] do
+      sh %Q{curl http://apache.mesi.com.ar/cassandra/2.0.9/apache-cassandra-2.0.9-bin.tar.gz | tar -C /tmp -xz}
+    end
+
+    task :install => ['ci:common:install']
+
+    task :before_script => ['ci:common:before_script'] do
+      sh %Q{sudo /tmp/apache-cassandra-2.0.9/bin/cassandra}
+      # Wait for cassandra to init
+      sh %Q{sleep 10}
+    end
+
+    task :script => ['ci:common:script'] do
+      this_provides = [
+        'cassandra',
+      ]
+      Rake::Task['ci:common:run_tests'].invoke(this_provides)
+    end
+
+    task :execute => [:before_install, :install, :before_script, :script]
+  end
+end

ci/common.rb

@@ -0,0 +1,75 @@
+require 'colorize'
+require 'time'
+
+def apt_update
+  sh "sudo apt-get update -qq"
+end
+
+def sleep_for(secs)
+  puts "Sleeping for #{secs}s".blue
+  sleep(secs)
+end
+
+def section(name)
+  timestamp = Time.now.utc.iso8601
+  puts ""
+  puts "[#{timestamp}] >>>>>>>>>>>>>> #{name} STAGE".black.on_white
+  puts ""
+end
+
+namespace :ci do
+  namespace :common do
+    task :before_install do |t|
+      section('BEFORE_INSTALL')
+      t.reenable
+    end
+
+    task :install do |t|
+      section('INSTALL')
+      marker_file = '/tmp/COMMON_INSTALL_DONE'
+      unless File.exists?(marker_file)
+        sh "pip install -r requirements.txt --use-mirrors 2>&1 >> /tmp/ci.log"
+        sh "pip install -r test-requirements.txt --use-mirrors 2>&1 >> /tmp/ci.log"
+        sh "pip install . --use-mirrors 2>&1 >> /tmp/ci.log"
+        sh "touch #{marker_file}"
+      else
+        puts "Skipping common installs, already done by another task".yellow
+      end
+      t.reenable
+    end
+
+    task :before_script do |t|
+      section('BEFORE_SCRIPT')
+      marker_file = '/tmp/COMMON_BEFORE_SCRIPT_DONE'
+      unless File.exists?(marker_file)
+        sh "sudo mkdir -p /etc/dd-agent/"
+        sh %Q{sudo install -d -o "$(id -u)" /var/log/datadog}
+        sh "sudo cp $TRAVIS_BUILD_DIR/datadog.conf.example /etc/dd-agent/datadog.conf"
+        sh "touch #{marker_file}"
+      else
+        puts "Skipping common env setup, already done by another task".yellow
+      end
+      t.reenable
+    end
+
+    task :script do |t|
+      section('SCRIPT')
+      t.reenable
+    end
+
+    task :run_tests, :flavor do |t, attr|
+      flavor = attr[:flavor]
+      filter = ENV['NOSE_FILTER'] || 'True'
+      if flavor == 'default'
+        nose = "(not requires) and #{filter}"
+      else
+        nose = "(requires in #{flavor}) and #{filter}"
+      end
+      # FIXME make the other filters than param configurable
+      sh %Q{nosetests -v -A '#{nose}' tests}
+      t.reenable
+    end
+
+    task :execute => [:before_install, :install, :before_script, :script]
+  end
+end

ci/database.rb

@@ -0,0 +1,47 @@
+require './ci/common'
+
+namespace :ci do
+  namespace :database do
+    task :before_install => ['ci:common:before_install'] do
+      # postgres
+      # TODO: rely on Travis preinstalled postgres instances, fetch it from PG repo?
+      sh "sudo service postgresql stop"
+      # FIXME: include this as a version number in the matrix
+      sh "sudo service postgresql start 9.3"
+
+      # mysql - should already be installed to - ensure it is started
+      sh %Q{sudo service mysql restart}
+      # couchdb - should already be installed to - ensure it is started
+      sh %Q{sudo service couchdb restart}
+
+      # don't really like it but wait a few seconds for all these services to init
+      # especially couchdb
+      sleep(5)
+    end
+
+    task :install => ['ci:common:install']
+
+    task :before_script => ['ci:common:before_script'] do
+      # postgres
+      sh %Q{psql -U postgres -c "create user datadog with password 'datadog'"}
+      sh %Q{psql -U postgres -c "grant SELECT ON pg_stat_database to datadog"}
+      sh %Q{psql -U postgres -c "CREATE DATABASE datadog_test" postgres}
+      sh %Q{psql -U postgres -c "GRANT ALL PRIVILEGES ON DATABASE datadog_test to datadog"}
+      sh %Q{psql -U datadog -c "CREATE TABLE Persons (PersonID int, LastName varchar(255), FirstName varchar(255), Address varchar(255), City varchar(255))" datadog_test}
+
+      # mysql
+      sh %Q{mysql -e "create user 'dog'@'localhost' identified by 'dog'"}
+    end
+
+    task :script => ['ci:common:script'] do
+      this_provides = [
+        'couchdb',
+        'mysql',
+        'postgres'
+      ]
+      Rake::Task['ci:common:run_tests'].invoke(this_provides)
+    end
+
+    task :execute => [:before_install, :install, :before_script, :script]
+  end
+end

ci/default.rb

@@ -0,0 +1,22 @@
+require './ci/common'
+
+namespace :ci do
+  namespace :default do
+    task :before_install => ['ci:common:before_install'] do
+      apt_update
+    end
+
+    task :install => ['ci:common:install'] do
+      sh %Q{sudo apt-get install sysstat -qq}
+    end
+
+    task :before_script => ['ci:common:before_script']
+
+    task :script => ['ci:common:script'] do
+      sh "find . -name '*.py' | xargs --max-procs=0 -n 1 pylint --rcfile=./.pylintrc"
+      Rake::Task['ci:common:run_tests'].invoke('default')
+    end
+
+    task :execute => [:before_install, :install, :before_script, :script]
+  end
+end