[CheckMK] Make staging monitoring systems match production

[acozine]

What maintenance needs to be done?

Upgrade/reinstall and rename the CheckMK staging server and site so our two monitoring environments are as similar as possible.

Level of urgency

• High
• Moderate
• Low

Why is this maintenance needed?

To make it easy to test future changes to our monitoring platform, we want the staging and production systems to be as similar as possible. In particular, we want:

• clear naming for CheckMK VMs - prod and staging names should be related
• clear DNS names for CheckMK - prod and staging site names should be related, and the sites should be easy to remember/find
• software version and type should match in staging and production; this allows us to
  • test performance in staging before moving to prod
  • test integrating our OOBM sites (which are already running the paid version)

Acceptance criteria

• we have two VMs for CheckMK - one prod, one staging - with identical resources (memory, CPU, storage), clear names that reflect their envs (e.g. if prod is 'checkmk-prod1' call the staging VM 'checkmk-staging1'), and the same version and type of CheckMK installed
• we have two sites for CheckMK - one prod, one staging - with names that match ('checkmk.princeton.edu' and 'checkmk-staging.princeton.edu')

Implementation notes, if any

• rebuild the VM for staging CheckMK, name it so it matches production (e.g. if prod is 'checkmk-prod1' call the staging VM 'checkmk-staging1')
• update princeton_ansible inventory as needed
• set up a new site for staging that matches prod ('checkmk.princeton.edu' and 'checkmk-staging.princeton.edu')
• install the paid version of CheckMK on the new staging VM, using our Ansible playbook/role
• back up the old, free-version staging CheckMK instance and restore it to the new, paid-version staging VM/site
• decommission the old staging site and VM

[acozine]

Putting everything on one server led to performance issues, so we have changed tacks and created a distributed architecture. We will have one production VM for monitoring production infrastructure and a second production VM for monitoring staging infrastructure. Since these are both production VMs, we can call them pulmonitor-prod1 and pulmonitor-prod2 - the first will run pulmonitor.princeton.edu/production and the second will run pulmonitor.princeton.edu/staging.

We will also have a production VM for each data center to monitor anything we cannot connect to from the outside.