Epic: last mile for monitoring

[acozine]

We would like to finish the work for a Minimum Viable Product for production monitoring. Here's what's left:

• complete configuration for distributed monitoring
• rename both OOBM sites (forrestal, new_south)
• confirm TLS monitoring [CheckMK] add tls integration checks #1211
• add monitoring for Solr 9 backups (Solr8 backups are currently monitored) see [CheckMK] Monitor Solr backups #5740
• confirm that all infrastructure work on [CheckMK] Make staging monitoring systems match production #5725 is done
• add all sites to the list of HTTP/S checks (including cert monitoring) on the canary VM, deduplicate as needed (can we automate this?)
• make sure Tower template works, see Create a Tower template for adding/updating the CheckMK agent #5718
• monitor disk read-only state, mounted disks (for example, Figgy), see [checkMK] Monitor disk read-only state #2471 and Can Datadog monitor mounts? #1000
• decide on a folder structure see [CheckMK] What folder structure should we be using?  #4846
• monitor nginxplus config, see [Check_MK] Monitor nginxplus config regularly #5364
• confirm that solr monitoring in CheckMK meets dev team needs, delete Datadog monitoring for Solr if it is duplicative, see Solr monitoring strategy ops-catchall#42
• deduplicate checks - anything that can be monitored from the main sites should be monitored from there (for example, temp and humidity settings); use the OOBM sites only for checks that cannot be implemented anywhere else
• document infrastructure (things like VM names, access instructions), maintenance (things like how to restart the service), and common tasks (things like adding a server, customizing checks) - see, for example, [CheckMK] Temp & Humidity Alert levels should be set separately per sensor ops-catchall#122 (comment) - we should put those directions somewhere besides the ticket