LME docker image updates; move to Kibana 8.16.x

[jparsonstf]

Is there guidance from the LME team on upgrading docker images used with LME? Was curious about how the team plans for upstream updates, etc.

Today I'm encountering an error with an Elastic Integration and wish to upgrade to the latest version of it, but it requires Kibana 8.16.2 (or higher). LME is starts with 8.15.3 via containers.txt...

[mreeve-snl]

So I don' think we've thought about this completely, but I'll mark this as a feature request... ability to update container software versions

The basic steps are as follows:

1. ensure backup is made of all data in elasticsearch:

lme-user@ubuntu:~$ sudo -i podman volume inspect lme_backups
[
     {
          "Name": "lme_backups",
          "Driver": "local",
          "Mountpoint": "/var/lib/containers/storage/volumes/lme_backups/_data",
          "CreatedAt": "2024-11-01T21:01:32.905866871Z",
          "Labels": {},
          "Scope": "local",
          "Options": {},
          "UID": 166536,
          "GID": 166536,
          "MountCount": 0,
          "NeedsCopyUp": true,
          "NeedsChown": true,
          "LockNumber": 4
     }
]

that data should be on disk in a similar directory ^^, I'd copy it somewhere safe just in case

2. update containers.txt with the new version you want, where 8.16.X is the version you want:

lme-user@ubuntu:~$ cat ~/LME/config/containers.txt
docker.elastic.co/elasticsearch/elasticsearch:8.16.X
docker.elastic.co/beats/elastic-agent:8.16.X
docker.elastic.co/kibana/kibana:8.16.X
docker.io/wazuh/wazuh-manager:4.9.1
docker.io/jertel/elastalert2:2.20.0

3. This step you'll either want to make a new ansible script or look at the lines in the ansible script here: https://github.com/cisagov/LME/blob/main/ansible/install_lme_local.yml#L472 , and run them manually via sudo -i podman pull X and sudo -i podman tag X.
   This makes sure that the container tag version LME_LATEST is updated which is the container tag used by the lme container versions in their systemd files.

4. Since they're all running as quadlets all you should have to do in theory to run the new version is: sudo -i podman auto-update

5. Fix any issues with data and configurations that result from breaking changes between 8.15 -> 8.16....

I haven't been able to test the above.... so proceed with caution, but I would love to hear any feedback you have if you decide to attempt this yourself.

At some point, we'll probably add update functionality that automates all the above steps, but the biggest hurdle for that is determining the results of 5... testing the elasticsearch container updates to make sure updating elasticearch doesn't completely brick all of LME, there are breaking changes going from 8.15 -> 8.16.

Hope this helps!