[Operational Awareness] HA support for KIM to improve availability and become scalable

[tobiscr]

Description

KIM is currently not running in HA mode as multiple replicas are not supported. This is blocking scalabiilty and can also have impacts on rolling updates.

To get rid of this restriction, KIM has to be adjusted to run in HA mode (multiple replicas) and it has to distribute the load proportionally over the available KIM instances.

AC:

• Implement a POC confirming that KIM is fully stateless and can run with multiple replicas.
  • Verify how load can be distributed over multiple instances
    • e.g. by sharding RuntimeCRs
    • using a shared queue like RabbitMQ
    • use leader election to ensure just on instance is processing the queue dispach events via Kubernetes events etc.
  • Propose an architectural solution for HA of KIM to the team which covers
    • pros / cons of the different options for HA (see list above)
    • how scalability will be supported by the chosen HA approach
• Depending on the architectural conclusion, adjust KIM to support HA

Reasons

HA is required to become scalable and improve availability of KIM.

Attachments