Make ssh-sync-server horizontally scalable

[therealpaulgg]

For the challenges, currently we store state in ChallengeResponseDict. This is thread-safe but single node.

This state is responsible for responding to challenges, but the problem with this approach is that it is not horizontally scalable...

If a user instantiates a challenge, that pops a thread on node A (which is probably fine). But then if we had a challenge response submitted to node B, node B would not be able to resolve the challenge as it isn't sharing the state that node A has.

If we used Redis to have a distributed state this might help. So node B checks redis, finds the state. It would also need to locate the machine that has the thread active, and notify it somehow. Could probably do this with a pub/sub system.

The app would still utilize channels, but would be receiving signals from Redis directly.

This may not be very necessary but would improve the scalability of ssh-sync.

[therealpaulgg]

https://redis.uptrace.dev/guide/go-redis-pubsub.html

[therealpaulgg]

In order to test this we would need some sort of load balancer to route traffic correctly...infra hell.

K8s, docker swarm

this may be the easiest way?