Reflect unhealthy providers in application status

[Vikrantpalle]

Feature or Problem

Scaler status returns a Failed status if it contains any providers which have failed their most recent health check and are in a Failed state.

Related Issues

• [FEAT] Reflect unhealthy providers in application status #447

Release Information

Consumer Impact

Testing

Unit Test(s)

Added 2 unit tests each for daemon and spread scalers

• test_unhealthy_providers_return_unhealthy_status
• test_healthy_providers_return_healthy_status

Acceptance or Integration

Manual Verification

Verified the scaler status after reconcile is run and that all unit tests passed.

[Vikrantpalle]

One potential edge case is if events ProviderHealthCheckFailed -> ProviderStopped get processed in the order ProviderStopped -> ProviderHealthCheckFailed. Then the status will remain in the Failed state for the provider and scaler. One potential fix is decaying out the provider but I'm not sure whether wadm already does this currently.

[brooksmtownsend]

Great work here @Vikrantpalle ! I have one request around how we're handling the health check event here, I think we should update the status with the message on the event but not actually do a full reconcile. Let me know if there's any more detail I can add there, and thank you for adding tests!

[Vikrantpalle]

I added a new Unhealthy status because I thought it would be more clear and wouldn't require matching the message inside the failed status to figure out whether a scaler is unhealthy. Feel free to let me know if you think this isn't required, I can revert to use the Failed status like before.

@brooksmtownsend

[brooksmtownsend]

Hey @Vikrantpalle I do like the idea of the Unhealthy status, for now though I think we have to use StatusType::Failed since using the new status will fail to deserialize on clients where that status doesn't exist yet 😕 It's not ideal, we should probably have a fallback there, but wash doesn't ATM.

I think it's good to keep the Unhealthy status, but if you can just make sure that the scaler uses Failed for now we can release this and then update the scaler in a future version to use Unhealthy

[brooksmtownsend]

Looks great!