feat: add health for monovertex

[kohlisid]

fixes #1952
For data health, this adds critical status, warning status to be added in a follow up

[codecov]

Codecov Report

Attention: Patch coverage is 28.12500% with 138 lines in your changes missing coverage. Please review.

Project coverage is 57.66%. Comparing base (a52102d) to head (b059b90).
Report is 7 commits behind head on main.

Files	Patch %	Lines
pkg/mvtxdaemon/server/service/mvtx_service.go	0.00%	37 Missing ⚠️
server/apis/v1/health.go	0.00%	36 Missing ⚠️
pkg/mvtxdaemon/server/service/health_status.go	65.33%	24 Missing and 2 partials ⚠️
server/apis/v1/handler.go	0.00%	18 Missing ⚠️
pkg/mvtxdaemon/client/restful_client.go	0.00%	9 Missing ⚠️
pkg/mvtxdaemon/server/daemon_server.go	0.00%	6 Missing ⚠️
pkg/mvtxdaemon/client/grpc_client.go	0.00%	5 Missing ⚠️
server/routes/routes.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1954      +/-   ##
==========================================
- Coverage   57.77%   57.66%   -0.12%     
==========================================
  Files         412      414       +2     
  Lines       28789    28966     +177     
==========================================
+ Hits        16633    16702      +69     
- Misses      11214    11326     +112     
+ Partials      942      938       -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[whynowy]

Using google.protobuf.Empty for the request is good enough.

[kohlisid]

Sure! For now I had copied over the pipeline template and was just adding the implementation.
I checked, and we wouldn't need it the pipeline API as well I guess
I'll update for both

[kohlisid]

  // GetPipelineWatermarks return the watermark of the given pipeline
  rpc GetPipelineWatermarks (GetPipelineWatermarksRequest) returns (GetPipelineWatermarksResponse) {
    option (google.api.http).get = "/api/v1/pipelines/{pipeline}/watermarks";
  };

  rpc GetPipelineStatus (GetPipelineStatusRequest) returns (GetPipelineStatusResponse) {
    option (google.api.http).get = "/api/v1/pipelines/{pipeline}/status";
  };

Actually, we won't need for these 2 pipeline level APIs to daemon

[whynowy]

This is right.

[kohlisid]

@whynowy
Sorry just to clarify, you mean that you are good with the current implementation or are you good to change them to empty?
We don't use the request so I believe we can update to empty

[whynowy]

Let's use empty for MonoVertex. For Pipeline, we need to do several iterations if we want to correct it, leave it for now.

[kohlisid]

Sure, Will create an issue to track that and refactor for later!

[kohlisid]

https://localhost:8443/api/v1/namespaces/default/mono-vertices/simple-mono-vertex/health
{
    "data": {
        "resourceHealthStatus": "healthy",
        "dataHealthStatus": "healthy",
        "resourceHealthMessage": "mono vertex \"simple-mono-vertex\" is healthy",
        "dataHealthMessage": "MonoVertex data flow is healthy",
        "resourceHealthCode": "M1",
        "dataHealthCode": "D1"
    },
    "errMsg": null
}

[kohlisid]

https://localhost:4327/api/v1/status
{
  "status": {
    "status": "healthy",
    "message": "MonoVertex data flow is healthy",
    "code": "D1"
  }
}

[vigith]

What about the case when pods are continuously being restarted? or what if the TPS is low and pending and is growing very slowly?

[vigith]

It looks like cases like pod restarting will be caught by resource health.