etcdclient randomly failing to delete keys

[akuzia]

I have long running processes with instance of *etcdv3.Client connected to etcd v3.5.5 cluster with tls & json token auth.
Basicaly it is grpc server that exposes parts of etcd api to outside world.

cluster auth config:
auth-token: jwt,pub-key=/etcd/pki/jwt.pub,priv-key=/etcd/pki/jwt.key,sign-method=RS256

Something like so:

type apiServer struct {
	etcd   *etcdv3.Client
}

// Del implements grpc v1.EtcdApiServer
func (s *apiServer) Del(ctx context.Context, req *v1.Del) (*v1.Deleted, error) {
	var total int64
	for _, key := range req.GetKey() {
		resp, err := s.etcd.Delete(ctx, key)
		if err != nil {
			s.logger.Error("etcd del error", zap.Error(err))
		} else {
			total += resp.Deleted
		}
	}

	return &v1.Deleted{
		Count: total,
	}, nil
}

Sometimes there are errors with warn logs after del operation has failed:

{"level":"warn","ts":"2023-08-09T20:33:26.315Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001adc00/192.168.16.6:2379","attempt":0,"error":"rpc error: code = PermissionDenied desc = etcdserver: permission denied"}

but the rest of the stream is processed without any errors on the same client instance.

smth like:
/web/foo.bar.com - deleted fine
/web/bar.bar.com - spits warn log
/web/baz.bar.com - deleted fine

All of the keys have same prefix /web
Connected user role:

Role web_rw
KV Read:
	[/web, /wec) (prefix /web)
KV Write:
	[/web, /wec) (prefix /web)

Is it related to my auth settings? I've had similar logs from retry_interceptor when auth token was expired.
Should I implement some kind of a retry? Is there an error I can compare to to trigger a retry?