Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deploying w/ Google Provider causing Halyard 500 - "spinnaker validation failed" #203

Open
EIrwin opened this issue Jan 6, 2021 · 6 comments
Open

Deploying w/ Google Provider causing Halyard 500 - "spinnaker validation failed" #203

EIrwin opened this issue Jan 6, 2021 · 6 comments

Comments

@EIrwin
Copy link

EIrwin commented Jan 6, 2021 •
edited
Loading

I wanted to run a test for a fresh install of Spinnaker w/ OSS Armory Operator since I had patched it together to get it up and running.

I am using a very similar setup to spinnaker-kustomize-patches and deploying using Kustomize.

After removing Spinnaker, Operator, and external resources and running a fresh install, I am receiving the following error preventing Spinnaker from being deployed.

Error from server (
SpinnakerService validation failed:
Halyard validator detected an error:
  got halyard response status 500, response: No message available
): error when creating "/Users/eric/source/repos/delivery/spinnaker": admission webhook "webhook-spinnakerservices-v1alpha2.spinnaker.io" denied the request:

Investigating further, I see the following logs in the halyard container in spinnaker-operator pod.

2021-01-05 23:00:56.946 ERROR 1 --- [nio-8064-exec-4] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.NullPointerException] with root cause

java.lang.NullPointerException: null
	at com.netflix.spinnaker.halyard.config.model.v1.node.Validator.validatingFileDecryptBytes(Validator.java:51) ~[halyard-config-operator-7162184.jar:na]
	at com.netflix.spinnaker.halyard.config.model.v1.node.Validator.validatingFileDecrypt(Validator.java:42) ~[halyard-config-operator-7162184.jar:na]
	at com.netflix.spinnaker.halyard.config.validate.v1.providers.google.GoogleAccountValidator.getJsonKey(GoogleAccountValidator.java:83) ~[halyard-config-operator-7162184.jar:na]
	at com.netflix.spinnaker.halyard.config.validate.v1.providers.google.GoogleAccountValidator.validate(GoogleAccountValidator.java:54) ~[halyard-config-operator-7162184.jar:na]
	at com.netflix.spinnaker.halyard.config.validate.v1.providers.google.GoogleProviderValidator.lambda$validate$0(GoogleProviderValidator.java:39) ~[halyard-config-operator-7162184.jar:na]
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1540) ~[na:na]
	at com.netflix.spinnaker.halyard.config.validate.v1.providers.google.GoogleProviderValidator.validate(GoogleProviderValidator.java:39) ~[halyard-config-operator-7162184.jar:na]
	at com.netflix.spinnaker.halyard.config.validate.v1.providers.google.GoogleProviderValidator.validate(GoogleProviderValidator.java:28) ~[halyard-config-operator-7162184.jar:na]
	at com.netflix.spinnaker.halyard.deploy.services.v1.ValidationRun$NodeValidator.validate(ValidationRun.java:109) ~[halyard-deploy-operator-7162184.jar:na]
	at com.netflix.spinnaker.halyard.deploy.services.v1.ValidationRun.validateNode(ValidationRun.java:69) ~[halyard-deploy-operator-7162184.jar:na]
	at com.netflix.spinnaker.halyard.deploy.services.v1.ValidationRun.visitNode(ValidationRun.java:49) ~[halyard-deploy-operator-7162184.jar:na]
	at com.netflix.spinnaker.halyard.deploy.services.v1.ValidationRun.visitNode(ValidationRun.java:60) ~[halyard-deploy-operator-7162184.jar:na]
	at com.netflix.spinnaker.halyard.deploy.services.v1.ValidationRun.visitNode(ValidationRun.java:60) ~[halyard-deploy-operator-7162184.jar:na]
	at com.netflix.spinnaker.halyard.deploy.services.v1.ValidationRun.run(ValidationRun.java:44) ~[halyard-deploy-operator-7162184.jar:na]
	at com.netflix.spinnaker.halyard.deploy.services.v1.DynamicValidationService.validate(DynamicValidationService.java:62) ~[halyard-deploy-operator-7162184.jar:na]
	at com.netflix.spinnaker.halyard.controllers.v1.ValidationController.validateConfig(ValidationController.java:42) ~[halyard-web-operator-7162184.jar:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) ~[spring-webmvc-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878) ~[spring-webmvc-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792) ~[spring-webmvc-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:652) ~[tomcat-embed-core-9.0.40.jar:4.0.FR]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) ~[tomcat-embed-core-9.0.40.jar:4.0.FR]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.31.jar:9.0.31]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:158) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:92) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:77) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:109) ~[spring-boot-actuator-2.2.5.RELEASE.jar:2.2.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:880) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1601) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.40.jar:9.0.40]
	at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

Seeing this, I commented out patch-google.yml out of my kustomization.yml file and I no longer receive the error.

patch-google.yml looks like the following:

apiVersion: spinnaker.io/v1alpha2
kind: SpinnakerService
metadata:
  name: spinnaker
spec:
  spinnakerConfig:
    config:
      providers:
        google:
          enabled: true
          primaryAccount: my-project
          accounts:
          - name: my-project
            requiredGroupMembership: []
            permissions: {}
            project: my-project
            jsonPath: encryptedFile:k8s!n:spin-secrets!k:gcp-sa.json
            alphaListed: false
            imageProjects: []
            consul:
              enabled: false
              agentEndpoint: localhost
              agentPort: 8500
              datacenters: []

Since the log we see has the following, it seems to imply that the jsonPath: encryptedFile:k8s!n:spin-secrets!k:gcp-sa.json could be problematic.

  return validatingFileDecrypt(p, n.getJsonPath());

I have validated that spin-secrets in fact has this gcp-sa.json

kubectl -n spinnaker get secrets spin-secrets -o json | jq  .data | jq "keys"                                                                                     eric@Erics-MacBook-Pro
[
  "cloudsql_password",
  "gcp-sa.json",
  "gitlab_token",
  "jenkins_password",
  "kubecfg_internal",
  "oidc_client_id",
  "oidc_client_secret",
  "slack_token"
]

I have attempted to disable validation for all providers, but had no success:

apiVersion: spinnaker.io/v1alpha2
kind: SpinnakerService
metadata:
  name: spinnaker
spec:
  validation:
    providers:
      gce:
        enabled: false
      kubernetes:
        enabled: false
      docker:
        enabled: false
      ci:
        enabled: false
      metricStores:
        enabled: false
      persistentStorage:
        enabled: false
      notifications:
        enabled: false

One important thing to note is that I DID have this running with the exact same configuration prior to tearing it down and rerunning a full deployment. I am not sure if this suggests that there is either a race condition, or problem with sequencing, but wondering if there is a bug deep in there.
@EIrwin EIrwin changed the title Enabling Google Provider causes Halyard 500 - "spinnaker validation failed" Deploying w/ Google Provider causing Halyard 500 - "spinnaker validation failed" Jan 6, 2021
@EIrwin
Copy link
Author

EIrwin commented Jan 14, 2021

Ran into this as well when patching canary with the following:

apiVersion: spinnaker.io/v1alpha2
kind: SpinnakerService
metadata:
  name: spinnaker
spec:
  spinnakerConfig:
    config:
      canary:
        enabled: true
        reduxLoggerEnabled: false
        defaultMetricsAccount: signalfx
        defaultMetricsStore: signalfx
        defaultJudge: NetflixACAJudge-v1.0
        stagesEnabled: true
        templatesEnabled: true
        showAllConfigsEnabled: true
        serviceIntegrations:
        - name: signalfx
          enabled: true
          accounts:
          - name: signalfx
            accessToken: encrypted:k8s!n:spin-secrets!k:signalfx_access_token
            endpoint:
              baseUrl: https://stream.us1.signalfx.com
            defaultScopeKey: server_group
            defaultLocationKey: server_region
            supportedTypes:
            - METRICS_STORE
        - name: google
          enabled: true
          accounts:
          - name: google
            project: my-project
            bucket: my-bucket
            jsonPath: encryptedFile:k8s!n:spin-secrets!k:gcp-sa.json
            rootFolder: kayenta
            supportedTypes:
            - CONFIGURATION_STORE
            - OBJECT_STORE
          gcsEnabled: true
          stackdriverEnabled: false

Using Operation Version 1.2.2 and using armory/halyard:operator-0ec2c8a .

It gives the following error:

{"level":"error","ts":1610584826.933298,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"spinnakerservice-controller","request":"spinnaker/spinnaker","error":"\"encryptedFile...\" specified for a non file property (encryptedFile:k8s!n:spin-secrets!k:gcp-sa.json), should be \"encrypted...\" instead","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/opt/spinnaker-operator/build/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/spinnaker-operator/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/spinnaker-operator/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/opt/spinnaker-operator/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/opt/spinnaker-operator/build/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/opt/spinnaker-operator/build/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/opt/spinnaker-operator/build/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"

@nasonfish
Copy link
Contributor

Did you ever find a workaround for this @EIrwin ?

@dogonthehorizon
Copy link
Contributor

@nasonfish are you asking because you're seeing this behavior as well?

@DmitrySolodovnyk
Copy link

DmitrySolodovnyk commented Jun 16, 2023 •
edited
Loading

I have similar behavior but with Google secret manager in my case:
...
gitrepo:
enabled: true
accounts:
- name: bitbucket-tools-tsimagine-com
username: spinnaker
sshPrivateKeyFilePath: encryptedFile:google-secrets-manager!p:11111111111111!s:spinnaker-rsa-key
...
{"level":"error","ts":1686924229.0962656,"logger":"spinvalidate","msg":"\nSpinnakerService validation failed:\nHalyard validator detected an error:\n Error decrypting secrets in config:\n Error decrypting secret for value 'encryptedFile:google-secrets-manager!p:11111111111111!s:spinnaker-rsa-key':\n Error creating decrypter for value 'encryptedFile:google-secrets-manager!p:11111111111111!s:spinnaker-rsa-key':\n secret engine google-secrets-manager not registered\n"

@DmitrySolodovnyk
Copy link

spec:
validation:
failOnError: false

  • doesn't help.

@michael-epperson
Copy link

michael-epperson commented Nov 3, 2023 •
edited
Loading

Having the same issue as OP. Using spinnaker-operator 1.3.1 in hosted in aws eks. I have the json credentials loaded as a k8s secret file and am loading it with jsonPath: encryptedFile:k8s!n:!k:.


java.lang.NullPointerException: null
        at com.netflix.spinnaker.halyard.config.model.v1.node.Validator.validatingFileDecryptBytes(Validator.java:52) ~[halyard-config-operator-a6ac1d4.jar:na]
        at com.netflix.spinnaker.halyard.config.model.v1.node.Validator.validatingFileDecrypt(Validator.java:43) ~[halyard-config-operator-a6ac1d4.jar:na]
        at com.netflix.spinnaker.halyard.config.validate.v1.providers.google.GoogleAccountValidator.getJsonKey(GoogleAccountValidator.java:83) ~[halyard-config-operator-a6ac1d4.jar:na]
        at com.netflix.spinnaker.halyard.config.validate.v1.providers.google.GoogleAccountValidator.validate(GoogleAccountValidator.java:54) ~[halyard-config-operator-a6ac1d4.jar:na]
        at com.netflix.spinnaker.halyard.config.validate.v1.providers.google.GoogleProviderValidator.lambda$validate$0(GoogleProviderValidator.java:39) ~[halyard-config-operator-a6ac1d4.jar:na]
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) ~[na:na]
        at com.netflix.spinnaker.halyard.config.validate.v1.providers.google.GoogleProviderValidator.validate(GoogleProviderValidator.java:39) ~[halyard-config-operator-a6ac1d4.jar:na]
        at com.netflix.spinnaker.halyard.config.validate.v1.providers.google.GoogleProviderValidator.validate(GoogleProviderValidator.java:28) ~[halyard-config-operator-a6ac1d4.jar:na]
        at com.netflix.spinnaker.halyard.deploy.services.v1.ValidationRun$NodeValidator.validate(ValidationRun.java:109) ~[halyard-deploy-operator-a6ac1d4.jar:na]
        at com.netflix.spinnaker.halyard.deploy.services.v1.ValidationRun.validateNode(ValidationRun.java:69) ~[halyard-deploy-operator-a6ac1d4.jar:na]
        at com.netflix.spinnaker.halyard.deploy.services.v1.ValidationRun.visitNode(ValidationRun.java:49) ~[halyard-deploy-operator-a6ac1d4.jar:na]
        at com.netflix.spinnaker.halyard.deploy.services.v1.ValidationRun.visitNode(ValidationRun.java:60) ~[halyard-deploy-operator-a6ac1d4.jar:na]
        at com.netflix.spinnaker.halyard.deploy.services.v1.ValidationRun.visitNode(ValidationRun.java:60) ~[halyard-deploy-operator-a6ac1d4.jar:na]
        at com.netflix.spinnaker.halyard.deploy.services.v1.ValidationRun.run(ValidationRun.java:44) ~[halyard-deploy-operator-a6ac1d4.jar:na]
        at com.netflix.spinnaker.halyard.deploy.services.v1.DynamicValidationService.validate(DynamicValidationService.java:62) ~[halyard-deploy-operator-a6ac1d4.jar:na]
        at com.netflix.spinnaker.halyard.controllers.v1.ValidationController.validateConfig(ValidationController.java:42) ~[halyard-web-operator-a6ac1d4.jar:na]
        at jdk.internal.reflect.GeneratedMethodAccessor1058.invoke(Unknown Source) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-5.3.13.jar:5.3.13]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-5.3.13.jar:5.3.13]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[spring-webmvc-5.3.13.jar:5.3.13]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.13.jar:5.3.13]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.13.jar:5.3.13]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.13.jar:5.3.13]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) ~[spring-webmvc-5.3.13.jar:5.3.13]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) ~[spring-webmvc-5.3.13.jar:5.3.13]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.13.jar:5.3.13]
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.3.13.jar:5.3.13]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:652) ~[tomcat-embed-core-9.0.41.jar:4.0.FR]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.13.jar:5.3.13]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) ~[tomcat-embed-core-9.0.41.jar:4.0.FR]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.62.jar:9.0.62]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.13.jar:5.3.13]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.13.jar:5.3.13]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183) ~[spring-security-web-5.4.9.jar:5.4.9]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) ~[spring-web-5.3.13.jar:5.3.13]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) ~[spring-web-5.3.13.jar:5.3.13]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.13.jar:5.3.13]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.13.jar:5.3.13]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.13.jar:5.3.13]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.13.jar:5.3.13]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:97) ~[spring-boot-actuator-2.4.13.jar:2.4.13]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.13.jar:5.3.13]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.13.jar:5.3.13]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.13.jar:5.3.13]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:764) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:888) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1597) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
        at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]```

michael-epperson added a commit to michael-epperson/halyard that referenced this issue Nov 3, 2023
@michael-epperson
Update FileService.java
949e29a 
As a potential fix for armory/spinnaker-operator#203
This fixes the getFileContentBytes to check for an encryptedFile rather than an encryptedSecret
@michael-epperson michael-epperson mentioned this issue Nov 3, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@DmitrySolodovnyk @dogonthehorizon @nasonfish @EIrwin @michael-epperson