Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ckd8s-cdktf-resolver: can't resolve terraform values that are nested in a helm chart #2364

Open
brent-at-aam opened this issue Jan 31, 2025 · 0 comments
Labels
bug Something isn't working needs-triage Priority and effort undetermined yet

Comments

@brent-at-aam
Copy link

Description of the bug:

This is kind of an odd one, but we are using cdk8s to synth dagster-user-deployment helm charts and running into an issue.

This particular chart allows you to pass env vars and configMaps/secrets to be referenced in the env and envFrom properties of ContainerSpec. There is also a toggle to pass this down to child k8s jobs that the dagster deployment creates. If that's enabled, it will take relevant entries from your chart values, convert them to json, and set it as a value for another env var.

 - name: DAGSTER_CONTAINER_CONTEXT
   value: {{ include "dagsterUserDeployments.k8sContainerContext" (list $ $deployment) | fromYaml | toJson | quote }}

If I pass a terraform output ref to an env var, for example an S3 bucket name. The chart synth fails with a message like:

# truncated for brevity
Error: Failed serializing construct at path '$PATH' with name '$NAME':
Error: Unable to find output defined for
{
  "k8s":{
    "env":[{"name":"S3_BUCKET","value":"${TfToken[TOKEN.40]}"},...],
    ...
  }
} (Inspected stacks: $STACK_0, $STACK_1) 

My assumption is that since the value is now just a json string, the ResovlerContext is getting the entire json string and can't match the token properly here

Reproduction Steps:

The dagster chart is not importable as it's getting caught in this bug cdklabs/json2jsii#1195, so we just use the Helm class directly:

from cdk8s import App as K8sApp
from cdk8s import Chart, Helm
from cdk8s_cdktf_resolver import CdktfResolver
from cdktf import App as TfApp
from cdktf import TerraformOutput, TerraformStack
from cdktf_cdktf_provider_aws.provider import AwsProvider
from cdktf_cdktf_provider_aws.s3_bucket import S3Bucket
from constructs import Construct


class ExampleStack(TerraformStack):
    bucket: TerraformOutput

    def __init__(self, scope: Construct, stack_id: str):
        super().__init__(scope, stack_id)
        AwsProvider(self, "aws", region="us-east-1")
        bucket = S3Bucket(self, "bucket", bucket="foo")
        self.bucket = TerraformOutput(self, "bucket_output", value=bucket.bucket)


class ExampleChart(Chart):
    def __init__(self, scope: Construct, chart_id: str, tf: ExampleStrack):
        super().__init__(scope, chart_id)
        Helm(
            self,
            "deployment",
            chart="dagster-user-deployments",
            repo="https://dagster-io.github.io/helm",
            version="1.9.6",
            namespace="dagster",
            release_name="foo-bar",
            values={
                "deployments": [
                    {
                        "name": "foo-bar",
                        "env":  [
                            { "name": "S3_BUCKET", "value": tf.bucket.value }
                        ],
                        "includeConfigInLaunchedRuns": { "enabled": True },
                        "port": 3030
                    }
                ],
                "serviceAccount": {"create": True, "name": "foo-bar" }
            }
        )


tf_app = TfApp()
stack = ExampleStack(app, "tf")
k8s_app = K8sApp(resolvers=[CdktfResolver(app=tf_app)]))
ExampleChart(k8ss_app, "k8s", stack)

k8s_app.synth()

Error Log:

Environment:

  • Framework Version: 0.0.138
  • OS: ubuntu 22.04

This is 🐛 Bug Report

@brent-at-aam brent-at-aam added bug Something isn't working needs-triage Priority and effort undetermined yet labels Jan 31, 2025
@brent-at-aam brent-at-aam changed the title ckd8s-cdktf-resolver: can't find resolve terraform values that are nested in a helm chart ckd8s-cdktf-resolver: can't resolve terraform values that are nested in a helm chart Jan 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs-triage Priority and effort undetermined yet
Projects
None yet
Development

No branches or pull requests

1 participant