Dynamic variables

[mattcasters]

So, what if we had a way to dynamically resolve variables. That was the main theme of the discussion under issue #3454.
The idea is simple enough but perhaps hard to explain in text form and so it got snowed under for a while.

Well, suppose you would be able to build plugins to resolve variables with metadata.
We could introduce a new syntax: #{resolver-name:variable}.

For example: #{pipeline:hop} with

• pipeline : being the name of the Variable Resolver metadata object
• hop : the name of the variable or expression you would like to see resolved.

The pipeline variable resolver would look something like this:

You could then make a pipeline to resolve a variable value. Here is an example with static data...

The pipeline gets the expression (or variable name) via the specified variable and looks up the actual value. In the simple scenario above we're using hardcoded value. You could also store the names and values in a database table or a file somewhere.
Other use cases would include the lookup of JSON Web Tokens, interactions with secret vaults and so on.

If you want to try this out, build pull request 4761

What resolver plugins or features did you think of? Let me know!

Cheers,

Matt

[dave-csc]

Hi @mattcasters,

an idea would the decryption/deobfuscations of variable with encrypted passwords. I actually have already implemented it as a "mapping" transform (check this post for rough details)

[mattcasters]

So yes, people find interesting ways to resolve the variables needed in a project.
Given the recursive nature of variable resolution, you could think of the different environments:

• Development: ${PASSWORD} resolves to value abcd in a config file, for a MySQL database on your laptop.
• Production: ${PASSWORD} resolves to #{AzureVault:MySQLPassword} (in a config file) which then uses an Azure Key Vault variable resolver to look up the password to a MySQL database running in the cloud somewhere.

Another obvious example are JSON Web Tokens where you first have to get a JWT from a webservice to be able to re-use that in the rest of a pipeline or workflow. It's possible to do this in a workflow but it's cleaner if we could build a JWT Variable Resolver for this of some kind.
What I tried to do now is start with the Pipeline Variable Resolver since it allows for many of the same things as a PoC.

These variable resolver plugins can be as simple as a single class file in the various plugin sections to get them to work, plus some library dependencies. I'll try to add a few other examples to the PR.

Cheers!

[nadment]

I love the idea of the JWT Variable resolver.
We could also imagine a Secret Variable Resolver ${SECRET:PROVIDER:KEY}.
Supports two providers: CONFIG and NEGOTIATE.
The CONFIG provider requires a local vault, while the NEGOTIATE provider will automatically try to retrieve credentials.

[mattcasters]

Absolutely @nadment. For now I've taken the #{name:value} format, but of-course the value can contain additional colons that you could work with, depending on what you want to achieve. Extra arguments wouldn't be an issue. Personally I would put as many configuration options as I can in the metadata but we can't argue over preferences ;-)

In any case, my opinion is that the value of the secret (after lookup) is stored only in the transform or action when needed, not in the whole pipeline or workflow. As such it feels more secure this way.

[hansva]

Great work! I agree that the value should only be on transform or action level, if needed you could add a cache to the provider but I wouldn't add the resolved item to the variables

[mattcasters]

Well, you could obviously define another variable with the secret value you get, but I would advise against it in our documentation. Who knows, maybe someone wants to synchronize or migrate secrets between 2 vaults or something like that. There might be actual legit use cases darnit.

[fpapon]

May be using placeholders with default value can be nice for example like:
{{my.arg:-defaultValue}}

[mattcasters]

Parameters, a special case of explitly defined variables in pipelines and workflows, can have default values.
But perhaps we can have a Default Value Variable Resolver plugin where you can simply define default values for variables. You'd get a similar effect.

[fpapon]

sounds good

[mattcasters]

I tried to build a quick secret lookup plugin for GCP Secret Manager and after some library shenanigans...

I'll try to spend some time over the weekend to give a number of Google libraries and Beam an update.
But anyway:

@Getter
@Setter
@GuiPlugin
@VariableResolverPlugin(
    id = "Variable-Resolver-GoogleSecretManager",
    name = "Google Secret Manager Variable Resolver",
    description = "Automatically look up values of secrets in Google Secret Manager",
    documentationUrl =
        "/variables/resolvers/google-secret-manager.html" // TODO: write this documentation
    )
public class GooleSecretManagerVariableResolver implements IVariableResolver {

  /** The name of the variable that will contain the expression in the pipeline. */
  @GuiWidgetElement(
      id = "projectId",
      order = "0!",
      label =
          "i18n:org.apache.hop.core.variables.resolver:GooleSecretManagerVariableResolver.label.ProjectId",
      type = GuiElementType.TEXT,
      parentId = VariableResolver.GUI_PLUGIN_ELEMENT_PARENT_ID)
  @HopMetadataProperty
  private String projectId;

  /** The name of the variable that will contain the expression in the pipeline. */
  @GuiWidgetElement(
      id = "locationId",
      order = "02",
      label =
          "i18n:org.apache.hop.core.variables.resolver:GooleSecretManagerVariableResolver.label.LocationId",
      type = GuiElementType.TEXT,
      parentId = VariableResolver.GUI_PLUGIN_ELEMENT_PARENT_ID)
  @HopMetadataProperty
  private String locationId;

  @Override
  public void init() {
    // Not used at this time.  If performance is too bad for the client construction we can still do
    // it.
    // For now we assume that it's all just using web services anyway.
  }

  @Override
  public String resolve(String secretId, IVariables variables) throws HopException {

    if (StringUtils.isEmpty(secretId)) {
      return secretId;
    }

    try {
      SecretManagerServiceSettings.Builder settingsBuilder =
          SecretManagerServiceSettings.newBuilder();
      if (StringUtils.isNotEmpty(locationId)) {
        String apiEndpoint = String.format("secretmanager.%s.rep.googleapis.com:443", locationId);
        settingsBuilder.setEndpoint(apiEndpoint);
      }
      SecretManagerServiceSettings settings = settingsBuilder.build();

      try (SecretManagerServiceClient client = SecretManagerServiceClient.create(settings)) {

        String actualLocationId = variables.resolve(locationId);
        String actualProjectId = variables.resolve(projectId);

        // Get the secret version name.
        //
        SecretVersionName secretName;
        if (StringUtils.isEmpty(actualLocationId)) {
          secretName =
              SecretVersionName.ofProjectSecretSecretVersionName(
                  actualProjectId, secretId, "latest");
        } else {
          secretName =
              SecretVersionName.ofProjectLocationSecretSecretVersionName(
                  actualProjectId, actualLocationId, secretId, "latest");
        }

        // Get the payload for this version
        //
        AccessSecretVersionResponse response = client.accessSecretVersion(secretName);

        // Create the secret.
        return response.getPayload().getData().toStringUtf8();
      }
    } catch (Exception e) {
      throw new HopException(
          "Error looking up secret key '" + secretId + "' in Google Secret Manager", e);
    }
  }

  @Override
  public void setPluginId() {
    // Nothing to set
  }

  @Override
  public String getPluginId() {
    return "Variable-Resolver-GoogleSecretManager";
  }

  @Override
  public void setPluginName(String pluginName) {
    // Nothing to set
  }

  @Override
  public String getPluginName() {
    return "Google Secret Manager Variable Resolver";
  }
}