Binding together ARM/Bicep deployment outputs and inputs in a GitOps way

[Xitric]

First of all, thanks for working on bringing GitOps principles to Azure - it is incredibly valuable!

Context

At my company, we are looking into adopting AzOps for declaratively managing our landing zones in Azure, encompassing everything from management groups, subscriptions, and custom role definitions, down to central container registries, key vaults, service principals, etc.

This involves a great deal of Bicep templates that we deploy at various scopes. Some of these deployments generate outputs that we rely on as inputs for deploying yet other Bicep templates to our landing zone.

Currently, we manage the sequencing of these deployments imperatively using pipelines. We developed a simple system for querying successful Azure deployments, looking up their outputs, and generating the parameter inputs for other deployments on the go. As a result, we have very few parameter files in git, and the few we have typically only make up a subset of the parameters required for any given Bicep template. Most parameters are queried right before deploying, and we never serialize them anywhere.

Let me give a concrete example: We have one Bicep template for expressing a set of custom role definitions. This template is deployed to a central identity subscription. Another Bicep template is deployed which needs to use one of these custom role definitions, so we need to know its name. Rather than manually looking it up in our otherwise hands-off environment and typing it into a parameter file (and remembering when to change it), we could simply query an output from the custom role definition deployment. By always referencing the latest successful deployment of custom role definitions, we know the role name will always be correct.

We have knowledge shared with other Azure clients who take a similar approach to ours. Some store their deployment outputs in table storage rather than relying on Azure deployment resources. But the core of the solution is still the same: Looking up deployment outputs when generating inputs for other deployments.

We also know of similar practices in both Crossplane and Azure Service Operator (which deal with declarative deployments of Azure resources in a similar fashion as AzOps), where deployment outputs are stores intermediately in ConfigMaps allowing for yet other deployments to depend on and query those outputs (https://azure.github.io/azure-service-operator/guide/configmaps/).

Question

We wonder how we might change things up to make our deployment strategy compatible with a declarative GitOps approach in AzOps. We really would like to avoid having to statically serialize deployment outputs into parameter files in git for a couple of reasons:

1. We can't rely on all outputs to be deterministic. For instance, the client ID of a service principal is generated upon creation, and recreating the service principal later would generate a new client ID.
2. We consider a major benefit of AzOps (or any GitOps approach) to be the ability to quickly and automatically bring an Azure environment back to its desired state if any portions ever get lost (could be disaster recovery or just an accidental deletion/modification of resources). Given point number 1, this could mean that outputs of one Bicep deployment now change, making the parameter files for other Bicep deployments in git potentially invalid. This could require many iterations of deploying and updating parameter files before the system is consistent again.

Specifically point 2 concerns us. If we deploy everything via a single push pipeline, we are locked into the parameter files from the commit at which the pipeline was initiated. But these files could become out of sync during pipeline execution, causing deployments to fail and requiring reruns. That seems inefficient. At the same time, we also acknowledge the value of having everything serialized in git with no external dependencies.

We know that by default AzOps is designed around only deploying the resources that have been modified via a merged pull request, but the ability to simply reconcile an Azure environment from the ground up in case of disasters is why we are considering AzOps in the first place.

Epilogue

So really, we would love to know how others tackle this, how we might change our practices, or if somehow AzOps can accommodate our current approach! We are always open to adapting our ways. At the same time, we are new to AzOps, and I believe a lot of our confusion stems from not understanding it well enough.

Looking forward to hearing your inputs!

[Xitric]

@daltondhcp I was wondering if there is any forum where we can discuss AzOps with maintainers or other end-users to ensure that we understand it properly and use it correctly, because I fear that we may be approaching our adoption of AzOps from the wrong angle. But really we would like some insights into AzOps best practices and how we can make it work for our use cases.

[Xitric]

@daltondhcp Sorry for pinging you again, but we are really still quite interested in discussing this topic since we can see more and more use cases for AzOps in our daily operations, yet we also feel that we are lacking some flexibility in terms of parameterizing our Bicep modules for it to truly be beneficial.

[daltondhcp]

Hey @Xitric, Please reach out to me at jodahlbo[at]microsoft.com and we can certainly set up a teams call to further discuss this, and then potentially update documentation as needed.