Best approach for Data Fix for missing/incorrect tagging information

[TJohnson93]

Probably a dumb question, I'm currently testing out this solution via the Power BI CM Connector (primarily to test out), I am in the process of setting up resource groups and BICEP deployment for FinOps Hub as I believe this product is the right approach forward in my organisation.

As many, I'm sure, have, I find myself in a fairly poor tagging environment in Azure (now fixed for anything new moving forward) and am missing many tags or incorrect tagging for existing resources since creation.

What would be the recommended approaches to apply a 'data-fix' to add missing tags and update those incorrect or inconsistent over roughly 4 years and a few thousand resources total in current state?

In testing, I've done find and replace in PBI model or reference data sets to solve many inconsistencies, just not set on an approach to solve the missing tags situation.

[arthurclares]

@TJohnson93 We don't have any solution in the Toolkit to help with tagging issues, but that is a good suggestion. However, @arthursilvany, has developed a PowerShell script that might help reconcile these missing tags: https://github.com/arthursilvany/ReconcileTagNames
@KevDLR @CaddyG Something else to be considered as part of #777

[TJohnson93]

This looks great from a tag key aspect and will have a closer look in the work environment this week. Need to look closer to see if it applies retrospectively. Cheers mate

[flanakin]

@TJohnson93 can you share some examples of the tagging issues you're facing? If we have some details, we might be able to come up with some solutions that we can add to the backlog. Specifically, we've talked about adding allocation to FinOps hubs, which would likely grow to include some tag cleanup. But in the short-term, this is something we could add examples of in the Power BI reports as a starting point.

[TJohnson93]

I'll try and break it up into a few use cases that I've come across so far.

Use Case #1: Resources with missing tags

Some key tags (typically mandatory) don't exist for resources like 'CostCentre', 'BusinessOwner', 'Application'. These just don't exist as a key:value pair for a group of resources

Use Case #2: Resources have a key tag but missing value

Resources have the key for tag. Example "CostCentre' but no value for tag or null.

Use Case #3: Resource key is inconsistent

Resources have a mandatory tag but the key was never validated and enforced through policy resulting in variations of tag keys. Example CostCentre, costCentre, cost centre, costcentre.

Use Case #4: Inconsistent tag values

Not consistent for naming of tag values, example for a department tag could be IT, Info Tech, Information Technology.

---

Obviously for each of these we can update the tags which then come through but not retrospectively.

[flanakin]

The best option for each of this is really to create tag policies with Azure Policy. But as you mentioned, that doesn't fix historical data. I think the best short-term solution is to patch the data in Power BI queries. I'm not going to say it's "easy" however. You'll need to think thru how to know what values you want to apply and create the right logic to replace the missing values using Table.ReplaceValues() or similar function. There are a number of options.

If you want to flesh out the requirements, we'd welcome a feature request. Some of these are easier than others, but there's some base-level infrastructure needed to make this all work that we could definitely add to a release.

In the meantime, if you're working with a Microsoft account team, it would be helpful to ask for a Data & AI CSA to help.

[estimpfle]

You don't' do the tags in FINOPS you do the tags in Azure. You want the tags to represent what is in Azure not what you add in later.
If you need to add in custom tags you could also do that by resource group and just create some type of Lookup sheet.