[Hubs] FinOps Hubs - Ingestion container is empty

[NavidKhan8]

Encountered issues with both connector solutions:

Issue1
Using Cost management app with EA account, after processing almost 20 GB data it failed with below error:

CostDetails
Failed to save modifications to the server. Error returned: 'There's not enough memory to complete this operation. Please try again later when there may be more memory available. '.

Issue2
Able to spin up the FinOps Hubs resources and exported cost management report at MG scope to the hub created storage account under msexports.

ADF pipeline ran automatically and it generated 2 .parquet files under the same storage account but under msexports/amortizedcost container

Checked the ADF pipeline all the tasks ran successfully.

Tried manually placing these generated .parquet files under ingestion container and opened Focus PowerBI report by updating connector to hub & storage account URL updated with access key but receiving below error

"We cannot convert value null to type Table"

Any pointer will be helpful?

[NavidKhan8]

@flanakin Also to add, after setting the finops hub, after downloading around 2Gb data it returned below error with out of memory error:

CostDetails
Failed to save modifications to the server. Error returned: 'There's not enough memory to complete this operation. Please try again later when there may be more memory available. '.
Query parameters
Storage URL
Failed to save modifications to the server. Error returned: 'There's not enough memory to complete this operation. Please try again later when there may be more memory available. '.
RangeStart
Failed to save modifications to the server. Error returned: 'There's not enough memory to complete this operation. Please try again later when there may be more memory available. '.
RangeEnd
Failed to save modifications to the server. Error returned: 'There's not enough memory to complete this operation. Please try again later when there may be more memory available. '.

[flanakin]

Failed to save modifications to the server. Error returned: 'There's not enough memory to complete this operation. Please try again later when there may be more memory available.'

What are you doing when you get this error? Are you refreshing the data, publishing from PBI Desktop to the service, refreshing the data already published to the PBI service, or something else?

@flanakin Also to add, after setting the finops hub, after downloading around 2Gb data it returned below error with out of memory error

Are you using Power BI Pro by any chance? There are known limits to Power BI Pro. I don't know the exact error message or what the limit is, but that's something to check.

@MSBrett, have you seen this before?

ADF pipeline ran automatically and it generated 2 .parquet files under the same storage account but under msexports/amortizedcost container

This is strange. I haven't heard of this happening before 🤔 What version are you using? The only thing I can think of that might be quirky is the use of management groups. Cost Management exports are very different for management groups. I'm not aware of that causing what you're seeing, tho.

Sorry for the slow response. I was out last week and am still playing catch-up...

[NavidKhan8]

@flanakin: No worries! And thank you for the reply.

Seems like directly fetching the data using cost management app may not work because the data for our EA account is huge.
When I get this error there is no data and cannot save with error as "cannot transform null to table"

I'm trying this with desktop app at the moment and when it works fine then planning to publish it to the workspace which is premium.

So I've tried finopshub option and after further debugging I was able to make ADF pipeline generate .parquet files at target ingestion container(it contains nearly 2GB of data) but here also facing the issue of out of memory error.

When I try to get data using direct data lake storage in PowerBI able to fetch the data but when I try the same using CostSummary or Focus reports getting memory errors.

Any pointers on fixing these issues?

Since I'm able to fetch the data directly from the generated .parquet data(2GB) with new PowerBI report but couldn't figure out on how to add those PowerBI functions for data transformation? Is it possible for you to share the modified PowerBI function when data is already fetched?

[flanakin]

@NavidKhan8, can you try editing the CostDetails query in Power BI and updating it to ftk_LoadData(1) (1 instead of 2)? We load actual cost data from the Cost Management connector by default, which is likely causing this memory issue. That's what 2 means. This will go away in FinOps hubs 0.2, which should be out by EOM.

[NavidKhan8]

@flanakin Tried updating the function by passing it 1 instead of 2 but still got the same error after download almost 1.8 GB of data from storage account

Below is the error message:

CostDetails
Failed to save modifications to the server. Error returned: 'Memory error: Memory Allocation failure . Try simplifying or reducing the number of queries. There's not enough memory to complete this operation. Please try again later when there may be more memory available. '.
Query parameters
Billing Account ID
Failed to save modifications to the server. Error returned: 'Memory error: Memory Allocation failure . Try simplifying or reducing the number of queries. There's not enough memory to complete this operation. Please try again later when there may be more memory available. '.
Number of Months
Failed to save modifications to the server. Error returned: 'Memory error: Memory Allocation failure . Try simplifying or reducing the number of queries. There's not enough memory to complete this operation. Please try again later when there may be more memory available. '.
Storage URL
Failed to save modifications to the server. Error returned: 'Memory error: Memory Allocation failure . Try simplifying or reducing the number of queries. There's not enough memory to complete this operation. Please try again later when there may be more memory available. '.
RangeEnd
Failed to save modifications to the server. Error returned: 'Memory error: Memory Allocation failure . Try simplifying or reducing the number of queries. There's not enough memory to complete this operation. Please try again later when there may be more memory available. '.

[NavidKhan8]

@flanakin - As I'm able to get the data directly from the storage account(ADF generated .parquet) to PowerBI, is it possible for you to share the steps needed for data transformation from here instead of ftk_LoadData(1) or ftk_LoadData(2)?

[flanakin]

@NavidKhan8 Can you email me at micflan@ms? Let's find some time to walk thru this together.

[NavidKhan8]

@NavidKhan8 Can you email me at micflan@ms? Let's find some time to walk thru this together.

Decoded the domain and sent an email.

[flanakin]

@allcontributors please add @NavidKhan8 for bug

[allcontributors]

@flanakin

I've put up a pull request to add @NavidKhan8! 🎉

[flanakin]

I'm working with Navid to update to FinOps hubs 0.2 from the latest release @ https://aka.ms/ftk/latest.

Will move this to 0.3 while we confirm the fix.

[flanakin]

Update: We're currently validating FinOps hubs 0.2.1-rc2 to address this.