Elasticsearch deployment automation does not work anymore

[thesse1]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

Terraform v1.6.4

AzureRM Provider Version

3.82.0

Affected Resource(s)/Data Source(s)

azurerm_elastic_cloud_elasticsearch

Terraform Configuration Files

provider "azurerm" {
  features {}
}

resource "azurerm_elastic_cloud_elasticsearch" "default" {
  name                        = "test-elasticsearch"
  resource_group_name         = "azure-demo-01-monitoring-rg"
  location                    = "westeurope"
  sku_name                    = "ess-consumption-2024_Monthly@TIDgmz7xq9ge3py"
  elastic_cloud_email_address = "[email protected]"

  logs {
    send_activity_logs = false
    send_subscription_logs = false
  }
}

Debug Output/Panic Output

https://gist.github.com/thesse1/82c1f72148844bc934a3c3464bfda5df

Expected Behaviour

It should create Azure Managed Elasticsearch

Actual Behaviour

It fails with error message:

  | creating Monitor (Subscription: "309065ca-a060-4592-8096-b74694126b61"
  | Resource Group Name: "azure-demo-01-monitoring-rg"
  | Monitor Name: "test-elasticsearch"): performing MonitorsCreate: unexpected status 400 with error: BadRequest: Cannot proceed with the request as the user is not authorized

Steps to Reproduce

terraform apply

I have been using Terraform for setting up my Azure infrastructure including an instance of Managed Elasticsearch for several months, and it has always been working fine until yesterday (08.12.23) morning CET. It failed for the first time yesterday (08.12.23) afternoon, and it has failed ever since.

Please find the Terraform configuration of a repro case in the attachment.

Since yesterday afternoon, it would always complain:

│ Error: creating Monitor (Subscription: "309065ca-a060-4592-8096-b74694126b61"
│ Resource Group Name: "azure-demo-01-monitoring-rg"
│ Monitor Name: "azure-demo-01-elasticsearch"): performing MonitorsCreate: unexpected status 400 with error: BadRequest: Cannot proceed with the request as the user is not authorized
│
│   with module.monitoring.azurerm_elastic_cloud_elasticsearch.default,
│   on monitoring\monitoring-elasticsearch.tf line 1, in resource "azurerm_elastic_cloud_elasticsearch" "default":
│    1: resource "azurerm_elastic_cloud_elasticsearch" "default" {

Yes, I am the owner of the resource group, and Terraform is working fine for dozens of other resources. Yes, I can log-in to Elastic Cloud with my Microsoft account [email protected]. Yes, I can create an Elasticsearch instance in the same RG with the same resource configuration with the same user in Azure portal.

I have tried the Terraform script with location westeurope, eastus and southeastasia. Same result.

I have tried creating the resource using Azure CLI:

az elastic monitor create -n test-elasticsearch -g azure-demo-01-monitoring-rg --user-info "{firstName:Thomas,lastName:Hesse,companyName:'Software AG',emailAddress:[email protected]}" --sku "{name:ess-consumption-2024_Monthly@TIDgmz7xq9ge3py}"

Result:

(BadRequest) Cannot proceed with the request as the user is not authorized
Code: BadRequest
Message: Cannot proceed with the request as the user is not authorized

I have exported an ARM template in the Azure Portal, cf. attachment. I can create the resource using the template in the Azure Portal, but it fails when I try the following:

az deployment group create --resource-group azure-demo-01-monitoring-rg --template-file ExportedTemplate-azure-demo-01-elasticsearch.json --parameters @ExportedTemplate-azure-demo-01-elasticsearch-parameters.json

Result:

{"status":"Failed","error":{"code":"DeploymentFailed","target":"/subscriptions/309065ca-a060-4592-8096-b74694126b61/resourceGroups/azure-demo-01-monitoring-rg/providers/Microsoft.Resources/deployments/ExportedTemplate-azure-demo-01-elasticsearch","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","details":[{"code":"BadRequest","message":"Cannot proceed with the request as the user is not authorized"}]}}

I have even tried calling the Azure Management API:

PUT https://management.azure.com/subscriptions/309065ca-a060-4592-8096-b74694126b61/resourceGroups/azure-demo-01-monitoring-rg/providers/Microsoft.Elastic/monitors/test-elasticsearch?api-version=2023-02-01-preview
{"location":"westeurope","properties":{"monitoringStatus":"Enabled","userInfo":{"emailAddress":"[email protected]"}},"sku":{"name":"ess-consumption-2024_Monthly@TIDgmz7xq9ge3py"},"tags":{}}

Result:

Status 400
{
    "error": {
        "code": "BadRequest",
        "message": "Cannot proceed with the request as the user is not authorized"
    }
}

I have tried multiple versions of the API.

Yes, I am using a valid token, the API is working fine for other resource types.

Currently I see no way of setting up the resource automatically. Please help!

Best regards,
Thomas

Important Factoids

No response

References

No response

[thesse1]

Some more results of my analysis:

• Terraform is working fine when I run it in an Azure DevOps pipeline. It only fails when I run it locally.
• az elastic monitor list is working fine. Only az elastic monitor create fails when I run it locally.
• az elastic monitor create is working fine when I run it in a Cloud Shell in Azure Portal, but not on my PC.
• Creating a token in Cloud Shell with az account get-access-token and using it in Postman on my PC is working fine, but not with a token created with az account get-access-token on my PC. When I compare the tokens, I see a difference in the unique_name claim. It is [email protected] for the token created in Cloud Shell and [email protected] for the token created locally, cf. attachments.

Maybe this is causing the issue? Please note that my Elastic Cloud username is [email protected]. Are you taking the Elastic Cloud username from the unique_name claim of the token and not from the userInfo.emailAddress of the request body? Or are you forwarding the token to Elastic Cloud? (I hope not…)

Anyhow: How can I log-in to Azure CLI in such a way that Terraform and az elastic monitor create are working locally again?

JWT_Payload_Local.json
JWT_Payload_Cloud_Shell.json

[sinbai]

Hi @thesse1 thanks for opening this issue. Per the description above, I assume that this is not a terraform issue. Since Terraform manages Azure resources by calling Azure Rest API. Terraform create resource azurerm_elastic_cloud_elasticsearch by calling this API. So, I recommend you open an issue in this API repo to get help.

[thesse1]

Hi @sinbai thanks for the hint. I have just opened a new issue here: Azure/azure-rest-api-specs#27048.