Terraform apply fails with clientCredentialsToken already expired. #201
Comments
|
Hi. Are you able to provide more context of your pipeline? Is there an |
|
@jaredfholgate Thank you for quick reply. Terraform init, successfully getting the keys. Given init was succesful, do I still need |
|
Hi. I'm not suggesting that is what you must always do to use WIF, but thought it might help in your context given the error message you got. It could be due to your storage account perms, computer clock or something else. |
|
Could be related to #89 (comment) If you are using workload identity and waiting more than an hour between plan and apply then it fails because the token is stored in the tfplan file |
|
@jaredfholgate Thanks for the suggestion. Not sure if this lies in the actual terraform or the pipeline implementation. After few tries, it looks like the auth process fails if there's nothing to be applied. |
Are there any workarounds for this? We'd like to introduce approval step between creating terraform plan and applying it, but everytime there's more than like 10 minutes (not an hour) between plan and apply, the token stored within the plan is already expired. There seems to be no way to force using fresh token instead the one stored within the plan. |
|
I just switched to the Azure CLI task to be honest - task: AzureCLI@2
displayName: terraform plan
inputs:
azureSubscription: ${{ parameters.service_connections.azure }}
addSpnToEnvironment: true
scriptType: pscore
scriptLocation: inlineScript
inlineScript: |
$env:ARM_USE_AZUREAD = 'true'
$env:ARM_SUBSCRIPTION_ID = & az account show --query id --output tsv
$env:ARM_TENANT_ID = $env:tenantId
$env:ARM_CLIENT_ID = $env:servicePrincipalId
$env:ARM_USE_OIDC = 'true'
$env:ARM_OIDC_TOKEN = $env:idToken
& terraform plan -out plan.tfplan |
How does that solve the problem? |
|
Then you can configure the backend using environment variables, which avoids the time limited token being stored in the plan file. |
|
I'm running into the same issue.
I don't want to manage my service connections by hand; that's why I'm using this task. Handling token expiry is something I expect this task to handle. |
|
Some example tasks you can use until someone has time to fix this task to support WIF plan output: https://github.com/Azure/alz-terraform-accelerator/tree/main/templates/ci_cd/azuredevops/templates/helpers |
|
Having expired token issues with workload identity in both tasks: TerraformTaskV4 and AzureCLI.
|
To be clear. This is really a limitation of the Terraform backend auth implementation as opposed to these tasks. We are trying our best work around this limitation in the tasks, but at the end of the day, the Terraform backend (and providers) are responsible for token management. If the backend supported Azure CLI auth like the providers, then it would be much easier to work around these timeout problems. As such, I suggest you upvote this issue: hashicorp/terraform#34322 Also, the backend and providers request a new access token for certain operations rather than using a cached one. If there was a way to supply an access token directly to the provider / backend or tell it to cache an access token, then these timeout problems would go away. For other methods (MSI and Client Secret) we don't see the same problem since the source creds do not have a timeout or have a long time out. But behind the scenes they are doing the same thing and getting a new access token. The access token timeout is much longer than the id token, but we see these timeout issues because the provider / backend keeps requesting new tokens throughout the run. This is not something that can be handled by the task since it is the inner working of the provider / Terraform CLI which the task has no control over. For Azure CLI, it is responsible for caching the access token, so that solves the problem, because the provider / backend just asks it for the token rather than trying to generate a new one each time. |
|
I keep running into this issue as well with azurerm_api_management, and still after updating to Terraform 1.9.2 and azurerm provider 3.112.0. APIM should take 8-15 min max but instead times out after 40+ min, resulting in OP's error. |
|
tl;dr: I was able to solve this by setting Reading through the Terraform documentation:
So, Terraform suggests to pass time-limited credentials, such as - task: TerraformTaskV4@4
displayName: Initialize Terraform
inputs:
provider: azurerm
command: init
commandOptions: -input=false
backendServiceArm: ${{ parameters.backendServiceArm }}
backendAzureRmResourceGroupName: ${{ parameters.backendAzureRmResourceGroupName }}
backendAzureRmStorageAccountName: ${{ parameters.backendAzureRmStorageAccountName }}
backendAzureRmContainerName: ${{ parameters.backendAzureRmContainerName }}
backendAzureRmKey: ${{ parameters.backendAzureRmKey }}
backendAzureRmUseEnvironmentVariablesForAuthentication: trueFor TerraformCLI, @hbuckle had already posted a working example... The important part is not to pass |
I added that option to help this scenario, but there are still some limitations around this. In any case I will close this issue for now since we released that update and come back to it as demand requires. As an FYI, AzAPI now implements the ability to get a new id token throughout the apply, but unfortunately, azurerm, azuread and the azurerm backend do not yet, so will need to wait for them to catch up before we hav an end to end solution. Docs here (see the second option): https://registry.terraform.io/providers/Azure/azapi/latest/docs/guides/service_principal_oidc#configuring-the-service-principal-in-terraform |
I have a pretty strange issue. Initially it was failed randomly, now it's failing consistently. It appears the issue is a token was valid 2 hours ago before it was requested?!
The pipeline is running under Azure Container Instance, and I have checked the log timestamp is matching with my local machine (give or take 10 seconds)
Strangely the first auth token negotiation earlier in the code seems to be successful. See timestamp
2023-12-06T00:01:25.5091505Zvs2023-12-06T00:01:31.0398817ZWhich leads me to another question whether the entire thing was attempted twice?
Full debug trace:
Terraform Apply task v4 definition:
The text was updated successfully, but these errors were encountered: