[BUG]: Agent self update fails

[alenros]

What happened?

Running the self-update for an agent (Windows 3.224.1) fails.
Same failure when trying to update all agents.
Same failure when trying to update from 3.220.5
Manually installing newer agent version works.

Tried placing the zipped agent in Azure DevOps server as explained in the documentation without it being detected - built-in version 2.217.2 is still the detected server version.

Attached log with VSTSAGENT_TRACE set to true.

Updating a 2.211.0 agent half-works, as it updates to the built-in 2.217.2, instead of to the latest 2.220.0.

Versions

Agent version: 3.224.1 Windows x64
Agent OS: Windows 11

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

Azure DevOps Server (Please specify exact version in the textbox below)

Azure DevOps Server Version (if applicable)

Azure DevOps 2022.01 (OnPrem)

Operation system

Windows Server 2022 Standard

Version controll system

Git, TFSVC

Relevant log output

[2023-11-28 07:54:23Z VERB VisualStudioServices] Started GET request to https://**********/tfs/_apis/distributedtask/packages/agent/win-x64/2.217.2
[2023-11-28 07:54:23Z INFO Agent] Refresh message received, kick-off selfupdate background process.
[2023-11-28 07:54:23Z VERB MessageListener] Entering DeleteMessageAsync
[2023-11-28 07:54:23Z VERB VisualStudioServices] Started DELETE request to https://**********/tfs/_apis/distributedtask/pools/3/messages/3
[2023-11-28 07:54:23Z VERB VisualStudioServices] Finished GET request to https://**********/tfs/_apis/distributedtask/packages/agent/win-x64/2.217.2 with status code 200
[2023-11-28 07:54:23Z INFO SelfUpdater] Version '2.217.2' of 'agent' package available in server.
[2023-11-28 07:54:23Z INFO SelfUpdater] Current running agent version is 3.224.1
[2023-11-28 07:54:23Z INFO SelfUpdater] We don't downgrade agent from 3.* to 2.*, skipping update
[2023-11-28 07:54:23Z INFO SelfUpdater] Can't find available update package.
[2023-11-28 07:54:24Z VERB VisualStudioServices] Finished DELETE request to https://**********/tfs/_apis/distributedtask/pools/3/messages/3 with status code 204
[2023-11-28 07:54:24Z VERB MessageListener] Entering GetNextMessageAsync
[2023-11-28 07:54:24Z VERB VisualStudioServices] Started GET request to https://**********/tfs/_apis/distributedtask/pools/3/messages
[2023-11-28 07:54:24Z VERB Agent] Auto update task running at backend, waiting for getNextMessage or selfUpdateTask to finish.
[2023-11-28 07:54:24Z INFO Agent] Auto update task finished at backend, there is no available agent update needs to apply, continue message queue looping.

[aleksandrlevochkin]

Hi @alenros thank you for reporting, we'll take a look

[dtemme]

Hi,

we are also hanging in an agent update loop.
Our agents always try to update themselves after updating from Azure DevOps Server 2022 to 2022.1.
Now our build queue is full and the agents are blocking requests because they try to update themselves the whole time.
This is quite urgent for us.

The log files say that the checksum check is missing.

[2023-12-04 06:59:59Z INFO SelfUpdater] Download agent: begin download
[2023-12-04 07:00:10Z INFO SelfUpdater] Download agent: finished download
[2023-12-04 07:00:10Z WARN SelfUpdater] Unable to perform the necessary checksum validation since the target package hash is missed
[2023-12-04 07:00:11Z INFO Agent] Ignore agent update exception. System.Threading.Tasks.TaskCanceledException: Agent package checksum validation failed.
There are possible reasons why this happened:
  1) The agent package was compromised.
  2) The agent package was not fully downloaded or was corrupted during the download process.
You can skip checksum validation for the agent package by setting the environment variable DISABLE_HASH_VALIDATION=true
   at Microsoft.VisualStudio.Services.Agent.Listener.SelfUpdater.DownloadLatestAgent(CancellationToken token) in D:\a\1\s\src\Agent.Listener\SelfUpdater.cs:line 375
   at Microsoft.VisualStudio.Services.Agent.Listener.SelfUpdater.SelfUpdate(AgentRefreshMessage updateMessage, IJobDispatcher jobDispatcher, Boolean restartInteractiveAgent, CancellationToken token) in D:\a\1\s\src\Agent.Listener\SelfUpdater.cs:line 82
   at Microsoft.VisualStudio.Services.Agent.Listener.Agent.RunAsync(AgentSettings settings, Boolean runOnce) in D:\a\1\s\src\Agent.Listener\Agent.cs:line 379

[alenros]

Hi,

we are also hanging in an agent update loop. Our agents always try to update themselves after updating from Azure DevOps Server 2022 to 2022.1. Now our build queue is full and the agents are blocking requests because they try to update themselves the whole time. This is quite urgent for us.

The log files say that the checksum check is missing.

[2023-12-04 06:59:59Z INFO SelfUpdater] Download agent: begin download
[2023-12-04 07:00:10Z INFO SelfUpdater] Download agent: finished download
[2023-12-04 07:00:10Z WARN SelfUpdater] Unable to perform the necessary checksum validation since the target package hash is missed
[2023-12-04 07:00:11Z INFO Agent] Ignore agent update exception. System.Threading.Tasks.TaskCanceledException: Agent package checksum validation failed.
There are possible reasons why this happened:
  1) The agent package was compromised.
  2) The agent package was not fully downloaded or was corrupted during the download process.
You can skip checksum validation for the agent package by setting the environment variable DISABLE_HASH_VALIDATION=true
   at Microsoft.VisualStudio.Services.Agent.Listener.SelfUpdater.DownloadLatestAgent(CancellationToken token) in D:\a\1\s\src\Agent.Listener\SelfUpdater.cs:line 375
   at Microsoft.VisualStudio.Services.Agent.Listener.SelfUpdater.SelfUpdate(AgentRefreshMessage updateMessage, IJobDispatcher jobDispatcher, Boolean restartInteractiveAgent, CancellationToken token) in D:\a\1\s\src\Agent.Listener\SelfUpdater.cs:line 82
   at Microsoft.VisualStudio.Services.Agent.Listener.Agent.RunAsync(AgentSettings settings, Boolean runOnce) in D:\a\1\s\src\Agent.Listener\Agent.cs:line 379

This seems like a different problem.
I suggest you open a new Issue with all the details.

[aleksandrlevochkin]

@alenros When you trigger an update, it will attempt to update to the latest version released with the on-premises ADO server that you use which in this case seems to be 2.217.2, so this behavior is expected.

But placing the agent in a folder as specified in the documentation should work. I tried to reproduce the problem with the same ADO server version and I was able to update the agent from 2.217.2 to 3.224.1 and then from 3.224.1 to 3.230.0 without any issues. Could you double check that you place the agent in the correct directory?

[aleksandrlevochkin]

Hi @dtemme it appears to be a different problem. Please, open a new issue

[alenros]

@aleksandrlevochkin It seems I have indeed placed the files in the wrong directory.
For those of you having a similar problem: put the agent release zip file in your Application Tier in the following folder %ProgramData%\Microsoft\Azure DevOps\Agents