[Question]: How to debug <fatal: fetch-pack: invalid index-pack output>?

[Davidswinkels]

Describe your question

We have setup a self hosted Azure DevOps agent on Azure Kubernetes Service. Most functionality works good.

However, the git checkout (simply checkout: self) in the Azure DevOps pipeline with the self hosted Agent on AKS sometimes runs into this error, where the agent is not able to receive objects from Azure DevOps. 80% of the time the self hosted agent can do a git checkout. In 20% of cases it cannot git checkout. We checked memory, authentication, networking and used system.debug: true to debug, but were not able to find a cause. Does anyone understand why git cannot checkout Azure DevOps on self hosted Azure DevOps agent?

remote: Found 736 objects to send. (194 ms)        
Receiving objects:   0% (1/736)
Receiving objects:   0% (1/736), 4.50 MiB | 4.23 MiB/s
Receiving objects:   0% (1/736), 8.76 MiB | 4.24 MiB/s
Receiving objects:   0% (1/736), 12.71 MiB | 4.15 MiB/s
Receiving objects:   0% (1/736), 14.45 MiB | 3.55 MiB/s
Receiving objects:   0% (1/736), 18.17 MiB | 3.40 MiB/s
Receiving objects:   0% (1/736), 21.87 MiB | 3.40 MiB/s
Receiving objects:   0% (1/736), 23.75 MiB | 2.75 MiB/s
Receiving objects:   0% (1/736), 27.28 MiB | 3.00 MiB/s
Receiving objects:   0% (1/736), 31.66 MiB | 3.43 MiB/s
Receiving objects:   0% (1/736), 34.18 MiB | 3.09 MiB/s
Receiving objects:   0% (1/736), 37.31 MiB | 3.21 MiB/s
Receiving objects:   0% (1/736), 40.22 MiB | 3.38 MiB/s
Receiving objects:   0% (1/736), 42.58 MiB | 2.95 MiB/s
Receiving objects:   0% (1/736), 43.99 MiB | 2.44 MiB/s
Receiving objects:   0% (1/736), 45.05 MiB | 2.15 MiB/s
Receiving objects:   0% (1/736), 48.30 MiB | 1.98 MiB/s
Receiving objects:   0% (1/736), 50.21 MiB | 2.04 MiB/s
Receiving objects:   0% (1/736), 54.63 MiB | 2.47 MiB/s
Receiving objects:   0% (1/736), 58.62 MiB | 3.11 MiB/s
Receiving objects:   0% (1/736), 62.75 MiB | 3.69 MiB/s
Receiving objects:   0% (1/736), 64.78 MiB | 3.44 MiB/s
Receiving objects:   0% (1/736), 68.64 MiB | 3.54 MiB/s
Receiving objects:   0% (1/736), 70.40 MiB | 3.04 MiB/s
Receiving objects:   0% (1/736), 72.60 MiB | 2.60 MiB/s
Receiving objects:   0% (1/736), 76.60 MiB | 2.85 MiB/s
Receiving objects:   0% (1/736), 79.65 MiB | 2.95 MiB/s
Receiving objects:   0% (1/736), 83.09 MiB | 2.97 MiB/s
Receiving objects:   0% (1/736), 87.17 MiB | 3.57 MiB/s
Receiving objects:   0% (1/736), 90.97 MiB | 3.59 MiB/s
fatal: fetch-pack: invalid index-pack output
##[warning]Git fetch failed with exit code 128, back off 4.437 seconds before retry.
git --config-env=http.extraheader=env_var_http.extraheader fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules origin --depth=1

Versions

Ubuntu 22.04
Azure DevOps agent version: 3.230.0
Self hosted on Azure Kubernetes Service

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Operation system

Ubuntu 22.04

Version controll system

No response

Azure DevOps Server Version (if applicable)

No response

[DenisRumyantsev]

Hi @Davidswinkels, can you try to reproduce this issue without using the agent by executing these git commands on the same environment on which the agent is running?

mkdir ./test-fetch/
cd ./test-fetch/
git init
git remote add origin <git-repo-url>
git fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules origin --depth=1

[Davidswinkels]

Hi @Davidswinkels, can you try to reproduce this issue without using the agent by executing these git commands on the same environment on which the agent is running?

mkdir ./test-fetch/
cd ./test-fetch/
git init
git remote add origin <git-repo-url>
git fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules origin --depth=1

Thanks for helping to debug. We tried to reproduce the issue on the pod with above command 5 times, but were not able to reproduce unfortunately.

[aleksandrlevochkin]

Hi @Davidswinkels, if the issue isn't reproducible anymore, can we close it?

[Davidswinkels]

Sure can be closed. Since it can't be debugged/fixed. It still happens now and then, but workaround is just to run the pipeline in Azure DevOps multiple times.