Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Push does not retry after 429 response from registry #1194

Closed
3 tasks done
grzleadams opened this issue Jul 26, 2024 · 1 comment
Closed
3 tasks done

Push does not retry after 429 response from registry #1194

grzleadams opened this issue Jul 26, 2024 · 1 comment

Comments

@grzleadams
Copy link

Contributing guidelines

I've found a bug, and:

  • The documentation does not mention anything about my problem
  • There are no open or closed issues that are related to my problem

Description

When pushing an image to our registry, we occasionally get a 429 Too Many Requests. Instead of retrying the push (which is the expected behavior, and supposedly supported by the Action), it just fails:

#15 exporting to image
#15 pushing layers 5.4s done
#15 pushing manifest for <registry>/<image>:<tag>@sha256:<hash>
#15 pushing manifest for <registry>/<image>:<tag>@sha256:<hash> 40.0s done
#15 ERROR: failed to push <registry>/<image>:<tag>: failed commit on ref "index-sha256:<hash>": unexpected status from PUT request to https://<registry>/v2/<image>/manifests/<tag>: 429 Too Many Requests

Expected behaviour

The push should be retried.

Actual behaviour

The push just fails.

Repository URL

No response

Workflow run URL

No response

YAML workflow

build-apt-dev-image:
    if: github.event.pull_request.draft != 'true'
    needs:
      - setup
    runs-on: large
    outputs:
      image: ${{ steps.setup.outputs.image }}@${{ steps.build.outputs.digest }}
    concurrency:
      group: apt-dev-${{ github.ref_name }}
      cancel-in-progress: ${{ github.ref_name != 'main' }}
    steps:
      - id: setup
        run: |
          image=${{ needs.setup.outputs.registry }}/<image>
          echo "image=${image}" >> $GITHUB_OUTPUT
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v2
      - uses: docker/login-action@v2
        with:
          registry: ${{ needs.setup.outputs.registry }}
          username: ${{ vars.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_PASSWORD }}
      - id: build
        uses: docker/build-push-action@v4
        with:
          context: .
          file: docker/apt_packages.Dockerfile
          push: true
          build-args: |
            cuda_image=<registry>/docker-cache/nvidia/cuda:12.5.1-devel-ubuntu24.04
          tags: |
            ${{ steps.setup.outputs.image }}:${{ needs.setup.outputs.tag }}
            ${{ steps.setup.outputs.image }}:${{ needs.setup.outputs.git_branch }}
          cache-to: type=registry,ref=${{ steps.setup.outputs.image }}-cache:${{ needs.setup.outputs.slug }},mode=max,image-manifest=true
          cache-from: |
              type=registry,ref=${{ steps.setup.outputs.image }}-cache:${{ needs.setup.outputs.slug }}
              type=registry,ref=${{ steps.setup.outputs.image }}-cache:main

Workflow logs

No response

BuildKit logs

No response

Additional info

We're using the Action distributed with GitHub Enterprise 3.12.1, but will try @v6 to see if it helps.

@grzleadams
Copy link
Author

It seems this is related to #719.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant