Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

helm proxy repo fails when processing index.yaml due to colon in url #519

Open
emileonhardt opened this issue Nov 19, 2024 · 5 comments
Open

helm proxy repo fails when processing index.yaml due to colon in url #519

emileonhardt opened this issue Nov 19, 2024 · 5 comments
Assignees
@mrprescott
Labels
enhancement

Comments

@emileonhardt
Copy link

What problem are you trying to solve?

I am proxying the bitnami helm chart repo (https://charts.bitnami.com/bitnami) for local access.
When I try to use the proxied repo with "helm repo add", this does not work:

Error: looks like "https://xxx/repository/helm-bitnami-proxy/" is not a valid chart repository or cannot be reached: empty index.yaml file

The downloaded index.yaml ist empty. In the Nexus log, I see the following stack trace:

[streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - Exception in thread "streamcopier-19-thread-2472" java.lang.IllegalArgumentException: NTFS ADS separator (':') in file name is forbidden.
2024-11-18 13:07:18,677+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at org.apache.commons.io.FilenameUtils.indexOfExtension(FilenameUtils.java:955)
2024-11-18 13:07:18,678+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at org.apache.commons.io.FilenameUtils.getExtension(FilenameUtils.java:614)
2024-11-18 13:07:18,678+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at com.sonatype.repository.helm.internal.metadata.IndexYamlAbsoluteUrlRewriterSupport.rewriteUrl(IndexYamlAbsoluteUrlRewriterSupport.java:78)
2024-11-18 13:07:18,678+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at com.sonatype.repository.helm.internal.metadata.IndexYamlAbsoluteUrlRewriterSupport.lambda$2(IndexYamlAbsoluteUrlRewriterSupport.java:68)
2024-11-18 13:07:18,678+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
2024-11-18 13:07:18,678+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
2024-11-18 13:07:18,678+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
2024-11-18 13:07:18,678+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
2024-11-18 13:07:18,678+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
2024-11-18 13:07:18,678+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2024-11-18 13:07:18,678+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
2024-11-18 13:07:18,679+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at com.sonatype.repository.helm.internal.metadata.IndexYamlAbsoluteUrlRewriterSupport.rewriteUrls(IndexYamlAbsoluteUrlRewriterSupport.java:71)
2024-11-18 13:07:18,679+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
2024-11-18 13:07:18,679+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
2024-11-18 13:07:18,679+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
2024-11-18 13:07:18,679+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
2024-11-18 13:07:18,679+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.Iterator.forEachRemaining(Iterator.java:116)
2024-11-18 13:07:18,679+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
2024-11-18 13:07:18,679+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
2024-11-18 13:07:18,679+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
2024-11-18 13:07:18,679+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
2024-11-18 13:07:18,679+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
2024-11-18 13:07:18,679+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2024-11-18 13:07:18,680+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
2024-11-18 13:07:18,680+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at com.sonatype.repository.helm.internal.metadata.IndexYamlAbsoluteUrlRewriterSupport.updateUrls(IndexYamlAbsoluteUrlRewriterSupport.java:58)
2024-11-18 13:07:18,680+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at com.sonatype.repository.helm.internal.metadata.IndexYamlAbsoluteUrlRewriter.lambda$0(IndexYamlAbsoluteUrlRewriter.java:54)
2024-11-18 13:07:18,680+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.function.Consumer.lambda$andThen$0(Consumer.java:65)
2024-11-18 13:07:18,680+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at org.sonatype.nexus.thread.io.StreamCopier.lambda$2(StreamCopier.java:148)
2024-11-18 13:07:18,680+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
2024-11-18 13:07:18,680+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
2024-11-18 13:07:18,680+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
2024-11-18 13:07:18,680+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2024-11-18 13:07:18,680+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2024-11-18 13:07:18,680+0100 ERROR [streamcopier-19-thread-2472] *TASK org.apache.felix.gogo.runtime.threadio.ThreadPrintStream - at java.lang.Thread.run(Thread.java:750)
2024-11-18 13:08:46,441+0100 ERROR

Do you have a workaround you are using at present?

No.

What feature or behavior is this required for?

Helm Proxy Repo.

How could we solve this issue? (Not knowing is okay!)

Supporting URLs with Colons in them.

Tell us about your Nexus Repository deployment: what version, operating system, and database are you using?

OSS 3.69.0-02 on a Windows VM with OrientDB.

Anything else?

This worked until a couple of days ago. I am attaching the offending index.yaml as a zip file.
index.zip
@cgunston
Copy link

cgunston commented Nov 25, 2024 •
edited
Loading

This issue is also affecting us as well. It appears to only be happening on our Bitnami helm proxy, which was working fine previously. Other helm proxies are still working ok.

Interestingly, our Bitnami proxy repo does have an index.yaml file in it with a file size of around 18mb but when browsing to it, a blank page is returned.

@glebsts
Copy link

glebsts commented Jan 7, 2025 •
edited
Loading

Having same issue. Cannot see index.html . Last bitnami-related record in Nexus' outbound.log is
[07/Jan/2025:09:05:48 +0200] - "GET https://charts.bitnami.com/bitnami/index.yaml HTTP/1.1" 302 0 9 "Nexus/3.64.0-04 (OSS; Linux; 5.14.0-362.13.1.el9_3.x86_64; amd64; 1.8.0_345) " [qtp410434371-2140325]
I curled this URL and it is redirect to
https://repo.broadcom.com/bitnami-files/index.yaml which seems to be valid repo index
so I wonder if I should proxy this instead..
I can't find any "follow redirects" setting in repo setup, only "circular redirects", and also as I have to explicitly approve ssl certs, I guess nexus declines using broadcom cert.

@emileonhardt
Copy link
Author

There is a workaround for this issue. A docker proxy repo on Nexus can be used to proxy Helm charts:

  1. Switch to using OCI as protocol for helm. This is anyway recommended.
  2. Setup a docker proxy repo on Nexus for https://registry-1.docker.io
  3. Point your OCI-URL to the docker proxy repo.

@glebsts
Copy link

glebsts commented Jan 7, 2025

  1. Switch to using OCI as protocol for helm. This is anyway recommended.
  2. Setup a docker proxy repo on Nexus for https://registry-1.docker.io
  3. Point your OCI-URL to the docker proxy repo.

Thanks @emileonhardt, will give it a try as well. Originally described in #211, comment here suggests that helm registry login should also be used.

@glebsts
Copy link

glebsts commented Jan 7, 2025 •
edited
Loading

Can confirm that switching to https://repo.broadcom.com/bitnami-files made it work, I am able to pull bitnami Helm charts without OCI docker proxy (which I still consider a hack because its semantically incorrect)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mrprescott mrprescott

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@glebsts @mrprescott @cgunston @emileonhardt