Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: create an alpine compatible image #632

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

samyuh
Copy link

@samyuh samyuh commented Jan 16, 2025

Problem

When importing a data source of a file that was previously compressed using the snappy compression algorithm, the compression fails with the following exception:

Error loading shared library ld-linux-x86-64.so.2: No such file or directory.

This problem occurs since the dockerimage does not containg the ld-linux-x86-64.so.2 lib, which is from glibc. Alpine images use musl making this incompatible.

Having this dependency being linked dynamically can mess things up. Ideally, we would like to have snappy being compiled against a MUSL image.

Snappy removed pure-java mode 1.1.9.0 (https://github.com/xerial/snappy-java/pull/381/files), which could be used to run snappy in non-supported systems with the use of one flag. We could force an older version but it was removed due to some problems (such as data corruption) and older versions than 1.1.9.0 contain critical CVEs (https://mvnrepository.com/artifact/org.xerial.snappy/snappy-java).

Workaround

Alpine images also have some packages that allow us to run binaries precompiled against glibc libraries (they don't allow us to compile against it), such as gcompat. In our case, libc6-compat provides /lib64/ld-linux-x86-64.so.2 compatible lib. Turns out that /lib64/ld-linux-x86-64.so.2 is a symlink to /lib/libc.musl-x86_64.so.1. Adding a /lib/ld-linux-x86-64.so.2 linked to /lib/ld-musl-x86_64.so.1 solves the issue since that is the place snappy is searching for the dependency.

This is the same solution employed by https://stackoverflow.com/a/55568352.

Fix

The library now includes native binaries compiled specifically for musl environments, eliminating the need for workarounds like installing gcompat or libc6-compat packages in Alpine containers.

This improvement means:

  • Direct compatibility with Alpine Linux
  • No need for additional compatibility packages
  • Better performance by using native musl binaries instead of compatibility layers
  • Simplified deployment in Alpine-based containers

Closes #616
Closes #239

@samyuh samyuh changed the title draft: feat: add musl support feat: add musl support Jan 17, 2025
@samyuh samyuh changed the title feat: add musl support feat: create musl compatible image Jan 17, 2025
@samyuh samyuh marked this pull request as draft January 17, 2025 17:09
@samyuh samyuh marked this pull request as ready for review January 18, 2025 01:00
@samyuh
Copy link
Author

samyuh commented Jan 18, 2025

Hey @xerial! This is a pull request that gives compatibility with systems using musl. Can you please take a look at this? Thanks!

@samyuh samyuh changed the title feat: create musl compatible image feat: create an alpine compatible image Jan 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Using snappy with MUSL Linux x86_64 builds for musl libc
1 participant