Skip to content

Latest commit

 

History

History

graalpy-freeze-dependencies-guide

README.md

Pinning Python Dependencies

Python libraries can be used in and shipped with plain Java applications.

The GraalPy Maven artifacts and GraalVM Polyglot APIs allow flexible integration with different project setups.

Using Python packages in Java projects often requires a bit more setup, due to the nature of the Python packaging ecosystem. GraalPy provides a python-embedding package that simplifies the required setup to ship Python packages as Java resources or in separate folders. The important entry points to do so are the VirtualFileSystem and the GraalPyResources classes.

Unlike with Java libraries, Python packages frequently specify their dependencies as a range rather than one specific version. This can create issues during development and testing, because transitive set of dependencies may change when some package publishes a new release.

We recommend pinning all transitive dependencies to single version and upgrade them manually in a controlled fashion. This guide shows how this can be done with the GraalPy Maven plugin. We will install package vaderSentiment, discover all its transitive dependencies and then pin them in the Maven.

1. What you will need

To complete this guide, you will need the following:

  • Some time on your hands
  • A decent text editor or IDE
  • A supported JDK1, preferably the latest GraalVM JDK

2. Solution

We recommend that you follow the instructions in the next sections and create the application step by step. However, you can go right to the completed example.

3. Writing the application

You can start with any Maven or Gradle application that runs on JDK 17 or newer. We will demonstrate on both build systems. A default Maven application generated from an archetype.

mvn archetype:generate -DarchetypeGroupId=org.apache.maven.archetypes \
  -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.5 \
  -DgroupId=example -DartifactId=example -Dpackage=org.example \
  -Dversion=1.0-SNAPSHOT -DinteractiveMode=false

And a default Gradle Java application generated with the init task.

gradle init --type java-application --dsl kotlin --test-framework junit-jupiter \
    --package org.example --project-name example --java-version 17 \
    --no-split-project --no-incubating

4. Adding packages

Most Python packages are hosted on PyPI and can be installed via the pip tool. The Python ecosystem has conventions about the filesystem layout of installed packages that need to be kept in mind when embedding into Java. You can use the GraalPy plugins for Maven or Gradle to manage Python packages for you.

For Maven, add dependency on GraalPy runtime, and configure the GraalPy Maven plugin:

pom.xml

    <dependency>
        <groupId>org.graalvm.polyglot</groupId>
        <artifactId>python</artifactId>
        <version>24.1.2</version>
        <type>pom</type>
    </dependency>

pom.xml

<build>
    <plugins>
        <plugin>
            <groupId>org.graalvm.python</groupId>
            <artifactId>graalpy-maven-plugin</artifactId>
            <version>24.1.2</version>
            <executions>
                <execution>
                    <configuration>
                        <packages> <!-- ① -->
                            <package>vaderSentiment==3.3.2</package>
                        </packages>
                    </configuration>
                    <goals>
                        <goal>process-graalpy-resources</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

For Gradle, add the GraalPy plugin, configure it, and add the dependency on the GraalPy runtime:

build.gradle.kts

plugins {
    application
    id("org.graalvm.python") version "24.1.2"
}

build.gradle.kts

graalPy {
    packages = setOf("vaderSentiment==3.3.2") // ①
}

dependencies {
    implementation("org.graalvm.python:python:24.1.2")
}

❶ The packages section lists all Python packages optionally with requirement specifiers. In this case, we install the vaderSentiment package and pin it to version 3.3.2. Because we are not specifying <pythonResourcesDirectory> the plugins will embed the packages into the resulting JAR as a standard Java resource.

5. Determining and Pinning Transitive Dependencies

When you package the application, it installs all the transitive dependencies in a newly created virtual environment:

./mvnw package
./gradlew assemble

If the compilation is successful, one can run the following command to get versions of all the installed Python packages if you use Maven:

On macOS and Linux:

./target/classes/org.graalvm.python.vfs/venv/bin/pip3 freeze -l

On Windows:

.\target\classes\org.graalvm.python.vfs\venv\Scripts\pip3.exe freeze -l

If you are using Gradle, run the following command:

On macOS and Linux:

./app/build/generated/graalpy/resources/org.graalvm.python.vfs/venv/bin/pip3 freeze -l

On Windows:

.\app\build\generated\graalpy\resources\org.graalvm.python.vfs\venv\Scripts\pip3.exe freeze -l

The output will look something like this:

certifi==2024.8.30
charset-normalizer==3.1.0
idna==3.8
requests==2.32.3
urllib3==2.2.2
vaderSentiment==3.3.2

Copy and paste the package names and versions. If you use Maven, paste them in the pom.xml section of the packages and wrap them in <package> xml tag:

pom.xml

<build>
    <plugins>
        <plugin>
            <groupId>org.graalvm.python</groupId>
            <artifactId>graalpy-maven-plugin</artifactId>
            <version>24.1.2</version>
            <executions>
                <execution>
                    <configuration>
                        <packages> <!-- ① -->
                            <package>vaderSentiment==3.3.2</package>
                            <package>certifi==2024.8.30</package>
                            <package>charset-normalizer==3.1.0</package>
                            <package>idna==3.8</package>
                            <package>requests==2.32.3</package>
                            <package>urllib3==2.2.2</package>
                        </packages>
                    </configuration>
                    <goals>
                        <goal>process-graalpy-resources</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

For Gradle, paste them into the packages block:

build.gradle.kts

packages = setOf(
    "vaderSentiment==3.3.2",
    "certifi==2024.8.30",
    "charset-normalizer==3.1.0",
    "idna==3.8",
    "requests==2.32.3",
    "urllib3==2.2.2"
)

Note: one can use other Python tools, such as pipdeptree to generate the following dependency tree, where we can also see the version ranges.

vaderSentiment==3.3.2
└── requests [required: Any, installed: 2.32.3]
    ├── certifi [required: >=2017.4.17, installed: 2024.8.30]
    ├── charset-normalizer [required: >=2,<4, installed: 3.1.0]
    ├── idna [required: >=2.5,<4, installed: 3.8]
    └── urllib3 [required: >=1.21.1,<3, installed: 2.2.2]

Warning: Is it not recommended to manually alter the virtual environment. Any changes will be overridden by the GraalPy build plugins.

8. Next steps

Footnotes

  1. Oracle JDK 17 and OpenJDK 17 are supported with interpreter only. GraalVM JDK 21, Oracle JDK 21, OpenJDK 21 and newer with JIT compilation. Note: GraalVM for JDK 17 is not supported. ↩