GraalVM support for native images

[klopfdreh]

Description

Because we want to use jgit in GraalVM native images it would be great if this could be achieved through this ticket.

Motivation

jgit in GraalVM native images.

Alternatives considered

N/A

Additional context

N/A

Spring Boot Workaround

@Configuration
@ImportRuntimeHints(GitHubRuntimeHints.class)
@Slf4j
public class GitHubRuntimeHints implements RuntimeHintsRegistrar {

    @Override
    public void registerHints(RuntimeHints hints, ClassLoader classLoader) {
        Arrays.stream(System.getProperty("java.class.path").split(File.pathSeparator)).forEach(classpathEntry -> {
            // If the classpathEntry is no jar skip it
            if (!classpathEntry.endsWith(".jar")) {
                return;
            }

            try (JarInputStream is = new JarInputStream(Files.newInputStream(Path.of(classpathEntry)))) {
                JarEntry entry = is.getNextJarEntry();
                while (entry != null) {
                    String entryName = entry.getName();
                    if (entryName.endsWith(".class") && entryName.startsWith("org/eclipse/jgit") && !entryName.contains("package-info")) {
                        String githubApiClassName = entryName.replace("/", ".");
                        String githubApiClassNameWithoutClass = githubApiClassName.substring(0, githubApiClassName.length() - 6);
                        log.info("Registered class {} for reflections and serialization.", githubApiClassNameWithoutClass);
                        hints.reflection().registerType(TypeReference.of(githubApiClassNameWithoutClass), MemberCategory.values());
                        hints.serialization().registerType(TypeReference.of(githubApiClassNameWithoutClass));
                    }
                    entry = is.getNextJarEntry();
                }
            } catch (IOException e) {
                log.warn("Error while reading jars", e);
            }
        });

        hints.reflection()
            .registerType(TypeReference.of(IOException.class),
                hint -> hint.withMembers(MemberCategory.values())
            );

        hints.resources()
            .registerPattern("application.yml");
    }
}

Before the native-image command is used you have to set the environment varialbes

# Git configuration home folder
ENV XDG_CONFIG_HOME=/writablePath/
# Don't read git system configuration
ENV GIT_CONFIG_NOSYSTEM=true

Important: this is not a good solution as it enables all jgit classes for reflections and serialization.

An example of a native build can be found here: https://github.com/klopfdreh/native-cloud-config-test/blob/main/client/Dockerfile (Note: This does not use builder-image with multi stage and is only an example)

[msohn]

I don't understand what you think in JGit is wrong or missing in order to allow using it in GraalVM native images.

[klopfdreh]

So there are some classes which are accessed via reflections and / or serialized. Also if static resources files are used from classpath they also need to be included ( example: JodaOrg/joda-time#784 ) If you don’t provide those hints it causes issues during runtime when using native-images with GraalVM. From what I tested IOException was used via reflections.

There is nothing wrong with JGit, but it requires some meta information files so that there are no issues with native-image. 😀

I created this ticket to check if reflection / serialization / static resources are used and to add those hints.

With my workaround mentioned in the description JGit is working without any issues in native images. 👍

For more information have a look at the GraalVM documentation: https://www.graalvm.org/latest/reference-manual/native-image/metadata/

Note: You can provide separate file for reflection / serialization / resources - the Spring Boot team documented it with my request: spring-projects/spring-boot#42515

[chirontt]

@klopfdreh
FYI: I've created a JGit HTTP server project and successfully compiled it to native image using GraalVM, and all it needs are a couple of configuration files for GraalVM to use. No need for any "Spring Boot Workaround" code.

[klopfdreh]

Well I used it for client purpose and had a runtime issue that IOException was used via Reflections. That’s the reason I created the ticket - to ensure all functionality can be used without any errors in a native image.

[chirontt]

For a JGit client, I've compiled the JGit PGM client app to native executable using GraalVM. In that project I needed to use the GraalVM's Tracing Agent to easily gather metadata and prepare configuration files, although the tool may not be perfect, i.e. it was missing many classes that I had to manually add them to the configuration files.

[klopfdreh]

For a JGit client, I've compiled the JGit PGM client app to native executable using GraalVM. In that project I needed to use the GraalVM's Tracing Agent to easily gather metadata and prepare configuration files, although the tool may not be perfect, i.e. it was missing many classes that I had to manually add them to the configuration files.

You shouldn’t need to configure anything as the library needs to provide that information.

As it is required to use native-maven-plugin for Spring Boot I also used it but it is not sufficient.

I just provided one way to make a native build available but JGIT itself should provide all required settings so that you don’t have to add anything.

The JGIT team should run the agent and add the meta data to the source code.

[chirontt]

The JGIT team should run the agent and add the meta data to the source code.

I'm not part of the JGit team, but I disagree with this statement. A bad consequence of specifying every classes in the JGit library to the GraalVM configuration files, just in case they are needed at runtime, will include all of them in the resultant native executable regardless they are actually used at runtime or not, and this defeats the main purpose of GraalVM of filtering out unused/unneeded classes from the executable, not to mention the increase in size & memory footprint of the resulting native executable. Worse, once all the classes are specified automatically in the configuration files as you suggested and be part of the library package, it's impossible to exclude any of them from the resulting executable even when you know in advance you won't need them at runtime in your app.

I think it should be up to the application designer to include/exclude what ends up in the native executable, using whatever tools available during design/building (even manually), rather than forcing the library designer to include all of the library's classes in those configuration files just in case. In other words, the onus should be on the application itself to make sure it still works when choosing an exotic (and still limited) technology such as GraalVM native image, rather than putting the burden of it on the JGit library to conform.

[klopfdreh]

I'm not part of the JGit team

Already browsed your GitHub account 😄

but I disagree with this statement. A bad consequence of specifying every classes in the JGit library to the GraalVM configuration files, just in case they are needed at runtime, will include all of them in the resultant native executable regardless they are actually used at runtime or not, and this defeats the main purpose of GraalVM of filtering out unused/unneeded classes from the executable, not to mention the increase in size & memory footprint of the resulting native executable.

Not all classes, only those which need reflection / serialization or those which are dynamic proxies. Resources also need to be added, as I pointed out in my previous comments. If a framework needs reflections there is nothing bad to just provide those classes in a reflect-config.json. If only those classes are added the footprint should be as big as JGIT requires it to be in a native image.

I think it should be up to the application designer to include/exclude what ends up in the native executable

That is wrong. If a framework is going to support Native Images it should provide the metadata information for it.

when choosing an exotic (and still limited) technology such as GraalVM native image

This is no argument. All technologies started as "exotic" and "new" until more and more developers are going to use them. 😄

Beside that many projects are providing those information already and I contributed to those projects. Here are some examples:

• Amazon AWS SDK for Java V2
• Amazon CRT Java
• Github API
• Snappy Java (used in Micrometer which is used in Spring)
• Joda Time
• Many Spring related projects (example on which I contributed: Spring Retry, Spring Cloud AWS, Spring Cloud Config)
• Spotbugs (fixes some issues regarding AOT / AOP with 4.9.x.x - released in future)

I just want to point you to examples showcasing related changes:

• Snappy Java: https://github.com/xerial/snappy-java/blob/ec23d7c611563bedce536ca4d02ebdb9a690ea91/src/main/resources/META-INF/native-image/org.xerial.snappy/snappy-java/resource-config.json
• Amazon AWS SDK for Java V2: https://github.com/aws/aws-sdk-java-v2/blob/0bef9fdaec48c6ce80a59d5efcac905916b2d3ec/services/s3/src/main/resources/META-INF/native-image/software.amazon.awssdk/s3/reflect-config.json
• Joda Time: https://github.com/JodaOrg/joda-time/tree/2ce9b4cdc207130a186dafbb8456cd0b0cfd9e4c/src/main/resources/META-INF/native-image/joda-time/joda-time

Also, as I pointed out I just wrote a workaround for us with a RuntimeHintsRegistrar of Spring Boot. This should not be used as JGIT should not depend on any Spring classes. Instead files like in the example showcases should be used to provide the information in a more generic way.