Async-profiler: Publish Jar (with embedded native agent) to Maven central

[advancedxy]

Some ideas about the Java API:

1. published to maven central, so it can be easily added as dependency in other Maven based projects.
2. Jar contains libAsynProfiler.so(dylib) so that the jar can be used independently
3. Java API execute() method supports 'file' argument (already committed)

[apangin]

OK, it would be nice, if someone volunteered to help with this.

[advancedxy]

I may help with jar packing and API in my spare time. But I have never published jars to central personally, that needs an experienced one

[incubos]

I can do the publishing part.

[advancedxy]

Some other issues popped during my experience with async-profiler, one feature I hope Java API has:
ability to specify dumped file path in API.
The rationale is that sometimes we want to defer the start of profiler: after loading other native library by calling System.load or similar, so the native library's symbol can be identified correctly. Then the profiler output can be dumped at the JVM exit. However it's impossible for Java API to specify an output file.

@apangin what do you think? If this seems ok. I will try to send a PR. And Let's keep this issue open for a while

[apangin]

Java API returns profiling data as a String, then you do whatever you want with this String: write to a file etc.
Shutdown Hook can be used to stop profiler before VM exit.

[advancedxy]

Java API returns profiling data as a String, then you do whatever you want with this String: write to a file etc.

Yeah, I know, but that requires wrapper code to flush the output into file. If we can flush output to file, the execute API can be used as same as Java Agent(or Attach)'s options.

[apangin]

OK. Now if execute(command) contains file= option, the output will be dumped directly to the given file.

[advancedxy]

OK. Now if execute(command) contains file= option, the output will be dumped directly to the given file.

Thanks, it works as expected after I check the master branch.

However my old approach was persisting execute's argument into vmEntry's internal _agent_args: so it's dumped automatically at JVM exit.

[apangin]

I would argue that Java API should not modify global arguments designed primarily for the agent loaded at start-up. At Java level an application can still use Java mechanisms such as Shutdown Hook.

[advancedxy]

I would argue that Java API should not modify global arguments designed primarily for the agent loaded at start-up. At Java level an application can still use Java mechanisms such as Shutdown Hook.

Yeah, that's a fair point. It's ugly to modify _agent_args. User can register a shutdown hook, however it would be more convenient if provided(or handled) by the Java API. I am still thinking other possible approaches.

[jesperpedersen]

Adding

public static final String EVENT_CPU   = "cpu";
public static final String EVENT_ALLOC = "alloc";
public static final String EVENT_LOCK  = "lock";

to AsyncProfiler.java would make the event parameter for start() more clear.

[apangin]

@jesperpedersen Agree

[felixbarny]

Jar contains libAsynProfiler.so(dylib) so that the jar can be used independently

How would you load the native library? OS-dependently export libasyncProfiler.so to a temp folder and load via System.load?

[JigarJoshi]

@apangin soft reminder #365

[tomsisso]

Hi
We were just in a middle of working on this:
https://github.com/taboola/async-profiler-actuator-endpoint
And we made it public today.
It is relevant to this discussion, so - sharing.
Feedback is welcome, share your thoughts :)
Thanks

[wyhasany]

@apangin I've seen your review remarks as #365 of @JigarJoshi PR.

I have following idea to fulfill your requirements:

1. Collect bunch of jars containing only native libraries:
2. async-profiler-linux-x64-1.8.2.jar with async-profiler-linux-x64-1.8.2.so
3. async-profiler-linux-x86-1.8.2.jar with async-profiler-linux-x86-1.8.2.so
4. async-profiler-linux-musl-x64-1.8.2.jar with async-profiler-linux-musl-x64-1.8.2.so
5. async-profiler-linux-arm-1.8.2.jar with async-profiler-linux-arm-1.8.2.so
6. async-profiler-linux-aarch64-1.8.2.jar with async-profiler-linux-aarch64-1.8.2.so
7. async-profiler-macos-x64-1.8.2.jar with async-profiler-macos-x64-1.8.2.so
8. async-profiler-linux-x64-2.0-b1.jar with async-profiler-linux-x64-2.0-b1.so
9. async-profiler-macos-x64-2.0-b1.jar with async-profiler-macos-x64-2.0-b1.so
   and so on. Where native library would be placed at root of jar. Native libraries should be provided rather manually as I don't see easy way to automatically build them at once per different platforms.
10. Collect main async-profiler sources at async-profiler-2.0-b1.jar
11. Extend native library loading in AsyncProfiler.java by providing another getInstance() with enumeration param representing one of native's library jar.

Thanks to such approach it would be easily to integrate mavenized async-profiler with IoC (i.e. Spring). I can imagine that someone would need to import on classpath async-profiler-macos-x64-1.8.2.jar for development and async-profiler-linux-x64-1.8.2.jar for production. Thanks to the IoC configuration proper native library would be loaded for each environment. Pls let me know what is your opinion for such approach.

I can either continue work on #365 PR or create brand new branch.

Remarks:
We could also validate if user mistakenly choose wrong library (as linux lib on mac os and so on). However it adds additional code to maintain so I rather think we can just fail on loading such library, try to load one in default locations if it's also fails then just throw exception.

[apangin]

@wyhasany Thank you for working on this. The plans sounds good to me. I'm only not sure what you mean by "enumeration param representing one of native's library jar"? Can you give an example please?

[wyhasany]

@apangin First of all thank you for your feedback, I can start work now 👍

Enum can looks like as following:

enum NativeJarLibrary {
    LINUX_X64_1_8_2("api.one.profiler.Linux64_1_8_2"),
    LINUX_X86_1_8_2("api.one.profiler.Linux86_1_8_2"),
    LINUX_MUSL_X64_1_8_2("api.one.profiler.LinuxMusl64_1_8_2"),
    LINUX_ARM_1_8_2("api.one.profiler.LinuxArm_1_8_2"),
    LINUX_AARCH64_1_8_2("api.one.profiler.LinuxAarch64_1_8_2"),
    MACOS_X64_1_8_2("api.one.profiler.MacOsX64_1_8_2"),
    MACOS_X64_2_0_0("api.one.profiler.MacOsX64_2_0_0"),
    LINUX_X64_2_0_0("api.one.profiler.Linux64_2_0_0");

    private String clazzNextToNativeLibrary;

    NativeJarLibrary(String clazzNextToNativeLibrary) {
        this.clazzNextToNativeLibrary = clazzNextToNativeLibrary;
    }

    Class<?> classNextToNativeLibrary(ClassLoader classLoader) {
        if (classLoader != null) {
            return getClass(classLoader);
        } else {
            return getClass(NativeJarLibrary.class.getClassLoader());
        }
    }

    private Class<?> getClass(ClassLoader classLoader) {
        try {
            return Class.forName(clazzNextToNativeLibrary, true, classLoader);
        } catch (ClassNotFoundException e) {
            //log warn
            return null;
        }
    }
}

based on class we can find native library. Current API then could be extended with following method:

+ public static AsyncProfiler getInstance(NativeJarLibrary nativeJarLibrary)

any client can call that getInstance if it fails because that user uses different architecture or he have forgotten to add dependency we'll log warning and fallback to standard native library resolution (as in getInstance()).

[felixbarny]

In the Elastic APM Java agent, we bundle async-profiler as a resource and load it depending on the system it's running on: https://github.com/elastic/apm-agent-java/blob/f9a1bfd6a15c9c8e2bb6512a6973375a78ebb97d/apm-agent-plugins/apm-profiling-plugin/src/main/java/co/elastic/apm/agent/profiler/asyncprofiler/AsyncProfiler.java#L94-L121

Could that approach be an option for the official async-profiler jar?

[martin-g]

Netty project uses https://github.com/trustin/os-maven-plugin and this is the best solution I have seen in Java world for detecting the OS and architecture and to decide what extension to produce at artifact build time and what dependency to download when using a third party library.

[wyhasany]

@felixbarny There are two differents:

• there would be many jars per native platforms, you keep all native libraries in one jar, as @apangin mentioned in PR he won't loose space for keeping unnecessary native libraries
• I won't add dynamic native library resolution, because it adds additional burden to maintenance as we can't be sure about all possible properties on different platforms. However client can choose proper platform for itself.

@martin-g Thank you for your recommendation. I don't know os-maven-plugin. I'm going to learn how we can use it to simplify async-profiler mavenizing.

[apangin]

@wyhasany TBH, I don't see much value in enumerating native libraries and letting developers choose one themselves.

1. There is no intention to support multiple major versions of a library in API, e.g. API 2.0 will support async-profiler 2.x agents, but not 1.x.
2. For each platform, there is exactly one matching native library - no need to choose from the enumeration. This means, Java class may load the right library automatically (if it presents on the class path).
3. I like the approach suggested by @felixbarny - it's simple, efficient, and does not let developer make a mistake. Only detection of glibc/musl may be a bit tricky, but doable. Without automatic detection users would need to implement selection logic themselves every time. Multiple JARs for different platforms is not a problem, as long as the required JARs are on the class path, and resource names don't clash.

[wyhasany]

@apangin ok. I'm going to do it as you wish :)

[apangin]

Never mind - it looks like I need to do the routine myself anyway, in order to be able to make releases on Maven Central.
I've just managed to published async-profiler API under the following artifact:

<dependency>
  <groupId>tools.profiler</groupId>
  <artifactId>async-profiler</artifactId>
  <version>1.8.3</version>
</dependency>

It does not include agent library though. This will be only included in v2.0.

[alexander-yakushev]

FWIW, a minor datapoint: clj-async-profiler does exactly what @felixbarny suggested – extracting the native binary into a temp directory at runtime and loading it with System.load. Hacky, but it works.

[felixbarny]

Some hardened systems are configured so that they don't run binaries from temp. A workaround for that would be to let users define the target folder. That can be one through a system property, for example. But temp is still a good default, I think.

[pan3793]

It does not include agent library though. This will be only included in v2.0.

Is there an estimated available time?

[apangin]

Is there an estimated available time?

I don't have an estimate, sorry. The task is pretty high on my list, but there are other current requests from business.

[pan3793]

@apangin I saw v2.0 has been released several days ago, seems this feature is absent, any update for future plan?

[apangin]

@pan3793 The task is in the nearest plans. No specific commitments though.