Openj9: VM Option -XX:hashCode=2 not working as expected

[guekho64]

I understand that every JVM implementation has different options, and that each one of them may or may not add compatibility for every option that any other JVM has, aside from the standard ones that are reglamentary, like -help.

That being said, I'd like to know if Eclipse OpenJ9 has support for using the _-XX:hashCode_ option, and if that's the case, what could be causing the JVM to not preserve hash codes between different executions of a program.

I admit my scenario it's quite odd, as I use this option to mantain consistency of the output from Object#hashCode between executions of a determined program, being able to store the results for later use.

Hotspot JVM outputs the desired results, as expected, but Eclipse J9's won't mantaint consistency between executions.

I've already read the docs, but they don't mention anything about _-XX:hashCode_. I went searching into the source code hosted in this repo, trying to find something about this, with no success.

Also I've alredy added some additional java options to ensure _-XX:hashCode_ isn't being ignored by the JVM, being those:

-XX:hashCode=2 -XX:+PositiveIdentityHash -XX:-IgnoreUnrecognizedVMOptions -XX:+UnlockExperimentalVMOptions

Is Eclipse OpenJ9 able to reproduce Hotspot JVM behavior? Or am I searching something unfindable?

[DanHeidinga]

OpenJ9 doesn't support the -XX:hashCode option.

I use this option to mantain consistency of the output from Object#hashCode between executions of a determined program, being able to store the results for later use.

Can you expand on what it does and why it's important to your use case? Would a JVMTI agent be a more portable choice to control the Object#hashCode value?

[pdbain-ibm]

OpenJ9's identity hashcode is based on the location of an object in the heap, which is non-deterministic and will generally vary from run to run. Are you calling theObject.hashCode() or System.identityHashCode(theObject)?

[guekho64]

@pdbain-ibm I've alredy tried both, with no success. Under normal circumstances, I only make use of theObject.hashCode().

@DanHeidinga The main reason I rely in -XX:hashCode, it's because I'm working on a certain API, and I'm trying to make some changes to it.

Unfortunately, because it's an API, I wasn't the only one who wrote some code based on it, so compatibility it's important. Here's an example of why I'm using -XX:hashCode

Code

public class BaseObject {
    public BaseObject (int objectID) {

        private int id = objectID;

        /* Original code made use of objectID, provided by any developer using the API,
            but there weren't any kind of restrictions on the value provided to the constructor,
            so, many times, different developers would use the same number, causing
            incompatibilities between their extensions. */

        // I had to override the value provided to the method, effectively ignoring the original value
        id = myValue();

        // Then, I had to register wich objectID was assigned to this object
        registerObject(this.get_id());

Once all the Objects who extended BaseObject were constructed, the registry was simply passed to a method that converted all the values inside to strings, saving the results into a file.

Now, here comes the interesting part: Because ID's were used later by both parts (API and Developer) to identify Objects and assign some specific properties to each one of them, I had to redirect each one of the custom ID's to the original Objects, but there wasn't a reliable way to tell which Object was which if there were two identical ID's.

That's why I used Object.hashCode() to determine the real identity of a determined Object, because, as far as -XX:hashCode=2 was added to the JVM arguments, Object1.hashCode() would always return a determined value, alongside other parameters I made use of, like the name of the package of the caller class and stuff like that, to ensure Objects would never have the same ID...

In fact, this is a little bit more complex, as the source code of the API is lost and the only way I was able to do all the aforementioned stuff was through ASM, obviously involving bytecode manipulation and things like that...

[JasonFengJ9]

@guekho64 Hotspot JVM -XX:hashCode=2 appears just return a constant 1 for most objects hashCode() except String objects from my quick experiments. So if your target objects are not String, Object1.hashCode() results could be hardcoded as 1 or just be ignored. If String objects have to be handled, my thought is to use the String values directly just like package name and other staff you mentioned.
With these modification, I hope your program can adopt OpenJ9 JVM and also doesn't require -XX:hashCode=2 -XX:+UnlockExperimentalVMOptions.

[guekho64]

You're _almost_ right. In fact, -XX:hashCode should return 1 when asking for an Object's hashCode, but, as far as I've seen, I believe that when Objects have more properties inside of them, such as Inner Classes, Methods, etc, the actual hashCode returned isn't 1, as it should be, or at least as docs say...

So, whether it's a bug or not, I make use of that property made available by -XX:hashCode=2, and you could even see it by yourself:

Just create a random Object, and if you wish, add some extra ingredients to it, such as some methods, maybe one inner class, or some random values...and invoke hashCode() on it to see what value you get!

[JasonFengJ9]

@guekho64 what Hotspot JVM version/platform do you run against the program?

[guekho64]

@JasonFengJ9 sorry for the late response. Here's what java -version outputs:

openjdk version "1.7.0_201"
OpenJDK Runtime Environment (Zulu 7.25.0.5-linux64) (build 1.7.0_201-b02)
OpenJDK 64-Bit Server VM (Zulu 7.25.0.5-linux64) (build 24.201-b02, mixed mode)

Although I've been using different JVM's since I started with this project of mine, including:

• OpenJDK 7/8
• Zulu JDK 7/8
• OpenJDK 8 (w/Shenandoah GC Patches)
• Oracle's JDK 7/8
• OpenJDK 8 (w/Eclipse J9's JVM)

And I haven't seen nothing out of usual when running my program with the discussed JVM option...

In fact, the program runs even better with Eclipse J9's JVM than with Hotspot, but there's this little detail that keeps me from using it for my project: The lack of support from this JVM for using the -XX:hashCode option, or a way to force the JVM to not scramble hashCodes every time the program is executed.

I know that there are many reasons to not preserve hash codes from execution to execution, like security and alike, and as such, I see 3 possibilities to solve this particular issue:

• Include support for -XX:hashCode JVM option, so user can decide whether or not hashCodes should change. (Like Hotspot does)

• Provide patches for users who might want this functionality...

And the last one:

• If none of the above is viable, maybe just giving out some clues on how to modify the source code should suffice, so people could just create custom builds of the JVM

Anyways, this is a personal project, so, in case that Eclipse OpenJ9 developers reject having official support for this feature, I can deal with it, just having a custom build of this JVM would be enough for me...although It may be useful for someone else, but I recognize that not always is that easy to add random features requested by people...

Another option could be to go back to Hotspot, but It'll be hard now that I know about this new virtual machine implementation, _sigh..._

[JasonFengJ9]

@guekho64 glad to hear that "In fact, the program runs even better with Eclipse J9's JVM than with Hotspot".
To preserve hash codes from execution to execution, a quick thought is to provide java.util. Random alike mechanism to generate pseudorandom numbers within java.lang.Object.hashCode() via natives.
Will let you know when such a patch is available for test.

[guekho64]

Thanks for letting me know! I'll look forward to it.

Something else...

Just as a side note: I'm no expert about this JVM, but I bet that a new JVM Option would be the way to go rather than a new method, 'cause that way you would avoid modifying the Object class, not to mention that the new method would be an exclusive method from this virtual machine, and code using this new method will have problems finding it if ever run on another machine, with a chance that it'll be running a Hotspot based JVM...

(Maybe options go here? https://github.com/eclipse/openj9/blob/838c53334b93b636da3019586170a3bc200647a2/runtime/oti/jvminit.h)

Another option that I didn't take in account earlier could be to add a special annotation, so objects that must have a constant hash between executions will be marked with this annotation. Obviously this isn't so "friendly" in terms of compatibility, as the annotiation would be available only in this JVM...

_I've always been worried about compatibility issues, as I mostly work with API's and extensions, where every piece of code should work flawlessly without causing problems with someones else's code, so I apologize If i'm asking too much about this. I trust that, whatever decision is taken, it'll be the most efficient one for the project_

[JasonFengJ9]

@guekho64 regarding the side note, you shouldn't have to modify the program to invoke a new method, either a -XX:hashCode JVM option or a system property will be needed to use the feature requested.
Note: a patch will be a jar file (for a java only solution) or a changeset (native code modification) to build a custom OpenJ9 JVM.
Thanks for your trust on OpenJ9 project!

[JasonFengJ9]

@guekho64 considering this is your personal project, I would propose a Java only patch to generate hash codes via java.util.Random within java.lang.Object. This function can be enabled via a system property preserveHashCode.

Important note: This patch doesn't apply to String objects which are handled differently by natives (clarification: String hashcode is calculated based on the String content hence its hashcode already is preserved from execution to execution). In addition footprint is going to be increased and the performance might be affected as well.

If your program can't tolerate those limitations, a native implementation need to be investigated instead.

Attached jar file is for OpenJ9 Java 8 and can be applied via -Xbootclasspath/p:/path/to/patch-openj9-java8.zip along with -DpreserveHashCode=true.
patch-openj9-java8.zip

[guekho64]

Thanks for putting your own effort into this!

I'm gonna try this patch as soon as possible, since I'm not in my developing environment right now, but I'll let you know if this worked at all or not.

How hard would it be to implement this behavior for String's too? As I said before, I use some Strings alongside other factors to determine which Object is which, and they're really useful because they contain the name of the class who invoked the method, and those names have very little chances to be repeated by someone else (Unless it's done on purpose...), so you could say that this acts as a way to confirm (Even more) if an Object it's the one being search...

A little hit on performance shouldn't be something to worry about, unless it _really_ affects the overall speed.

If necessary, I offer you my help to try to implement this.

[JasonFengJ9]

@guekho64 no problem, it's our job to help out OpenJ9 usages, :)
Just to clarify about String hashcode, actually it was calculated based on the String content hence its hashcode already is preserved from execution to execution even without this patch.
So please give this patch a try and see if it unblocks your program from adapting OpenJ9 jvm.
I don't believe the speed will be a big hit but you will find out if it is noticeable for your program.

From the specification perspective (https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Object.html#hashCode()), _This (hashcode) integer need not remain consistent from one execution of an application to another execution of the same application._ , preserving hashcode from execution to execution is not required by Java spec.

Note: I don't expect this patch to be accepted into OpenJ9 repo due to the footprint increasing. A proper implementation would be made within natives.

OpenJ9 project always welcome contributions, big or small. If you would like to take this implementation, I (and others) will help. Thanks for your interest.

[guekho64]

@JasonFengJ9

Just a few seconds ago I have just tested my program alongside the proposed patches and...it didn't work, unfortunately.

it didn't work wether I was asking for Objects or other custom types hashCode(). In the meantime I'm gonna try to search for the Hotspot implementation and try to adapt it here, if possible.

Maybe I should open another issue for this...

Also, ironically, because the program is way faster than before, a new problem has arised:

Pointer: 00000000FFFFFFFF has is not object aligned (to 8 bytes) 
00:00:04.191 0x7fb3940b5900    j9mm.141    *   ** ASSERTION FAILED ** at ./MarkingScheme.hpp:70: ((false))

Brief Explanation: This problem appears after the program exits sucessfully, and immediatly I run the program again, within the next 1 || 2 seconds, and appears at random phases of the program. Maybe due to the use of -Xbootclasspath/p:path? It's the first time I see this error.

I guess that, maybe, there's a race condition hidden somewhere? Or somehow the JVM didn't release the memory used in the last process, and the OS reassigns the same memory area to the new process, failing due to the unclean state of that specific area?

openjdk version "1.8.0_192"
OpenJDK Runtime Environment (build 1.8.0_192-b12_openj9)
Eclipse OpenJ9 VM (build openj9-0.11.0, JRE 1.8.0 Linux amd64-64-Bit 20181107_88 (JIT enabled, AOT enabled)
OpenJ9   - 090ff9dcd
OMR      - ea548a66
JCL      - b5a3affe73 based on jdk8u192-b12)

[JasonFengJ9]

@guekho64 any chance you could share a testcase to show that JVM doesn't preserve the hash codes from execution to execution?

Yes, please open another issue for the assertion failure.

Please be cautious about searching for Hotspot implementation, OpenJ9 was (and still is) a clean room JVM implementation which means no code/algorithm can be copied from Hotspot.

[guekho64]

@JasonFengJ9

The next piece of code yields the next results:

Code

package misc.testing;

import static java.lang.System.out;
import static misc.testing.Main.Local.Methods.getDate;

import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;

public final class Main {
    public static final class Local {
        public static final class Methods {
            public static final String getDate () {
                final DateTimeFormatter formatter = DateTimeFormatter.ofPattern("MM-dd-yyyy HH:mm:ss");
                return LocalDateTime.now().format(formatter);
            }
        }
        public static final class Types {
            public static final class CustomizedObject {
                private final int placeholderInteger;
                private final String placeholderString;
                private final Object placeholderObject;
                public CustomizedObject () {
                    super();
                    this.placeholderInteger = 10;
                    this.placeholderString = "string";
                    this.placeholderObject = new Object();
                }
            }
        }
    }
    public static final void main (final String... arguments) {
        final Object normalObject = new Object();
        final Local.Types.CustomizedObject customizedObject = new Local.Types.CustomizedObject();
        out.println(normalObject.hashCode());
        out.println(customizedObject.hashCode());
        out.print(getDate());
    }
}

Results:

1155869325
155629808
01-04-2019 22:04:34

431529176
1429008869
01-04-2019 22:08:28

155629808
1429008869
01-04-2019 22:08:44

1155869325
155629808
01-04-2019 22:09:22

These results were obtained using the patches alongside with:

openjdk version "1.8.0_192"
OpenJDK Runtime Environment (build 1.8.0_192-b12_openj9)
Eclipse OpenJ9 VM (build openj9-0.11.0, JRE 1.8.0 Linux amd64-64-Bit 20181107_88 (JIT enabled, AOT enabled)
OpenJ9   - 090ff9dcd
OMR      - ea548a66
JCL      - b5a3affe73 based on jdk8u192-b12)

[JasonFengJ9]

@guekho64 did you run the program with -DpreserveHashCode=true?

[JasonFengJ9]

@guekho64 I am not able build the original code snippet due to missing class misc.testing.Main.Local.Methods.getDate. Commenting out the class in question (along with its references), the hashcode output appears same from execution to execution as following:

openjdk version "1.8.0_192"
OpenJDK Runtime Environment (build 1.8.0_192-b12_openj9)
Eclipse OpenJ9 VM (build openj9-0.11.0, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20181107_95 (JIT enabled, AOT enabled)
OpenJ9   - 090ff9dcd
OMR      - ea548a66
JCL      - b5a3affe73 based on jdk8u192-b12)

1915244828
226796111

Could you supply a testcase without dependency (maybe attach the .class file as well)?

[guekho64]

@JasonFengJ9

Wow, such a mess right here, heh. In fact, the "missing" class is just an inner class, as the code shows, and the "depency" is supplied within the code, so I don't undertand what's going wrong at all...and getDate isn't a class at all, it's a method, lol.

Maybe Eclipse is doing something weird when compiling the code? 'Cause I use that IDE for my tests. And it works just fine by copying & pasting the above snippet in a new class...

And yes, I was using the new -DpreserveHashCode=true option while doing my tests, so this situation is quite...odd.

Fortunately, for odd situations, odd solutions.

Solution?

As I was seeing that accomplishing this was way harder than I initially thought, then I came up with a new way to verify objects, without using hash codes at all. In fact, ignoring them completely:




Here's my workaround:

Please note that whenever you add and/or remove an element from a `List`, and then extract a SHA-256 string from it, the resulting String will change, and if the same `List` is restored to a previous state, it'll also recover its old SHA-256 key.

SHA-256 strings also get influenced somehow if the name of the file is changed, but restoring previous file name will also restore previous results...

HashingTest.java
HashingTest.jar

[JasonFengJ9]

@guekho64 just realize that I modified the package name when coping the testcase code which caused The import misc.testing.Main cannot be resolved error. I do apologize not checking the code thoroughly before assuming the _class is missing_. (Honestly I didn't expect it imports a class/method within itself).
Now I run the original testcase and it still works as expected in a few runs with following result along with JVM version:

openjdk version "1.8.0_192"
OpenJDK Runtime Environment (build 1.8.0_192-b12_openj9)
Eclipse OpenJ9 VM (build openj9-0.11.0, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20181107_95 (JIT enabled, AOT enabled)
OpenJ9   - 090ff9dcd
OMR      - ea548a66
JCL      - b5a3affe73 based on jdk8u192-b12)

1915244828
226796111

Good to hear that you applied -DpreserveHashCode=true earlier, but it is not clear why your environment got inconsistent result.

Anyway, it appears you find a solution without relying on Object.hashCode(). Hope your program works well with OpenJ9 JVM.

P.S. It won't be a trivial implementation to preserve hashcode from execution to execution, since this function is not required by Java spec, it probably won't be a priority item either.

So if you agree, can this issue be closed?

[guekho64]

@JasonFengJ9

"Good to hear that you applied -DpreserveHashCode=true earlier, but it's not clear why your environment got inconsistent results" ~ Yes, I thought the same. I have no idea why your patch didn't work as expected on my side...

In fact, that's why the code outputs the date each time is executed, to show that each time the code was run, it printed a totally new value. Maybe the answer will never be found...

---

Anyways, fortunately I found a way to overcome my dependency on -XX:hashCode=2, and, for the sake of knowledge, I left a workaround within my last comment, so I hope nobody else has to depend on this kind of obscure VM options...

In fact, I did so because it was my last ressort while trying to solve my problem, but initially I didn't expect to migrate away from Hotspot...

_Then let's close this issue once for all!_

PS: I'm just curious if the hash printed by my test, in my environment, would be the same if a different computer executed (Or compiled & executed) the same code...

[JasonFengJ9]

@guekho64 I was able to get same result when running same .class binary at two different systems.

Output

Machine 1 - a cloud machine image

Linux rhel7x64xxx 3.10.0-862.9.1.el7.x86_64 #1 SMP Wed Jun 27 04:30:39 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

1915244828
226796111
01-06-2019 22:20:34

1915244828
226796111
01-06-2019 22:20:39

Machine 2 - a physical machine
Linux rhel6x64xxx 2.6.32-279.22.1.el6.x86_64 #1 SMP Sun Jan 13 09:21:40 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

1915244828
226796111
01-06-2019 22:20:51

1915244828
226796111
01-06-2019 22:20:53

openjdk version "1.8.0_192"
OpenJDK Runtime Environment (build 1.8.0_192-201812170453-b12)
Eclipse OpenJ9 VM (build master-f9ab45bfe, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20181217_142 (JIT enabled, AOT enabled)
OpenJ9 - f9ab45bfe
OMR - 0d514697
JCL - 7105b0a753 based on jdk8u192-b12)

As per https://docs.oracle.com/javase/8/docs/api/java/util/Random.html
_If two instances of Random are created with the same seed, and the same sequence of method calls is made for each, they will generate and return identical sequences of numbers. In order to guarantee this property, particular algorithms are specified for the class Random. Java implementations must use all the algorithms shown here for the class Random, for the sake of absolute portability of Java code._
It doesn't explicitly state that this is true for two instances in different JVM executions though my limited experiments show that appears the case.

One of possibilities for getting inconsistent results is that J9VMInternals.fastIdentityHashCode(this) is invoked when System.propertiesInitialized is false at early stage of VM startup before java.util.Random can be instantiated, i.e., any object requesting a hashcode before Random object instantiation is going to get inconsistent results. It is a timing thing. My hope was that the objects you are interested are created after System.propertiesInitialized is set to true and Random object has been created.
Anyway I am glad you find a solution within your implementation.

[guekho64]

Thanks for helping me solve this one, I really appreciate your time spent here. I hope that Eclipse J9 will continue growing and getting better (Even better than it alredy is).

See ya until the next issue.