Gson: Gson doesn't deserialise Long numbers correctly
Gson shouldn't cast a number to a Double if a number does not have decimal digits. It's clearly wrong.
public class GsonVsJackson {
public static void main(String[] args) throws Exception {
testGson();
System.out.println("========================");
testJackson();
}
public static void testGson() {
System.out.println("Testing Gson 2.8");
Gson gson = new Gson();
HashMap<String, Object> newTest = new HashMap<>();
newTest.put("first", 6906764140092371368L);
String jsonString = gson.toJson(newTest);
System.out.println(jsonString); // output ok: {"first":6906764140092371368}
Map<String, Object> mapFromJson = gson.fromJson(jsonString, Map.class);
Number numberFromJson = (Number) mapFromJson.get("first");
System.out.println(numberFromJson.getClass() + " = " + numberFromJson); // java.lang.Double val 6.9067641400923709E18
long longVal = numberFromJson.longValue();
System.out.println(longVal); // output rounded: 6906764140092370944
}
public static void testJackson() throws Exception {
System.out.println("Testing Jackson");
ObjectMapper jackson = new ObjectMapper();
HashMap<String, Object> newTest = new HashMap<>();
newTest.put("first", 6906764140092371368L);
String jsonString = jackson.writeValueAsString(newTest);
System.out.println(jsonString); // output ok: {"first":6906764140092371368}
Map<String, Object> mapFromJson = jackson.readValue(jsonString, Map.class);
Number numberFromJson = (Number) mapFromJson.get("first");
System.out.println(numberFromJson.getClass() + " = " + numberFromJson); // java.math.BigInteger = 6906764140092371368
long longVal = numberFromJson.longValue();
System.out.println(longVal); // output OK: 6906764140092371368
}
}
Kind Regards,
Daniele
danieleguiducci
All 34 comments
It's not a bug, and gson does it fully legit: JSON does not distinguish between integers or floats, and does not care the size of numbers. Numerics in JSON are just numbers, and java.lang.Double is the best and largest primitive-counterpart candidate to hold a JSON number.
If you need a long value at a call-site, then just call its longValue() method. If, for any particular reason, you need a behavior you are talking about, then you have to implement a custom type adapter factory. Say, something like this (not sure if it's implemented right, though):
final class BestNumberTypeAdapterFactory
implements TypeAdapterFactory {
private static final TypeAdapterFactory bestNumberTypeAdapterFactory = new BestNumberTypeAdapterFactory();
private static final TypeToken<Number> numberTypeToken = new TypeToken<Number>() {
};
private BestNumberTypeAdapterFactory() {
}
static TypeAdapterFactory getBestNumberTypeAdapterFactory() {
return bestNumberTypeAdapterFactory;
}
@Override
public <T> TypeAdapter<T> create(final Gson gson, final TypeToken<T> typeToken) {
if ( Number.class.isAssignableFrom(typeToken.getRawType()) ) {
final TypeAdapter<Number> delegateNumberTypeAdapter = gson.getDelegateAdapter(this, numberTypeToken);
final TypeAdapter<Number> numberTypeAdapter = new NumberTypeAdapter(delegateNumberTypeAdapter).nullSafe();
@SuppressWarnings("unchecked")
final TypeAdapter<T> typeAdapter = (TypeAdapter<T>) numberTypeAdapter;
return typeAdapter;
}
return null;
}
private static final class NumberTypeAdapter
extends TypeAdapter<Number> {
private final TypeAdapter<Number> delegateNumberTypeAdapter;
private NumberTypeAdapter(final TypeAdapter<Number> delegateNumberTypeAdapter) {
this.delegateNumberTypeAdapter = delegateNumberTypeAdapter;
}
@Override
public void write(final JsonWriter out, final Number value)
throws IOException {
delegateNumberTypeAdapter.write(out, value);
}
@Override
public Number read(final JsonReader in)
throws IOException {
final String s = in.nextString();
return parsers.stream()
.map(parser -> parser.apply(s))
.filter(Objects::nonNull)
.findFirst()
.get();
}
}
private static <N extends Number> Function<String, N> parseOnNull(final Function<? super String, ? extends N> parser) {
return s -> {
try {
return parser.apply(s);
} catch ( final NumberFormatException ignored ) {
return null;
}
};
}
private static final ImmutableList<Function<? super String, ? extends Number>> parsers = ImmutableList.<Function<? super String, ? extends Number>>builder()
.add(parseOnNull(Byte::parseByte))
.add(parseOnNull(Short::parseShort))
.add(parseOnNull(Integer::parseInt))
.add(parseOnNull(Long::parseLong))
.add(parseOnNull(Float::parseFloat))
.add(parseOnNull(Double::parseDouble))
.build();
}
However, it can only work if your code can tell Gson to deserialize numbers. This won't work:
@SuppressWarnings("unchecked")
final Map<String, Object> mapFromJson = jackson.readValue(jsonString, Map.class);
But this will:
private static final TypeToken<Map<String, Number>> stringToNumberMapTypeToken = new TypeToken<Map<String, Number>>() {
};
...
final Map<String, Object> mapFromJson = gson.fromJson(jsonString, stringToNumberMapTypeToken.getType());
lyubomyr-shaydariv
on 24 May 2017
I'm not saying either that is not legit or it's a bug, i'm saying that this can be done better without loosing precision, like Jackson does :)
danieleguiducci
on 25 May 2017
@danieleguiducci Ah, I didn't notice the precision problems, sorry. Then you have just to use a type adapter like the above (exclude the Byte, Short, Integer, and Float parsers) that would implement the best number parsing strategy for you. :) IMHO, using such an adapter (as well as for Gson) would cause performance slowdown, so, I'd say you have to be free to choose what to pay for.
lyubomyr-shaydariv
on 26 May 2017
Thank you @lyubomyr-shaydariv , I already solved the issue by replacing Gson with Jackson.
I think that the correctness of an algorithm should has an higher priority over the performance.
I wrote this post hopping to help the Gson team to improve the quality of their library.
Have a good weekend!
Kind regards,
Daniele
danieleguiducci
on 26 May 2017
Fighting with the same issue at the moment. The task is pretty simple - log responses with pretty print - but can't use GSON as it mangles unix timestamps. Other solutions have their own culprits (like JSONObject who escapes forward slashes without any way to stop it) but at least I don't have 2 converted to 2.0 and timestamps look normal not like 1.47668631713E12.
mikhailmelnik
on 8 Sep 2017
It's not a bug, JSON does not distinguish between integers or floats, and does not care the size of numbers.
Gson is created for java not for javascript. Java does care the data type and size. JSON is not used for javascript only. Hotdog is not dog, there is no need to stick to its literal meaning. Number adapter won't work for Object type, e.g., ArrayList This is an old issue existed for many years. It's easy for to fix it. I hope it will be fixed in 2018. Other json libs do not have this bug (I think it is).
zenglian
on 3 Mar 2018
@zenglian Do you plan to make a pull request in original google repo?
boyan-velinov
on 13 Apr 2018
@zenglian Oh, you've quoted my comment not referring me directly.
Java does care the data type and size.
Yes, it does but you should not write heuristics at the adapter side trying to pick up the "best" type you can find for a particular number literal. Let me ask you: is 1
- a
Byte - a
Short - an
Integer - a
Long - a
Float - a
Double BigIntegerBigDecimalAtomicIntegerAtomicLong- or any other
Numberfrom a third-party libraries like Google Guava, Apache Commons, etc?
Please how could you know the original type?. Let your mappings do and should declare the type themselves. For example, when you deserialize a list of users for List<Object>, why you don't write a heuristics type adapter to determine that _this object may look like a user_ having a list of LinkedTreeMap instances? Therefore ArrayList<Object> is wrong for your case and for mine. Again, this is NOT a bug. Please read the discussion at #1267.
lyubomyr-shaydariv
on 13 Apr 2018
It makes more sense to guess 1 as an integer number than a floating number. I believe most people will make such assumption. And in java int can be automatically cast to double.
List
zenglian
on 13 Apr 2018
I agree with the comment above and naturally we, as people, consider 1 as integer, not as rational or real.
The same should apply for the code as well. When there is a possibility to parse 1 it should be done to the most commonly used type, namely Integer.
Of course, the value can be greater then Integer.MAX_VALUE, so in this case, it could be parsed to Long. And again if it is greater then Long.MAX_Value, the result will be Double.
At least fromJson() and toJson() applied consequently on one String, should result in the same string. Now, this is not the case.
Let’s give an example:
Code:
public static void main(String[] args) {
String jsonString = "{\"a\":1}";
System.out.println("intput" + jsonString);
Map<String, Object> jsonMap = new Gson().fromJson(jsonString, new TypeToken<Map<String, Object>>() {
}.getType());
System.out.println("output" + new Gson().toJson(jsonMap));
}
Result:
intput{"a":1}
output{"a":1.0}
boyan-velinov
on 13 Apr 2018
And in java int can be automatically cast to double.
Only if these are primitives. You cannot cast java.lang.Double to java.lang.Integer and vice versa. That's why java.lang.Number is designed to be _convertible_ to main numeric types in Java.
Seriously, why is 1 not a java.lang.Byte by this approach?
lyubomyr-shaydariv
on 13 Apr 2018
This is not a debate contest, so do not try to find my logic hole. Instead, try to find out what is the user requirements. When the input is {"a":1}, guess user most likely wants integer or float.
zenglian
on 13 Apr 2018
@zenglian It would be nice if user requirements could always align with how JSON is designed. Please re-read the linked PR discussion explaining the cons of such a change and why it's not a bug. Thank you.
lyubomyr-shaydariv
on 13 Apr 2018
Since gson is for Java, I think it has to do some kind of adjustment. We do not need byte, but "long" should be fine for most integers (but it over flows for big integer -- it should be rare).
Maybe both sides can accept the solution that user can custom Object deserializer.
zenglian
on 14 Apr 2018
If we are after user requirements my only one was to have json_in and json_out in json_in -> Object -> json_out conversion identical while current implementation completely breaks it.
mikhailmelnik
on 14 Apr 2018
@lyubomyr-shaydariv Please note that in Java Double d = 1 is illegal while Double d = 1.0 is legal.
Below is from intellij idea.
Again, JSON is not designed for Java while gson does.
zenglian
on 15 Apr 2018
I do not want to follow the "what is a number and what not" discussion. But I have the same problem.
Read: 1513206000000
Write: 1.513206E12
This is not the same, not mathematical, not string, not what-ever-type. No philosophy, please.
The big problem is, that the code does even not follow the TypeAdapterFactory pattern.
If you have a look at the TypeAdapters class, you will be surprised:
There is a "TypeAdapterFactory INTEGER_FACTORY". This could be replaced.
But there is no "LONG_FACTORY". Why not?
So the TypeAdapter.LONG could not be replaced like the other TypeAdapterFactories.
You should follow your own patterns.
Why is there not LONG_FACTORY?
The TypeAdaper LONG is hard-coded in GSON.class. Not implemented via factory.
Next error: in GSON.longAdapter():
It is implemented like this:
If there is a LongSerializationPolicy.DEFAULT then use the default "TypeAdapter.LONG", if not than use a hard coded TypeAdapter using String. This is wrong, there is a LongSerializationPolicy.STRING. But there could also be a LongSerializationPolicy.WHATEVER. But this will never work, because the code does not even realize that there is a "STRING" Policy.
By the way: The serialize-method of the LongSerializationPolicy is not even used. Instead it is hard-coded in GSON.class. If there is a LongSerializationPolicy, you should use the serialize method of the matching policy. Not your own implementiation. Or kick the Policy methods, as dead code.
So: Do not philosphy about "what is a number". Think about your own patterns and code.
This IS a bug.
berndwaibel
on 25 Jan 2019
@berndwaibel
Long is a special case in JSON as JSON doesn't support the entire range of Java long. Hence, we have the ability to serialize/deserialize long as a number or a String.
I am not even sure what is it that you are complaining about as a bug. Can you show some code that illustrates the problem?
The original poster is expecting Gson to take a number and then try to convert it to long or int, if possible. We actually used to do that in Gson 1.x but decided to change behavior in Gson 2.x.
The primary reason is that the output type shouldn't change based on the data. Otherwise, the client code gets arbitrary ClassCastExceptions.
inder123
on 26 Jan 2019
The JsonReader does convert LONG to decimal, loosing digits.
The input and output are different.
I append a java class to show the meaning, you can run it, and see that input and output are different, and that by this GSON puts in errors when using the JsonReader.
It shows that the reader does, by using decimal, loose digits.
The biggest long: 9223372036854775807 is NOT equal to 9.223372036854776E18.
The other things are architecture things (LongSerializationPolicy, TypeAdapterFactory).
They are nice architecture patterns, but not usable, as it is hard coded not to use them.
It is not possible to define own TypeAdapterFactory or LongSerializationPolicy, cause hard-coded values are used. The patterns died using the "performance" argument.
The attached class is from an real example, as it reads Atlassian Jira JSON Files.
Need to zip it, as .java is not accepted.
GsonForJira.zip
berndwaibel
on 26 Jan 2019
What happens if you specify types instead of using raw ArrayList instances? Ie. replace this:
public ArrayList projects;
public ArrayList versions; // optional
with this:
public ArrayList<Project> projects;
public ArrayList<Version> versions; // optional
swankjesse
on 26 Jan 2019
I normally would not even describe the imported class (as this is Jira, and I do not even know the whole Jira class structure). And most values are optional. I thought, for using JSON, I do not need to describe a whole Java Class hierarchy, or? The class is only necessary for creating the first JSON file.
I think I tried, but got an different result (if i remember):
If I do this, I get empty fields, cause then the whole attributes will exported (empty or not).
So the input and output differs again.
berndwaibel
on 26 Jan 2019
The thing GSON is missing is a precise representation of JSON.
In Scala's play-json, I have never had this problem. When I parse JSON and don't know / can't specify a deserialized type, I get a JsValue, it can be a JsObject, JsArray, JsNull, JsNumber, JsString. JsNumber's value is a BigDecimal - it precisely represents JSON and can be round-tripped without loss. JsObject is a Map<String, JsValue>, JsArray is a List<JsValue>. This is very straightforward when I consider JSON ITSELF to be my basic data type. Especially when writing middleware that just moves JSON values around and maybe wants to pick out one value while the rest of the object is of arbitrary structure.
Let me deserialize to the json object model. I know it's JSON. I want to deal with it as JSON.
Alternatively, can someone tell me how to force it to default to deserializing numbers as BigDecimal instead of stupidly guessing Double (which is pretty much always going to lose information)?
jkoberg
on 30 Jan 2019
Gson has that already.
JakeWharton
on 30 Jan 2019
@Jake: The comment "Gson has that already." is not really helpful. As we shown, it is not using it.
I delivered code, try it.
And than tell me why is the result an decimal number?
berndwaibel
on 30 Jan 2019
I am open to any help which shows the error in my code, if I use JSON without building a class tree.
berndwaibel
on 30 Jan 2019
I wasn't replying to you. I was replying to the person who derailed the thread asking for a feature that already exists.
JakeWharton
on 30 Jan 2019
Ok. To whom are you replying, and which feature does GSON already have?
berndwaibel
on 30 Jan 2019
The person whose comment was directly prior to mine.
On Wed, Jan 30, 2019 at 3:34 PM berndwaibel notifications@github.com
wrote:
Ok. To whom are you replying, and which feature does GSON already have?
—
You are receiving this because you commented.Reply to this email directly, view it on GitHub
https://github.com/google/gson/issues/1084#issuecomment-459098679, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAEEEWiTosPTURmEFcWxzzpYzyq2USGUks5vIgHxgaJpZM4Nj87p
.
JakeWharton
on 30 Jan 2019
Oh, ok, so you may answer this:
can someone tell me how to force it to default to deserializing numbers as BigDecimal instead of guessing Double (which is pretty much always going to lose information)?
(I left out the unnecessary words ...)
berndwaibel
on 30 Jan 2019
Probably not possible unless Gson lets you override the type adapters for the built-in types. If it did you could write a type adapter to read it as a string and then convert to whatever you wanted.
JakeWharton
on 30 Jan 2019
This is definitely a bug. Also there is no easy way out. As many many others have pointed out. They were forced to switch library. So either Google decides to fix this issue or people will use other libraries.
dexception
on 31 Aug 2019
This is a serious bug. This should not be default behavior.
mvkarthikeyan
on 5 Dec 2019
hello from 2020, this bug still exists!!! Google, Google.... :(
jet-snowman
on 5 Apr 2020
I don't think this is bug, but the problem is that the NumberAdapter is not replaceable as the default adapters are hardcoded in UnmodifiableList. Ideally you should be able to remove current NumberAdapter and use your one. In my case, I needed this specifically:
JsonToken.NUMBER -> {
val value = reader.nextString()
return if (value.contains(".")) {
value.toDouble()
} else {
value.toLong()
}
}
To achieve so, I use reflection to replace the adapter internally - it is super-ugly and bad, but it works:
class Gson {
private val gson = com.google.gson.GsonBuilder().create()
fun <T : Any> fromJson(payload: String, c: KClass<T>) = try {
gson.fromJson(payload, c.java)
} catch (e: Exception) {
throw JsonSyntaxException("Invalid JSON: ${payload.truncate(128)}", e)
}
fun toJson(payload: Any) = gson.toJson(payload)
class LongObjectTypeAdapter(private val gson: com.google.gson.Gson) : TypeAdapter<Any>() {
override fun read(reader: JsonReader): Any? {
val token = reader.peek()
when (token) {
JsonToken.BEGIN_ARRAY -> {
val list = ArrayList<Any?>()
reader.beginArray()
while (reader.hasNext()) {
list.add(this.read(reader))
}
reader.endArray()
return list
}
JsonToken.BEGIN_OBJECT -> {
val map = LinkedTreeMap<String, Any?>()
reader.beginObject()
while (reader.hasNext()) {
map.put(reader.nextName(), this.read(reader))
}
reader.endObject()
return map
}
JsonToken.STRING -> return reader.nextString()
JsonToken.NUMBER -> {
val value = reader.nextString()
return if (value.contains(".")) {
value.toDouble()
} else {
value.toLong()
}
}
JsonToken.BOOLEAN -> return java.lang.Boolean.valueOf(reader.nextBoolean())
JsonToken.NULL -> {
reader.nextNull()
return null
}
else -> throw IllegalStateException()
}
}
override fun write(out: JsonWriter, value: Any?) {
if (value == null) {
out.nullValue()
} else {
val typeAdapter = gson.getAdapter(value.javaClass)
if (typeAdapter is LongObjectTypeAdapter) {
out.beginObject()
out.endObject()
} else {
try {
typeAdapter.write(out, value)
} catch (e: Exception) {
// we don't care
out.nullValue()
}
}
}
}
}
companion object {
private val FACTORY = object : TypeAdapterFactory {
override fun <T> create(gson: com.google.gson.Gson, type: TypeToken<T>): TypeAdapter<T>? {
return if (type.rawType == Any::class.java) LongObjectTypeAdapter(gson) as TypeAdapter<T> else null
}
}
init {
try {
val field = ObjectTypeAdapter::class.java.getDeclaredField("FACTORY")
val modifiersField = Field::class.java.getDeclaredField("modifiers")
modifiersField.isAccessible = true
modifiersField.setInt(field, field.modifiers and Modifier.FINAL.inv())
field.isAccessible = true
field.set(null, FACTORY)
} catch (e: Exception) {
throw IllegalStateException(e)
}
}
}
}
knyttl
on 6 May 2020
Related issues
GoogleCodeExporter
·
32Comments
priyankajagtap18
·
14Comments
GoogleCodeExporter
·
16Comments
kramer
·
26Comments
GoogleCodeExporter
·
31Comments
Most helpful comment
It's not a bug, JSON does not distinguish between integers or floats, and does not care the size of numbers.Gson is created for java not for javascript. Java does care the data type and size. JSON is not used for javascript only. Hotdog is not dog, there is no need to stick to its literal meaning. Number adapter won't work for Object type, e.g., ArrayList
This is an old issue existed for many years. It's easy for to fix it. I hope it will be fixed in 2018. Other json libs do not have this bug (I think it is).