Ecto: Error when loading specific values in fields of type {:map, :float}

[tallysmartins]

Environment

• Elixir version (elixir -v): Elixir 1.6.4
• Database and version (PostgreSQL 9.4, MongoDB 3.2, etc.): PostgreSQL 9.6.7
• Ecto version (mix deps): master branch
• Database adapter and version (mix deps): postgrex 0.14.0-dev
• Operating system: Debian Jessie

Current behavior

When I have a schema with a field of type {:map, :array}, some values are parsed to exponential floats and are saved as integers in the database.

For example, %{"50" => 416500.0} fails, because it is saved as an integer 416500, even after Float.parse(416500), which returns an exponential form for it. I don't know if it has something to do with the jsonb fields, but for simple floatfields it works.

Float.parse("416500")
{4.165e5, ""}

I've added an example to ecto integration tests that shows the break point. See this commit https://github.com/tallysmartins/ecto/commit/5048199197ff2bd3e1480cd8286931f2d07cddde and the correspondent build on Travis-ci

     ** (ArgumentError) cannot load `%{"50" => 416500}` as type {:map, :float} for field `scores` in schema Ecto.Integration.Post
     code: assert [_] = TestRepo.all from p in Post, where: p.title not in []
     stacktrace:
       (ecto) lib/ecto/schema.ex:1794: Ecto.Schema.load!/5
       (ecto) lib/ecto/schema.ex:1742: Ecto.Schema.safe_load_zip/4
       (ecto) lib/ecto/schema.ex:1743: Ecto.Schema.safe_load_zip/4
       (ecto) lib/ecto/schema.ex:1730: Ecto.Schema.__safe_load__/6
       (ecto) lib/ecto/repo/queryable.ex:316: Ecto.Repo.Queryable.process_source/6
       (ecto) lib/ecto/repo/queryable.ex:191: anonymous fn/6 in Ecto.Repo.Queryable.preprocessor/3
       (elixir) lib/enum.ex:1294: Enum."-map/2-lists^map/1-0-"/2
       (ecto) lib/ecto/repo/queryable.ex:152: Ecto.Repo.Queryable.execute/4
       (ecto) lib/ecto/repo/queryable.ex:18: Ecto.Repo.Queryable.all/3
       integration_test/cases/repo.exs:44: (test)

Expected behavior

Save and load the field the same way as simple float field.

[wojtekmach]

It's curious that the integration test fails on pg but passes on mysql.

[tallysmartins]

@wojtekmach good point! but actually I didn't add a value for field to a test case for mysql, just for pg :thinking:

[wojtekmach]

@tallysmartins integration_test/cases/repo.exs and others are shared between different adapters, see e.g.: https://github.com/elixir-ecto/ecto/blob/master/integration_test/mysql/all_test.exs#L12

[michalmuskala]

It seems this is an issue with how postgresql handles JSON with floats in exponential notation.

# create table foo(map jsonb);
CREATE TABLE
# insert into foo(map) VALUES ('{"foo": 416500.0}');
INSERT 0 1
# insert into foo(map) VALUES ('{"foo": 4.165e5}');
INSERT 0 1
# SELECT * FROM foo;
        map
-------------------
 {"foo": 416500.0}
 {"foo": 416500}
(2 rows)

[josevalim]

@michalmuskala the worst part is that Postgres is semantically correct, there are no integers/floats in JSON. I guess we need to do casting here on Ecto side.

[michalmuskala]

Should we do for {:map, _} type the same thing we do for {:embed, _} in here? https://github.com/elixir-ecto/ecto/blob/master/lib/ecto/adapters/sql.ex#L47

[josevalim]

@michalmuskala if it fixes the bug, sure. :) We may need to do a couple extra things though.

[hoarf]

Any chance that this problem is also happening with {:array, :float}? I have a issue with an array of floats sometimes failing to be inserted. (Elixir 1.7.3, ecto, 2.2)