Julia: deepcopy fallback of objects with finalizers is broken

Created on 30 May 2016 · 15Comments · Source: JuliaLang/julia

Both on 0.4 and 0.5:

/tmp >> julia5
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.5.0-dev+4359 (2016-05-27 08:02 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit e11ff35* (3 days old master)
|__/                   |  x86_64-pc-linux-gnu

julia> y = deepcopy(BigFloat[1, 2])
2-element Array{BigFloat,1}:
/home/mauro/julia/julialang_upstream/deps/srccache/mpfr-3.1.3/src/get_str.c:153: MPFR assertion failed: size_s1 >= m

signal (6): Aborted
while loading no file, in expression starting on line 0
unknown function (ip: 0x7fdc3b909294)
unknown function (ip: 0x7fdc3b90a6d9)
mpfr_assert_fail at /home/mauro/julia/julialang_upstream/deps/srccache/mpfr-3.1.3/src/mpfr-gmp.c:305
mpfr_get_str_aux at /home/mauro/julia/julialang_upstream/deps/srccache/mpfr-3.1.3/src/get_str.c:153
mpfr_get_str at /home/mauro/julia/julialang_upstream/deps/srccache/mpfr-3.1.3/src/get_str.c:2518
regular_eg at /home/mauro/julia/julialang_upstream/deps/srccache/mpfr-3.1.3/src/vasprintf.c:1082
partition_number at /home/mauro/julia/julialang_upstream/deps/srccache/mpfr-3.1.3/src/vasprintf.c:1640
mpfr_snprintf at /home/mauro/julia/julialang_upstream/deps/srccache/mpfr-3.1.3/src/printf.c:169
string at ./mpfr.jl:857
show at ./mpfr.jl:868
unknown function (ip: 0x7fda363020a6)
jl_call_method_internal at /home/mauro/julia/julialang_upstream/src/julia_internal.h:88
jl_apply at /home/mauro/julia/julialang_upstream/src/julia.h:1388
#sprint#131 at ./strings/io.jl:36
#sprint#131 at ./strings/io.jl:36
jl_call_method_internal at /home/mauro/julia/julialang_upstream/src/julia_internal.h:88
jl_apply at /home/mauro/julia/julialang_upstream/src/julia.h:1388
#sprint at ./null:0
alignment at ./show.jl:1157
alignment at ./show.jl:1195
unknown function (ip: 0x7fda368bd64e)
jl_call_method_internal at /home/mauro/julia/julialang_upstream/src/julia_internal.h:88
print_matrix at ./show.jl:1316
print_matrix at ./show.jl:1294
unknown function (ip: 0x7fda368c00b9)
jl_call_method_internal at /home/mauro/julia/julialang_upstream/src/julia_internal.h:88
jl_apply at /home/mauro/julia/julialang_upstream/src/julia.h:1388
showarray at ./show.jl:1501
show at ./replutil.jl:4
unknown function (ip: 0x7fda368d80bd)
jl_call_method_internal at /home/mauro/julia/julialang_upstream/src/julia_internal.h:88
display at ./REPL.jl:114
unknown function (ip: 0x7fda368e05d6)
jl_call_method_internal at /home/mauro/julia/julialang_upstream/src/julia_internal.h:88
display at ./REPL.jl:117
unknown function (ip: 0x7fda368e4176)
jl_call_method_internal at /home/mauro/julia/julialang_upstream/src/julia_internal.h:88
display at ./multimedia.jl:153
unknown function (ip: 0x7fda368e6609)
jl_call_method_internal at /home/mauro/julia/julialang_upstream/src/julia_internal.h:88
print_response at ./REPL.jl:134
unknown function (ip: 0x7fda368e8408)
jl_call_method_internal at /home/mauro/julia/julialang_upstream/src/julia_internal.h:88
print_response at ./REPL.jl:121
unknown function (ip: 0x7fda368ea148)
jl_call_method_internal at /home/mauro/julia/julialang_upstream/src/julia_internal.h:88
#18 at ./REPL.jl:630
unknown function (ip: 0x7fda3690284c)
jl_call_method_internal at /home/mauro/julia/julialang_upstream/src/julia_internal.h:88
run_interface at ./LineEdit.jl:1570
unknown function (ip: 0x7fdc3862caca)
jl_call_method_internal at /home/mauro/julia/julialang_upstream/src/julia_internal.h:88
run_frontend at ./REPL.jl:881
run_repl at ./REPL.jl:166
unknown function (ip: 0x7fdc3daba19d)
jl_call_method_internal at /home/mauro/julia/julialang_upstream/src/julia_internal.h:88
_start at ./client.jl:364
unknown function (ip: 0x7fdc3865b994)
jl_call_method_internal at /home/mauro/julia/julialang_upstream/src/julia_internal.h:88
unknown function (ip: 0x401bf5)
unknown function (ip: 0x40167a)
unknown function (ip: 0x7fdc3b8f6740)
unknown function (ip: 0x4016c8)
Allocations: 2688926 (Pool: 2687706; Big: 1220); GC: 4
zsh: abort (core dumped)  julia5

However, below example behaves differently. On 0.5:

julia> y = deepcopy(BigFloat[1, 2]);

julia> y[2]
2.000000000000000000000000000000000000000000000000000000000000000000000000000000

julia> y[3]
ERROR: BoundsError: attempt to access 2-element Array{BigFloat,1} at index [3]
 in getindex(::Array{BigFloat,1}, ::Int64) at ./array.jl:311
 in eval(::Module, ::Any) at ./boot.jl:225
 in macro expansion at ./REPL.jl:92 [inlined]
 in (::Base.REPL.##1#2{Base.REPL.REPLBackend})() at ./event.jl:46

on 0.4:

julia> y = deepcopy(BigFloat[1, 2]);

julia> y[2]
2.000000000000000000000000000000000000000000000000000000000000000000000000000000

julia> y[3]
get_str.c:153: MPFR assertion failed: size_s1 >= m
... (same error as above)

Source

mauro3

Most helpful comment

To fix, we could probably refuse to deepcopy objects that have finalizers. Although really you should just structure your program not to depend on this function.

I would advocate instead that for objects with finalizers, deepcopy should serialize and then deserialize an object – if you can't do that for a type, then it's ok to fail. In general, deepcopy should have the same effect as serializing and deserializing something except that it's much more efficient.

StefanKarpinski on 1 Jun 2016

👍2

All 15 comments

Actually, something deeper than just display is amiss, as tests in ODE.jl fail when deepcopy is used:
https://github.com/JuliaLang/ODE.jl/pull/98. But I suspect if that above is fixed the rest should be fixed too. I will check.

mauro3 on 30 May 2016

Works for me on both 0.4 and 0.5.

julia> versioninfo()
Julia Version 0.5.0-dev+4393
Commit 15626c3 (2016-05-29 22:12 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin15.3.0)
  CPU: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, haswell)

dpsanders on 30 May 2016

Oh, how strange. It works sometimes and fails randomly...

dpsanders on 30 May 2016

To fix, we could probably refuse to deepcopy objects that have finalizers. Although really you should just structure your program not to depend on this function.

vtjnash on 30 May 2016

My understanding was that if I want to make sure that I do get a full copy of an arbitrary type, I ought to use deepcopy.

Anyway, should do something like x = hasfinalizer(y) : copy(y) ? deepcopy(y)? But is there a hasfinalizer function? Or should I just never use deepcopy?

mauro3 on 31 May 2016

Duplication of an entity that has indirect internal structure or makes indirect use of its own content are canonical examples of type characteristics where instances need to be 'deepcopy'ed to assure proper behavior and handling of both the original and the duplicate. The fix should be to deepcopy, so that it copy deeply.

If you want to augment the behaviour of copy to type-specialize so that types with initializers aor finalizers just know to pre-evaluate initializers and re-register finalizers at each copying -- ok, and its ok to just recognize and properly handle other things that would require deepcopies then remove deep copy because copy does the right thing always.

While there are two, copy and deepcopy, we should keep the symbols' semiotics in line with their meaning and their purposeful application. Deep copying has been a band-aid allowing many programming languages to keep copying fast for lots of things and to keep copying available for all. Julia's pre-execution dispatch specialization does give her the way to subsume deepcopy in copy without introducing overhead when copying stuff.

JeffreySarnoff on 31 May 2016

@JeffreySarnoff deepcopy has a different purpose than copy and should be kept separated, it's not just a band-aid. Consider e.g. wanting to save a Dict{T,Vector{S}} while keeping working on the original.

carlobaldassi on 31 May 2016

@carlobaldassi How is that different from copying the Dict{T,Vector{S}} into the stream of a newly opened file?

JeffreySarnoff on 31 May 2016

I consider the special case to be shallowcopy (copying the bowl without duplicating the soup) and deepcopy to be designer instructed copying.

JeffreySarnoff on 31 May 2016

@JeffreySarnoff Well, in one case you have an object in memory, in the other a file on disk. In any case that was just an example, in general copy is understood to be shallow and therefore not adequate for all cases.

carlobaldassi on 31 May 2016

👍1

To fix, we could probably refuse to deepcopy objects that have finalizers. Although really you should just structure your program not to depend on this function.

StefanKarpinski on 1 Jun 2016

👍2

Fix by Scott: https://github.com/ScottPJones/julia/tree/spj/fixbig

His comment on julia-dev:

I have the issue #16667 fixed for BigInt and BigFloat types, which is where 
this problem has already been seen in the wild.
Although it is not a general fix for deepcopy of types with finalizers, it 
should be performant, and it would be trivial to backport to v0.4.x (for 
v0.4.6).

If somebody can merge this in, and put it on the backport list for v0.4.x, 
that would be great. (it modifies one line in base/deepcopy.jl and adds 
tests in test/copy.jl)
https://github.com/ScottPJones/julia/tree/spj/fixbig

Thanks in advance.

mauro3 on 8 Jun 2016

Given this is all about finaliers that's almost certainly the wrong fix.