Julia: cfunction issue with FunctionWrappers

[tkoolen]

From https://github.com/yuyichao/FunctionWrappers.jl/issues/8. This appears to be an issue in Base. Sorry for not having a smaller repro, I'm a bit pressed for time.

Consider (from https://github.com/yuyichao/FunctionWrappers.jl/issues/8#issuecomment-413189252), with FunctionWrappers 1.0.0:

module Parametron2

using FunctionWrappers: FunctionWrapper

struct LazyExpression{F, A}
    f::F
    args::A
    LazyExpression(f::F, args...) where {F} = new{F, typeof(args)}(f, args)
end

# Evaluation
@inline evalarg(x) = x
@inline evalarg(x::LazyExpression) = x()
@inline (expr::LazyExpression{F, A})() where {F, A} = evalexpr(expr.f, expr.args)
@generated function evalexpr(f::F, args::Tuple{Vararg{Any, N}}) where {F, N}
    # equivalent to f(map(evalarg, args)...), minus the inference issues
    argexprs = [:(evalarg(args[$i])) for i = 1 : N]
    quote
        Base.@_inline_meta
        f($(argexprs...))
    end
end

# Wrapping
const WrappedExpression{T} = LazyExpression{FunctionWrapper{T, Tuple{}}, Tuple{}}

Base.convert(::Type{WrappedExpression{T}}, expr::LazyExpression) where {T} =
    LazyExpression(FunctionWrapper{T, Tuple{}}(expr))

function wrap(expr::Parametron2.LazyExpression)
    T = typeof(expr())
    convert(Parametron2.WrappedExpression{T}, expr)
end

end

module M

import Main.Parametron2

f1 = [1]
converted = Parametron2.wrap(Parametron2.LazyExpression(convert, Vector{Float64}, f1))
converted.f()

end

A possible result of pasting this into the REPL:

ERROR: TypeError: in CallWrapper, in cfunction, expected Core.Compiler.VarState(Core.Compiler.Const(false, false), false), got Array{Float64,1}
Stacktrace:
 [1] macro expansion at /home/twan/.julia/packages/FunctionWrappers/mJrYu/src/FunctionWrappers.jl:111 [inlined]
 [2] do_ccall at /home/twan/.julia/packages/FunctionWrappers/mJrYu/src/FunctionWrappers.jl:102 [inlined]
 [3] (::FunctionWrappers.FunctionWrapper{Array{Float64,1},Tuple{}})() at /home/twan/.julia/packages/FunctionWrappers/mJrYu/src/FunctionWrappers.jl:118
 [4] top-level scope at none:0

julia> versioninfo()
Julia Version 1.0.0
Commit 5d4eaca0c9 (2018-08-08 20:58 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-6950X CPU @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, broadwell)

cc: @yuyichao

[tkoolen]

This is still present on latest master, 1.1.0-DEV.143, 8555aa0aa3.

[tkoolen]

So the way I'm working around this right now for Parametron is by basically calling the old cfunction backend instead of @cfunction. See https://github.com/tkoolen/Parametron.jl/blob/45a4bdcdd1ca6202d679861bf9db9583d4bb9ae5/src/FunctionWrappersQuickFix.jl#L86.

@chkwon is doing the same thing in PATHSolver.jl, see https://github.com/chkwon/PATHSolver.jl/blob/master/src/FunctionWrappersQuickFix.jl.

Could you please not remove jl_function_ptr and any other relevant parts of the old cfunction backend until this is fixed? It would kill our packages.

[chkwon]

Please...

[rdeits]

I just tried repeatedly pasting the above code into the REPL in the latest Julia nightly and was not able to reproduce the issue.

julia> module M

       import Main.Parametron2

       f1 = [1]
       converted = Parametron2.wrap(Parametron2.LazyExpression(convert, Vector{Float64}, f1))
       converted.f()

       end
Main.M

julia> versioninfo()
Julia Version 1.3.0-DEV.92
Commit 73366aef59 (2019-04-23 14:33 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E3-1535M v6 @ 3.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)

perhaps it's fixed?

[yuyichao]

I remember there was a inference/compiler PR related to llvmcall with a very similar error message.

[rdeits]

Hm, looks like there's still a problem. I modified Parametron.jl to switch back to using FunctionWrappers.jl, and while most of the tests pass, I still get the following error:

julia Model: unconstrained: Error During Test at /home/rdeits/.julia/dev/Parametron/test/model.jl:37 Got exception outside of a @test TypeError: in cfunction, expected Core.Compiler.Const(getindex, false), got Parametron.Functions.QuadraticFunction{Float64} Stacktrace: [1] macro expansion at /home/rdeits/.julia/packages/FunctionWrappers/mJrYu/src/FunctionWrappers.jl:111 [inlined] [2] do_ccall at /home/rdeits/.julia/packages/FunctionWrappers/mJrYu/src/FunctionWrappers.jl:102 [inlined] [3] FunctionWrapper at /home/rdeits/.julia/packages/FunctionWrappers/mJrYu/src/FunctionWrappers.jl:118 [inlined] [4] macro expansion at /home/rdeits/.julia/dev/Parametron/src/lazyexpression.jl:59 [inlined] [5] evalexpr at /home/rdeits/.julia/dev/Parametron/src/lazyexpression.jl:56 [inlined] [6] LazyExpression at /home/rdeits/.julia/dev/Parametron/src/lazyexpression.jl:53 [inlined] [7] update! at /home/rdeits/.julia/dev/Parametron/src/moi_interop.jl:133 [inlined] [8] update!(::Parametron.Model{Float64,OSQP.MathOptInterfaceOSQP.Optimizer}) at /home/rdeits/.julia/dev/Parametron/src/model.jl:137 [9] solve! at /home/rdeits/.julia/dev/Parametron/src/model.jl:156 [inlined] [10] #test_unconstrained#1(::Float64, ::typeof(Main.ParametronTest.ModelTests.test_unconstrained), ::Parametron.Model{Float64,OSQP.MathOptInterfaceOSQP.Optimizer}, ::Array{Parametron.Functions.Variable,1}, ::Parametron.Parameter{Array{Float64,2},getfield(Main.ParametronTest.ModelTests, Symbol("##3#7")){Random.MersenneTwister},true}, ::Parametron.Parameter{Array{Float64,1},getfield(Main.ParametronTest.ModelTests, Symbol("##4#8")){Random.MersenneTwister},true}, ::Parametron.Parameter{Float64,getfield(Main.ParametronTest.ModelTests, Symbol("##5#9")){Base.RefValue{Float64}},false}) at /home/rdeits/.julia/dev/Parametron/test/model.jl:27 [11] test_unconstrained(::Parametron.Model{Float64,OSQP.MathOptInterfaceOSQP.Optimizer}, ::Array{Parametron.Functions.Variable,1}, ::Parametron.Parameter{Array{Float64,2},getfield(Main.ParametronTest.ModelTests, Symbol("##3#7")){Random.MersenneTwister},true}, ::Parametron.Parameter{Array{Float64,1},getfield(Main.ParametronTest.ModelTests, Symbol("##4#8")){Random.MersenneTwister},true}, ::Parametron.Parameter{Float64,getfield(Main.ParametronTest.ModelTests, Symbol("##5#9")){Base.RefValue{Float64}},false}) at /home/rdeits/.julia/dev/Parametron/test/model.jl:27 [12] top-level scope at /home/rdeits/.julia/dev/Parametron/test/model.jl:71

using Julia 1.3.0-DEV.92 as above.

[moustachio-belvedere]

Apologies if this adds unhelpful noise but I started experiencing the same issue but only at Julia v1.3, and in slightly modified form on v1.4 rc1. Versions 1.0, 1.1, 1.2 all worked fine. See full issue with MWE here

https://github.com/yuyichao/FunctionWrappers.jl/issues/12