Julia: Allow conversion from Nullable{T} to T?

[nalimilan]

Conversion from NullableArrays to special array types currently doesn't work very well when converting to a non-Nullable element type. This contrasts with standard Arrays, for which specific methods are provided. For example, when trying to convert to a sparse matrix:

julia> x = NullableArray([1 2])
1×2 NullableArrays.NullableArray{Int64,2}:
 1  2

julia> convert(SparseMatrixCSC{Int,Int}, x)
ERROR: MethodError: Cannot `convert` an object of type Nullable{Int64} to an object of type Int64
This may have arisen from a call to the constructor Int64(...),
since type constructors fall back to convert methods.
 in copy!(::Base.LinearFast, ::Array{Int64,1}, ::Base.LinearFast, ::Array{Nullable{Int64},1}) at ./abstractarray.jl:551
 in convert(::Type{SparseMatrixCSC{Int64,Int64}}, ::NullableArrays.NullableArray{Int64,2}) at ./sparse/sparsematrix.jl:272

This could be fixed in Base by defining:

Base.convert{T}(::Type{T}, x::Nullable) = convert(T, get(x))

This definition makes sense to me, as we already allow conversion _to_ Nullable. Of course, null values would trigger a NullException just like calling get directly.

Cc: @johnmyleswhite @davidagold @quinnj

[TotalVerb]

I really think there should be some well-defined semantics (axioms, if you will) for convert:

• It should be bidirectional; oftype(y, oftype(x, y)) should either be y or an error (with minor exceptions of 0.0 vs -0.0, etc.). This justifies the new definition.
• Converted value should in some sense be the "same" as pre-converted value. In most cases x == convert(T, x) holds; but this is not true in this case, and also in some other cases like convert(String, ::Vector{Char}). Rather I guess this sense of "same"ness is a little more loose.

[JeffreySarnoff]

+1 I find a similar need for allowing the collapsing of possibles over a narrow interval to an exemplar

[nalimilan]

@TotalVerb These indeed sound like useful rules. Though please move this discussion to a separate issue as it's likely to move quite far from this particular one.

[quinnj]

Yeah, I think this is a good definition to have. I like that we're getting some fleshed out semantics and usage with Nullables.

[tkelman]

We've moved the other way with some of the linear algebraic array wrapper types, deprecating convert there. And we avoided calling encoding of vectors of integers into strings "convert."

Won't this do things like drop the nullable wrapper if you assign a nullable value to a field where the field type is not nullable? It might be important to keep the wrapper present, since you're somewhat losing information going from a nullable type to its wrapped type.

[JeffreySarnoff]

cost vs benefit favors the benefit imo

[eschnett]

Automatically converting Nullable{T} to T is a bit like treating a one-element matrix as a scalar: It's often what you need, and there is basically no other definition that makes sense. Yet such an automatic mechanism can come back to bite you in the end.

In a script where all the types are easily understood, this will clearly be helpful. In larger packages where types are parametric, you'll get into trouble since you might not know whether a given type T is itself a nullable type or not.

I'd rather have a really simple way to treat a Nullable as its contained object. For example, Swift has some interesting ideas regarding using question marks or similarly very compact notions. In Julia, ?= could be a good syntax.

[quinnj]

^ I agree that we should certainly consider more succinct syntax for Nullables as they become more integrated in the language.

[johnmyleswhite]

At this point, I think the best thing is for someone to implement some new syntax and see how much pushback they get. I'd suggest starting with ?Int being desugared into Nullable{Int} since it seems to have zero semantic complexity. The major difficulties are (1) being able to modify the parser effectively and (2) resolving any syntactic ambiguities with the ternary operator.

I really like the idea of something like ?= desugaring into a lift construction, but I think that's going to take some time to get totally right.

[nalimilan]

It might be important to keep the wrapper present, since you're somewhat losing information going from a nullable type to its wrapped type.

@tkelman IMHO the distinction between Nullable{T} and T is important only for missing values (indeed, some proposed to replace it with Union{T, Void}). What kind of confusion could arise from such a conversion?

In larger packages where types are parametric, you'll get into trouble since you might not know whether a given type T is itself a nullable type or not.

@eschnett Why wouldn't you know whether T is nullable or not? By definition, the conversion would only happen when a specific type is requested/needed.

While I agree the ? syntax would be really useful, I think it's mostly orthogonal with convert. To me, the fact that NullableArrays provides methods to convert to non-nullable Array is the sign that we need this feature; but currently it won't work for conversions from Array{Nullable} or to other AbstractArray.

[tkelman]

T can't have a missing value, Nullable{T} can - that seems like an important distinction to preserve in many cases. If there are APIs that we're going to start transitioning to "result or none," Nullable is a type stable way to do that. At this point we don't know whether compiler improvements could ever get the implicit non-type-stable Union versions of that to match the performance of an explicit version that uses Nullable.

[TotalVerb]

Even so, there's a semantic difference between Union{T, Void} and Nullable{T}. The former is a T, or a missing value; the second is a container that may contain a T, or may not. This conversion seems reasonable to me regardless.

[eschnett]

I assume that you mostly want to convert from Nullable{T} to T when you know that the content is not null. Otherwise, I'd argue that having an implicit conversion that can throw an exception is a bad idea; you might as well keep this information explicit.

If you know that you have a nullable object that is not null, then I wonder why you know that it is not null. Maybe there is an if statement nearby that checks this? If so -- wouldn't you usually want to combine the isnull test and the destructuring into the contained object? You can't safely (i.e. without risking an exception) convert to T unless you check for null; and if you check, that would be the natural point in the code to perform the conversion. See the destructuring if statement in Swift that does this safely (in the sense of not throwing an exception).

[TotalVerb]

convert can and does already throw exceptions frequently:

julia> double(x)::Int = 2x

julia> double(1.5)
3

julia> double(1.8)
ERROR: InexactError()
 in double(::Float64) at ./<missing>:0

convert is used with the understanding that it might throw, and this conversion is compatible with this use.

[tkelman]

If we think of a Nullable as a container of either 0 or 1 elements, then we should ask: in what cases does convert(eltype(container), container) work currently?

[evanfields]

My not-a-PL-designer-heavily-biased-opinion; take it or leave it :)

• Automatically unwrapping Nullables is asking for trouble. Imagine a function database_retrieve which accepts some kind of query and returns a Nullable{Float64}. The Nullable represents that there may or may not be a value, right? What should happen if I call round(database_retrieve(query))? I argue that shouldn't sometimes work and sometimes not, depending on whether the query happened to return a non-null Nullable. Calling round on a maybe-something-maybe-nothing is (should be) a type error. I'd like to see a no method error or similar in that case.
• Automatically wrapping Nullables seems much safer to me. Functions which accept Nullable arguments can be cumbersome to call if you have to wrap concrete values by hand. Similarly it could be a little cleaner for functions which return Nullables to be able to break by returning concrete values (which are automatically wrapped). In a context where a Nullable is expected, automatically wrapping a value doesn't loose the information that there might be something; there might not.
• Sugar is nice, and I'd be in favor of more sugar to make using Nullable smoother. On the one hand something like ?T as shorthand for Nullable{T} seems great. Though it is hard to enter at the start of REPL lines, right - triggers help mode? Sugar for function application would also be a welcome addition. If my memory were better or my brush with Haskell more recent, I'd probably say something about the Maybe monad.

[nalimilan]

Automatically unwrapping Nullables is asking for trouble. Imagine a function database_retrieve which accepts some kind of query and returns a Nullable{Float64}. The Nullable represents that there may or may not be a value, right? What should happen if I call round(database_retrieve(query))? I argue that shouldn't sometimes work and sometimes not, depending on whether the query happened to return a non-null Nullable. Calling round on a maybe-something-maybe-nothing is (should be) a type error. I'd like to see a no method error or similar in that case.

convert does not happen automatically when passing argument to functions in Julia, so that's not a risk.

[TotalVerb]

@nalimilan I think the concern is with cases where implicit conversion does happen, such as

const A = Int[]
push!(A, tryparse(Int, readline()))

It is not immediately clear to me why allowing this is a bad thing, but I suppose this is cause for pause.

[evanfields]

@nalimilan Do correct me if I'm wrong, but with function return type annotation in 0.5, convert gets called on the returned value if it's not of the promised type, right? So there could be somewhat subtle automatic unwrapping if a function is supposed to return a T and the user tries to return a Nullable{T}. Which I think is a less worrisome case than what I described above, but for similar reasons I'm hesitant about automatic unwrapping.

[johnmyleswhite]

Also worth noting: implicit conversion also happens when assigning to the fields of types.

[nalimilan]

Yeah, these are cases in which implicit conversion can happen, but I don't see it as an issue. I agree this decision needs some reflection, though.

[davidagold]

I'm a little torn. One the one hand, this definition certainly makes sense. On the other hand, I'm partial to @eschnett 's reasoning: in order to use it safely (i.e. without the possibility of throwing an exception) you need to check isnull(x), and you may as well include the conversion logic in consequence of the check.

If we are to go with the latter approach, then this requires writing (potentially many) conversion methods like convert{T}(SparseMatrixCSC{Int,Int}, AbstractMatrix{Nullable{T}}). But, if we want users to be able to, say, convert to non-nullable matrices and supply a default value for null entries, then we'd need to define those methods anyway, so the latter approach doesn't necessarily introduce much extra work.

[KristofferC]

Moved to https://github.com/JuliaArchive/Nullables.jl/issues/9