Julia: things we should deprecate, 0.6 edition

[StefanKarpinski]

I'd like to collect a list of things that should be deprecated before 0.6 is released, even if we don't implement the corresponding feature until 1.0. Some examples I can think of:

• [x] Strings: "hello".data access – we need to change the representation of strings to make them more efficient, which is almost certainly not going to allow accessing their underlying bytes like this. So we need to deprecate this and provide an alternative API for this sort of thing. https://github.com/JuliaLang/julia/pull/19449.

• [x] Backtick syntax: deprecate using shell-special characters inside of backtick command syntax. At this point we can write most shell constructs in Julia, but it's gotten a bit more verbose and inconvenient than I'd like. So, in 1.0 I'd like to make, e.g. `echo hello & echo world | sort >> file.txt` construct a pipeline that runs two echo commands in parallel and connects them to the stdin of a sort command and then connects the output of sort to the file file.txt opened in append mode. https://github.com/JuliaLang/julia/pull/19786

• [x] the elementwise meaning of | and & given we're about to have .| and .& https://github.com/JuliaLang/julia/pull/19791

• [x] depreaction of sumabs, sumabs2, maxabs and minabs which are not necessary with fast function arguments. https://github.com/JuliaLang/julia/pull/19616

• [x] Array(T, dims...) (#19989)

• [x] partial linear indexing, done in https://github.com/JuliaLang/julia/pull/20079

• [x] produce/consume (issue https://github.com/JuliaLang/julia/issues/17699)

• [x] vectorized isinteger, et al. (#19925)

• [x] vectorized frexp (#19817)

• [x] promote_eltype_op (#19814)

• [x] deprecate vectorized ifelse in favor of dot syntax (https://github.com/JuliaLang/julia/pull/19931)

• [x] deprecate _ as a rvalue (#9343, #20328)

• [x] ~ macro syntax the overall plan from https://github.com/JuliaLang/julia/issues/19598#issuecomment-270717141:

  • [x] change the one user of this as a macro – i.e. StatsModels (https://github.com/JuliaStats/StatsModels.jl/pull/9)

  • [ ] update DataFrames to depend on the new version of StatsModels (issue/PR link?)

  • [x] change parsing of ~ in Base from macro to plain infix operator

• [ ] Open for read and write: we need to get rid of readandwrite so whatever deprecation we need to make there should be made. This may be harder since we may need to implement the change. https://github.com/JuliaLang/julia/pull/12807

• [x] Inner constructor syntax, see https://github.com/JuliaLang/julia/issues/19598#issuecomment-270158542

• [ ] Static parameter syntax, see https://github.com/JuliaLang/julia/issues/19598#issuecomment-270158542

• [ ] Deprecate reducer form of @parallel: https://github.com/JuliaLang/julia/pull/20259, waiting for a review

Probably in 1.0 rather than 0.6:

• [ ] Various special functions, pending #4301/#8536

• [ ] full – @Sacha0, how are we on this?

[tkelman]

• the elementwise meaning of | and & given we're about to have .| and .&
  edit: done in https://github.com/JuliaLang/julia/pull/19791 – @StefanKarpinski
• macro parsing of ~ if the stats packages can come up with a @formula replacement in time

[nalimilan]

macro parsing of ~ if the stats package can come up with a @formula replacement in time

Should probably go into StatsModels now.

[ararslan]

Regarding formulas, see https://github.com/JuliaStats/StatsModels.jl/issues/3.

[Sacha0]

Alongside @tkelman's mention of vectorized & and |, remaining vectorized methods for abs, abs2, imag, trunc, floor, ceil, mod, div, rem, clamp, and a number of hyperbolic trig functions should be deprecated. Likely vectorized big methods as well, and possibly (discussion ongoing) vectorized float, real, and (two-argument) complex methods. (I have stale pull requests open for most of those deprecations, blocked on whatever mark we set for completion of the sparse broadcast overhaul.) (edit: These are done, less those struck through.)

With devectorization complete, deprecation of expm, logm, sqrtm, and friends (i.e. matrix functions specially named due to former conflict with vectorized scalar functions) in favor of non-m equivalents might be worth considering. Best!

Edit: Also full. (I hope to see deprecating full through after completing the sparse map/broadcast overhaul (post 0.6).)

[andreasnoack]

I'd also propose depreaction of sumabs, sumabs2, maxabs and minabs which are not necessary with fast function arguments.

Edit: done in https://github.com/JuliaLang/julia/pull/19616 – @StefanKarpinski

[simonbyrne]

Various special functions, pending #4301/#8536

[simonbyrne]

For strings, the decision was to define a codeunit(s::String, offset::Int) function for people who need access to bytes.

[StefanKarpinski]

If Jeff's recent String experiment pans out, we'll do that; otherwise we rename the .data field.

[tkelman]

• partial indexing https://github.com/JuliaLang/julia/issues/14770, but maybe too big of a job to fit before 0.6 feature freeze

• vectorized isinteger (and maybe others) on arrays, replace with all(isinteger, x) (edit: isinteger done - @Sacha0)

[vtjnash]

If Jeff's recent String experiment pans out, we'll do that; otherwise we rename the .data field.

I think we said that Jeff's recent String experiment is expected to merge in the next few months, otherwise we don't spend any other effort on changing the representation. Probably by 1.0, we will need to have the infrastructure in place to handle Arrays more efficiently via an underlying Buffer type. At that time, we can delete the special case code for String and go back to having a .data field, by showing that it gives the same (or better) performance.

[JeffBezanson]

Another candidate: produce and consume, in favor of channels.

[amitmurthy]

Another candidate: produce and consume, in favor of channels.

https://github.com/JuliaLang/julia/issues/17699

[blakejohnson]

With devectorization complete, deprecation of expm, logm, sqrtm, and friends (i.e. matrix functions specially named due to former conflict with vectorized scalar functions) in favor of non-m equivalents might be worth considering.

I agree, but these need to wait one release so that we can suggest the alternate spelling without the m.

[ararslan]

The manual has claimed the Array(T, dims) syntax to be "deprecated" for a while now, but no deprecation warning is emitted. If we really want that syntax to be deprecated, we should formally deprecate it.

[goszlanyi]

What was wrong with Array(T,dims)?
It was concise, easy to remember and similar to zeros(T,dims) and ones(T,dims).
Could you point to the issue where it happened, please.

[ararslan]

What was wrong with Array(T,dims)?

It's not a proper type constructor; Array is a parametric type, so it should be constructed with a type parameter.

Could you point to the issue where it happened, please.

Jeff added the text in this commit: https://github.com/JuliaLang/julia/commit/23a6995a560ae88c14bf8e299faec7a4688df3f9

[StefanKarpinski]

Ok, as far as I can tell the remaining items to deprecate are:

[List moved to top comment]

Everything else is either already deprecated or has a fresh PR pending.

[mbauman]

partial linear indexing – any chance of this, @mbauman?

I can definitely deprecate it. I'm not sure I'll be able to resolve all the deprecations in a timely manner, though. Right now it's blocked on #18457. Once that is merged I can push a deprecation branch and we can crowd-source the deprecation fixes.

[ararslan]

~ macro syntax – how do we do this?

This is what @tkelman suggested for that:

maybe Base could keep ~ parsing as a macro but always throwing a depwarn? Looks like this may not change in Base for 0.6, but if packages can be made ready then maybe we hard change the parsing during 1.0-dev and packages might not notice if they've transitioned to this?

That would be the least immediately-breaking course of action, I would think. Once we do stop parsing ~ as a macro, we just have to make sure it still parses as an infix call with the same precedence.

[simonbyrne]

• [x] vectorized frexp (#19817)

[JeffBezanson]

Also:

• [ ] "Inner constructor" syntax. To be specific, the following constructor:

type Foo{T}
    x
    Foo(x) = new(x)
end

currently gets rewritten to

    (::Type{Foo{T}}){T}(x) = new{T}(x)

which is bad, because the definition looks like it defines Foo(x) but it doesn't.

• [ ] Static parameter syntax. The above constructor should instead be written as

    Foo{T}(x) where T<:Any = new{T}(x)   # or possibly just `new(x)`

These require #18457.

[simonbyrne]

There are also some changes to LibGit2 we should make before 1.0 (#19839), but I'm not sure if I'll have time to get to them before then.

[pabloferz]

I'd add

• [x] promote_eltype_op (#19814)

[timholy]

Possibly come up with two different names for one, #16116

[tkelman]

full – @Sacha0, how are we on this?

ref https://github.com/JuliaLang/julia/pull/18850 and https://github.com/JuliaLang/julia/pull/17082, planned for post-0.6 was my understanding

[tkelman]

~ macro syntax – how do we do this?

Discussion on the triage call is that the deprecation of ~ should be done in DataFrames since it's the only place out of all registered packages that defines a macro ~. If StatsModels.jl can be registered and a release-able branch of DataFrames can hook up to it in the next couple weeks, then we could upper-bound all existing releases of DataFrames so they're not installable on 0.6, only versions that support the new @formula macro from StatsModels.jl would support 0.6-and-newer. Then we could change the parsing in Base for 0.6 without needing any other deprecation.

[Sacha0]

full – @Sacha0, how are we on this?

ref #18850 and #17082, planned for post-0.6 was my understanding

Depending on when true feature freeze occurs, I might manage to complete full deprecation this cycle (edit: not enough time). I planned to prioritize other projects (stopgap map[!]/broadcast[!] over structured matrices, two-argument sparse broadcast!, any remaining devectorization tasks), though I would be happy to reprioritize full? Best!

[Sacha0]

vectorized isinteger, et al.

@StefanKarpinski, have you functions apart from isinteger in mind? iszero and isreal also reduce over arrays (though not isimag); latest discussion in https://github.com/JuliaLang/julia/pull/17623#discussion_r92529652 and https://github.com/JuliaLang/julia/pull/18495#issuecomment-267215901. Best!

[tkelman]

I didn't even realize until #19928 that ifelse is vectorized over AbstractArray{Bool} first inputs, good candidate for replacing with dot call broadcast?

[simonster]

In https://github.com/JuliaLang/julia/pull/19912#issuecomment-271145955 @stevengj proposed:

• fill!(X, a) -> X .= a (rejected as non-equivalent, edit @StefanKarpinski)

[JeffBezanson]

fill! and a .= b don't do the same thing. If both arguments are arrays, fill! still tries to set every element of a to the same array b, while .= copies elements from b to a.

julia> a = Array{Any}(3)
3-element Array{Any,1}:
 #undef
 #undef
 #undef

julia> b = rand(2,2)
2×2 Array{Float64,2}:
 0.30943   0.803133
 0.188654  0.709538

julia> fill!(a, b)
3-element Array{Any,1}:
 [0.30943 0.803133; 0.188654 0.709538]
 [0.30943 0.803133; 0.188654 0.709538]
 [0.30943 0.803133; 0.188654 0.709538]

[simonster]

That is a good point, but is there any difference between a[:] = b and fill!?

[JeffBezanson]

I think a[:] = b is more similar to a .= b, except the number of elements have to match since it doesn't broadcast.

[simonster]

Ah, I see. So I guess fill! needs to stick around.

[andreasnoack]

I wouldn't miss the behavior of fill! described by @JeffBezanson.

[Sacha0]

Possibly ! vectorized over AbstractArray{Bool} as in https://github.com/JuliaLang/julia/issues/20037#issuecomment-272647370? (Edit: Also over BtArray. Likewise ~.) Best!

[tkelman]

How's DataFrames doing, can we remove ~ parsing as a macro yet?

[quinnj]

@nalimilan ^?

[StefanKarpinski]

Can I have a link to a PR for the DataFrames ~ change so we can track it from here more easily?

[StefanKarpinski]

I've reorganized this issue so that all the todos are in the top comment. A few comments:

• @nalimilan (or anyone who knows): if you could fill in the relevant issue/PR links for the ~-related issues in the top comment, that would be helpful.

• anyone: a PR in this repo changing ~ into an infix operator can be done in parallel and makes a pretty solid up-for-grabs project for someone wanting to hack on the parser a bit.

• @timholy: differentiating the two meanings of one doesn't involve any deprecation afaict – we can just introduce a new function and people can start using it.

• @JeffBezanson: are we realistically still going to deprecate inner constructor syntax and static parameter syntax in this release? If so cool, but obviously should happen soon.

• @musm, @simonbyrne: what's the status of moving "various special functions" out of base and deprecating them? Is that realistically still going to happen in this release?

• @Sacha0: is the full thing still plausible in this release?

If the answer to any of the above is "no", that's ok, just want to update the list so we know what we're still waiting on.

[nalimilan]

@ararslan is the one to ask regarding the ~ changes, but at least the needed changes have been merged in StatsModels (https://github.com/JuliaStats/StatsModels.jl/pull/9, I've added the link to the description). We still need to make DataFrames use that, though.

[StefanKarpinski]

Now that the StatsModels change has happened (thanks, @ararslan!), I don't think the other two changes need to happen in order since they're about tagging new versions and getting the dependencies right. So the next action item there seems to be a PR to change ~ to an operator.

[timholy]

Yes, I agree that the changes for one are easy (docstring and adding a new function). Fixing base can come later.

[stevengj]

I actually started on a patch for one. Making Base dimensionally correct requires a big effort

[stevengj]

See #20268.

[mbauman]

Is it possible to deprecate using _ as an rvalue?

[StefanKarpinski]

Good call, @mbauman – need to do that now in order to use it for discarding values in 1.0.

[amitmurthy]

Also

• [ ] Deprecate reducer form of @parallel - https://github.com/JuliaLang/julia/pull/20259, waiting for a review.

[ararslan]

A few FYIs from me:

Special functions: I've been working on moving special functions out of Base. From a package standpoint it's nearly there, but from a modification to Base standpoint there's still a fair bit of work to be done. In particular, to remove the dependency on openspecfun, we'll need to reimplement rem2pi in pure Julia, which requires double-double arithmetic. Unless we can extend and use Base.TwicePrecision for this, we need a solid double-double arithmetic implementation in Base. I think it's unlikely that this will happen for 0.6.

FFTW: As another data point, I've also been working on moving FFTW into a package. That too is going well, though there is still quite a bit of work to be done in moving from makefiles to BinDeps. Similarly unlikely to make it into 0.6.

[StefanKarpinski]

As a temporary solution, could we just copy-pasta the one function you need from openspecfun into libjulia for now, with the longer term plan of reimplementing it in pure Julia?

[ararslan]

I'm cool with that.

[JeffBezanson]

How about moving SharedArrays to a package?

[ararslan]

Hm, SharedArrays seem pretty convenient for the parallel facilities we provide out of the box. If anything I would suggest that sparse arrays go first.

[JeffBezanson]

I think SharedArrays are pretty limited and difficult to use. They are not used anywhere in Base itself and SharedArrays is very self-contained. Of course sparse should be moved too, but I have a sense it will be more difficult. It has binary dependencies for example.

[ararslan]

Yeah, makes sense. Perhaps JuliaParallel could adopt a SharedArrays.jl package. @andreasnoack?

[ararslan]

Special functions are on their way out! See #20427

[PallHaraldsson]

"the elementwise meaning of | and & given we're about to have .| and .& #19791"

[EDIT: Made question clearer, I think..:] Does #19791 mean we already disallow overloading | and & ? I think that should be done; added to the list of deprecations if not already done.

[StefanKarpinski]

Does this that we can disallow overloading | and & ? I think it should be added to the list of deprecation if not already done.

Apropos of what?

[JeffBezanson]

I propose closing this. We can still entertain deprecation PRs as they are posted, but I think we're done actively seeking deprecations for 0.6.

[StefanKarpinski]

Sure, I think this has served its purpose now.