Julia: replace: allow transform function as first argument?

[StefanKarpinski]

The replace function allows a string, an s"..." replacement specifier or a function as its third argument. For the sake of do syntax, it might make sense to support the function as the first argument, like so:

julia> replace("Hello, world.", r"\w+") do word
           n = 0
           for c in word
               n += lowercase(c) in "aeiou"
           end
           return n
       end
"2, 1."

If someone decides to do this, while they're at it, an explanation of replacement patterns – i.e. s"..." strings and some more examples in the replace docs wouldn't hurt. E.g.:

julia> replace("Hello, world.", r"(\w+)(\W+)(\w+)", s"\3\2\1")
"world, Hello."

[affans]

I would like to try to tackle this. But where would I find the source code for handling the do syntax? Doing a search for do isn't helpful.

[StefanKarpinski]

See this section: https://docs.julialang.org/en/latest/manual/functions/#Do-Block-Syntax-for-Function-Arguments-1. You don't need to do anything special for do syntax, just add a method to replace where the replacement function is the first argument.

[nalimilan]

(For another, non-string extension of replace, see https://github.com/JuliaLang/julia/pull/22324.)

[StefanKarpinski]

@rfourquet, @nalimilan: does non-string extension collide with this change as far as you can tell?

[nalimilan]

It wouldn't collide strictly speaking since this one would dispatch on AbstractString, but the semantics would be different, which could be confusing (or not). The implementation at #22324 takes a predicate function which returns true for entries which should be replaced with the element provided as an argument to replace. So the function is the predicate, and the replacement is fixed, which isn't the case here.

It could be made more general so that it can return either nothing, in which case nothing happens, or another value, in which case the entry is replaced with it. That general API would also work here. It would be a bit more complex, but it would reduce the number of different signatures needed.

[rfourquet]

It could be made more general so that it can return either nothing, in which case nothing happens, or another value, in which case the entry is replaced with it.

This is exactly what #22324 does, with less general cases being defined in terms of this one (cf. the diff, where it is documented in terms of Nullable while the actual implementation replaced it with Union{Some{T}, Void} to test performance differences).

[rfourquet]

And if we go with this more general API, this should obviously wait till @nalimilan's work of replacing Nullable in base gets merged.

[nalimilan]

Ah, right, the function I described is just a convenience helper based on the Nullable method.

[twistedcubic]

From what I could find I don't think the branch replacing Nullable has been merged yet, but I started thinking about this, and I am going to implement a replace that takes a function as the first argument, much like what open does.

I don't understand the comment

It could be made more general so that it can return either nothing, in which case nothing happens, or another value, in which case the entry is replaced with it.

Shouldn't whether this new replace returns either nothing or another value be determined entirely by the user's function? I looked at the diff in #22324, but didn't find the equivalent there.

[JeffBezanson]

I don't think we need the nothing feature, since if you want no replacement you can return the argument unmodified.

[nalimilan]

@twistedcubic Yes, it's been merged several months ago, see https://github.com/JuliaLang/julia/pull/23642. But we've decided against using nothing to mean "keep original value" in https://github.com/JuliaLang/julia/pull/25697, as it's not very useful as @JeffBezanson noted, and it's ambiguous when nothing is a possible value (unless you require wrapping all value in Some, which is cumbersome).

[twistedcubic]

Thanks. I have been experimenting with variations of:

function replace(f::Function, s::AbstractString, pair::Pair; count::Integer=typemax(Int))
         res = replace(s, pair, count)
         return f(res)
 end

But have been running into issues with

replace("world.", "."=>"1") do word
       return "hey " * word
   end

such as

ERROR: MethodError: no method matching keys(::Pair{String,String})
Closest candidates are:
  keys(::Cmd) at process.jl:819
  keys(::Tuple) at tuple.jl:42
  keys(::Tuple, ::Tuple...) at tuple.jl:48
  ...
Stacktrace:
 [1] findnext(::Base.EqualTo{String}, ::Pair{String,String}, ::Int64) at ./array.jl:1635
 [2] findnext at ./deprecated.jl:730 [inlined]
 [3] findnext(::Pair{String,String}, ::String, ::Int64) at ./deprecated.jl:57
 [4] #replace#343 at ./strings/util.jl:386 [inlined]
 [5] replace at ./strings/util.jl:380 [inlined]
 [6] replace(::String, ::Pair{String,String}, ::Int64) at ./deprecated.jl:57
 [7] #replace#156 at ./REPL[250]:2 [inlined]
 [8] replace(::Function, ::String, ::Pair{String,String}) at ./REPL[250]:2
 [9] top-level scope

I'm going to chew on it more. In the meanwhile:

• Wouldn't we want to add the same API to replace!?
• Do we want to allow function-transform-as-first-arg for replace signatures other than String replacements? Although the problem with that is ambiguity with the replace that takes a predicate as the first arg, since such as function would no longer dispatch on AbstractString.

[nalimilan]

Honestly I'm not so sure it's a good idea to add yet another method to the already crowded replace. I have a hard time grasping all the different variants that we support. Do we have a strong use case for this method? The example given that @StefanKarpinski above isn't very realistic. Also as @twistedcubic noted this method doesn't generalize to non-string collections due to ambiguities.

BTW, we could probably deprecate replace(pred::Function, A, new) in favor of replace(x -> pred(x) ? new : x), which would at least simplify a bit the situation.

[twistedcubic]

@StefanKarpinski and @JeffBezanson thoughts on whether we should keep pursuing allowing a transform function as the first arg?