Crystal: Missing retry

[technorama]

Retry provides an elegant way to loop over operations that may experience periodic failures. This feature is missing from crystal.

retries = 0
backoff = 0.5
begin
  TCPSocket.new host, port
rescue
  raise if retries >= 3
  sleep backoff * retries
  retry
end

[refi64]

I've got to admit...I'm not a big Ruby fan, but retry is a _really_ cool feature.

[asterite]

Can't this be done with a loop?

[asterite]

I never used retry in my life, and if this is going to be needed in 0.01% of the code then for me it's not worth the trouble adding this feature.

[asterite]

Furthermore, you can define a backoff method that uses a loop and yields, and then reuse this method, making retry even less attractive to justify adding it.

[technorama]

@asterite

• It's not just for backoffs.
• A generic backoff method won't work if you want to retry some errors and but not others.
• A generic backoff method won't work if you need to do something before retrying (resetting state, adding or changing parameters, updating buffers, etc).

Retry shows intent. "I want to run this once but possibly retry for some errors possibly doing other things before retrying". A loop indicates you intend to run something many times. Retry is clear. loop { } with break statements is not clear without carefully reading the loop structure.

Not to mention many ruby projects find retry succinct and useful.

• There are 19 uses of retry in the ruby stdlib, not including c extensions and their .rb files.
• Rails uses it.
• ActiveRecord uses it.
• Sequel uses it.
• HAML uses it.
• Compass uses it.
• Rainbows uses it.
• Unicorn uses it.
• Puma uses it.
• Resque uses it.
• Lots of other projects use it.

If it involves an operation that _may_ fail retry is useful. Many libraries that use Sockets use it but from the list above you can see other libraries find uses for it that don't rely on IPC.

Some additional examples since you have never used it before:

opts.order! { |o| code_or_file ||= o } rescue retry

def translate_offset(byte_offset)
  ...

  begin
    @wrapped_string.byteslice(0...byte_offset).unpack('U*').length
  rescue ArgumentError
    byte_offset -= 1
    retry
  end
end

def foo params
  bar params
rescue SomeError
  if SomeError.message =~ /bar/
    params["default"] = "bar"
    retry
  end
# don't retry with any other error type
end

[ozra]

I like the look of that! Brings fault handling in to the first room. Shit does happen.

[asterite]

a = 0
begin
  puts a # What's the type of `a` here?
  a = nil
rescue
  retry
end

You see, it's pretty simple to propose new features, but when you want to add them you need to think:

1. How it affects the type inference logic. Now there might be a retry in a rescue, so you need to rebind the type of variables at the exit of the blocks, etc. This, of course, slows down the compiler.
2. How to codegen it.
3. Add documentation to it, explain it to users, all users must understand how it works, etc.

Now, if you can implement retry in some other way using the existing tools, everything is simplified. Java and C#, pretty popular languages with exception handling, don't have retry. I'm sure people found their way. So I can list all Java and C# libraries and say "Look, the don't use retry" and that'll beat the list of Ruby programs that use retry.

And, again, if it's going to be used in 0.01% of the cases, I'd like to keep that feature off from the language.

[ozra]

Well a is obviously (Int32|Nil). that aside, definitely recognise the hard work it would require.
For me this is not a very important feature.
But, I'd be happier if it existed than not.

[asterite]

On the other hand, if you manage to rewrite code with retry to some code without retry, we might implement this because then it's a free feature with zero effort from the compiler. I have an idea of how to do tihs, but we might need to implement #1277 first.

[asterite]

What I meant with the above point is to think of a rewrite or lowering. case is rewritten to a series of if/elsif/else. I imagine an exception handler with retry to be rewritten to a loop and break, sort of. The main problem is that the handler can be inside a while, so using break won't work...

[ozra]

With #1277 it sounds like it should be possible to pull off indeed :)

[technorama]

@asterite My feature proposals are to add clarity to the language. Readability and transparency trump zero overhead in my book. If I wanted zero computational overhead with tons of programmer overhead I would put up with C++.

Retry tells me the operation is expected to work under normal conditions but it may "retry" when encountering some failures. A loop at a glance tells me this will probably run more than once under normal conditions. I would need to read it carefully to figure out the loop is only for failure conditions.

The compiler problem seems like it can be worked around without overhead. Would a compiler internal multi loop break help? Something like throw/catch with an additional return type?

[technorama]

@asterite I could make the case that ensure is a 1% use case and other languages like c++ and php get by fine/miserably(php) without it. I could also make the same case for exceptions. I did make the case that else statements in methods/begin are confusing and not used anywhere (even in ruby) but somehow those still exist in crystal.

Does retry aid in understanding what the code is for and how it works? It does for me and I think it does for other people even if you haven't used it.

[Sija]

How is state of the things today, would it be possible to implement? It would be a very welcomed feature for v0.19 indeed.

[Sija]

@asterite ping

[Sija]

@asterite regarding transforming the code I've come up with sth like this:

# sample code
require "socket"

host = "doomed-to-fail4.com"
port = 777

retries = 0
backoff = 0.5

# Transform this block:
# begin
#   TCPSocket.new host, port
# rescue
#   raise if retries >= 3
#   sleep backoff * retries
#   retries += 1
#   retry
# end

# ... into below:
x = -> {
  begin
    TCPSocket.new host, port
  rescue
    raise "enough" if retries >= 3
    sleep backoff * retries
    retries += 1
    :__retry__
  end
}

loop do
  break unless x.call == :__retry__
end

Since begin ... end blocks are having their own scope wrapping them in closure wouldn't do much of a difference, right? :__retry__ placeholder is there just for illustrational purposes, it might be some compiler type or other, more differentiable type. Ignore me if that doesn't make much sense.

[lbguilherme]

Is there any good use case other than retrying a specified number of times until a exception is not raised? We could simply have a method implementing this idiom.

require "socket"

host = "doomed-to-fail4.com"
port = 777

def retry(limit, *, backoff=nil)
  attempts = 1
  loop do
    begin
      yield
    rescue error
      raise error if attempts == limit
      sleep backoff*attempts if backoff
      attempts += 1
    end 
  end
end

sock = retry(3, backoff: 0.5) { TCPSocket.new host, port }

It is both not a new keyword and easier to use.

[Sija]

@lbguilherme Sure, we could, but this can be said for most of the syntactic sugar like property or newly added select. Your example is not very flexible too (nor easier to use) since one might want to handle rescue blocks differently, depending on the case. Not to mention multiple uses of it by many great Ruby projects—see https://github.com/crystal-lang/crystal/issues/1736#issuecomment-148168437...

[chaniks]

retry always saved me from the pyramid of doom 😉

[RX14]

I like the solution proposed by @lbguilherme, could it be accepted into the stdlib?

[Sija]

@RX14 I second that. btw, there's missing return before yield which should exit the loop on success.

[codenoid]

please, i want that method

[drhuffman12]

See also the Retriable gem https://rubygems.org/gems/retriable/ . We use it at work for microservice/api calls.

[bararchy]

@drhuffman12 it seems @Sija has created something similar in Crystal https://github.com/Sija/retriable.cr
What about adding the logic into the STDLIB ?

[sdogruyol]

I don't think we need a dedicated retry keyword in the language itself. Even in Ruby, it's a bit redundant..

[RX14]

I think this issue can be closed. I don't think we're going to implement retry.

[digitalextremist]

Was really sad to see retry not be included, but retriable above is a great alternative so far.