Rust: Tracking issue for Arc/Rc extras

[alexcrichton]

This is a tracking issue for the arc_counts, rc_counts, and rc_would_unwrap features. The counting functions have been around for awhile but would_unwrap was added recently.

cc #27718, more discussion there.

[timcharper]

@alexcrichton I think there may be a good use for strong_count in detecting un-fulfillable promises. Although, I am struggling with it, because most cases I think of are racy.

If you try and get the value of a promise synchronously, and there is only one instance of the promise, then you can fail the promise ahead of time because either nobody is listening, or nobody is fulfilling.

See the discussion here: https://github.com/viperscape/rust-promise/issues/3

[timcharper]

The more I think about this, I think the more your point about hard_count being finicky is corroborated. This case is racey and complex. It'd be better / more deterministic to just have the promise register itself to listen for a thread panic, and just fail if it does.

[alexcrichton]

@timcharper yeah in general the methods on Arc are inherently racy, but there's a point that can be made where there's external synchronization in play so the raciness doesn't actually happen in some specific scenarios (in which case these may be desired)

[jethrogb]

I rewrote my usecase for strong_count to work with Option<Arc<_>> and Arc::try_unwrap so I no longer actually need this.

[shahn]

If this gets stabilized, these APIs should get added to the respective Weak as well I'm thinking

[johalun]

What is planned for arc_counts? I would like to use it because I need to do ffi call to c library when the last Arc pointer is dropped but don't wanna start using it if it might disappear soon.. Maybe a different approach would be to do the call when the inner structure is being dropped, that would definitely be when the last pointer is dropped because is has only one owner..

[ticki]

May I ask: What is the point of Arc and Rc keeping the weak reference count? They should be irrelevant to them.

[Gankra]

@Ticki I don't understand the question. Rc/Weak are dual types that have different interpretations of the same memory. Individual Rc/Weaks do not store a count. Rather, they all share RcBoxes to coordinate these counts.

[ticki]

I don't understand the question. Rc/Weak are dual types that have different interpretations of the same memory. Individual Rc/Weaks do not store a count. Rather, they all share RcBoxes to coordinate these counts.

Yeah, that's right. They're just pointers to the actual data plus a weak and a strong reference count. What I ask is: why do they keep track of the weak reference count? For example, C++ and Obj-C do not count the weak references.

[timcharper]

If I were to venture a guess, strong reference counts are used to know when the ARC contained data should be freed, weak reference counts are used to know when the ARC container should be freed.

I'm surprised to learn that C++ / Obj-C don't do this (at least, under the hood). What data structure does the weak reference query to learn if the memory has been freed?

[ticki]

In fact, I was wrong about C++ shared_ptr:

The control block is a dynamically-allocated object that holds:

• either a pointer to the managed object or the managed object itself;
• the deleter (type-erased);
• The allocator (type-erased);
• the number of shared_ptrs that own the managed object;
• the number of weak_ptrs that refer to the managed object.

(emphasis mine)

Source: http://en.cppreference.com/w/cpp/memory/shared_ptr

[Gankra]

Obj-C strategy is explained here: https://mikeash.com/pyblog/friday-qa-2010-07-16-zeroing-weak-references-in-objective-c.html

Basically: every class instance has a set of "locations of weak pointers". When its strong count hits 0, it traverses the set, nulls out those pointers, and then frees all of its allocations.

This can't be implemented in Rust, because it requires weak pointers to have significant locations in memory. Would necessitate overloading move (never gonna happen), or runtime support (unlikely to happen).

That said, @pnkfelix has been working on the notion of stack introspection hooks for GC's, which could maybe accommodate this use-case..? Not something we would retrofit Arc/Rc to use, but something someone could do with their own custom types.

[jminer]

Does it look like either rc_counts or rc_would_unwrap will be stablilized? I'm wrapping a C library and have a place I'd like to test strong_count == 1 or use would_unwrap, and similarly to @SimonSapin's usecase, I can't use try_unwrap. The check is in drop of a wrapper struct, where I only have &mut self (I need to free some memory when the last reference is dropped). I can probably add another struct to wrap the contents of the Rc, but the code won't be as clear.

[SimonSapin]

Another use case for strong_count:

The heapsize crate traces through things and counts heap-allocated memory. Doing so with Arc<T> or Rc<T> is tricky: https://github.com/servo/heapsize/issues/37. One idea is to divide by the strong reference count: consider that each reference "owns" a fraction of the memory.

[pnkfelix]

Another use case for strong_count:

The heapsize crate traces through things and counts heap-allocated memory. Doing so with Arc or Rc is tricky: servo/heapsize#37. One idea is to divide by the strong reference count: consider that each reference "owns" a fraction of the memory.

If another reader sees this and like me thinks "won't the need to handle cycles provide a separate place to represent this information?" then you should read the linked issue. (Quick answer: maybe yes, but it sounds like it doesn't currently handle cycles well anyway, and the comment thread has some interesting alternative ideas beyond a separate map of seen addresses.)

[SimonSapin]

@pnkfelix indeed. One of the alternative idea is to return 0 if the strong reference count is more than 1, and that would also need stabilizing one of the methods covered in this issue.

[camlorn]

These may be useful for debugging. I don't have a concrete use case at the moment. Somehow I've managed to avoid shared ownership thus far.
However, I could see it as potentially beneficial in the future if I had to track down a reference cycle or figure out why something isn't dying. I do agree that the use in actual programs is less clear, but these are maybe worth having just so you can print them.

[ticki]

@camlorn Cycle detection in debug mode could be awesome actually.

[camlorn]

Agreed. I thought of it as well. But this didn't seem like the place to suggest it and in truth that sounds like it would probably be very complicated and RFC-ish anyway.
Printing the counts in debug is admittedly much worse in terms of what we have, but having these as unstable and/or killing them feels like a step backward in this department. Though perhaps only a small one.
Still, cycle detection would be a major win over C++, do that now.

[DemiMarie]

Actually, cycle detection in could be useful for far more than just debugging: one might be able to use it to implement a cycle collector a la Python that not only detected cycles, but freed the ones that were no longer reachable.

[camlorn]

@DemiMarie
That would be unsafe. Create a cycle that is accessible from the stack but for which there is no Rc on the stack. Watch your program die.

You'd also need special compiler machinery for tracking the accessibility of stuff from the stack, and at that point we might as well just add a full GC.

[SimonSapin]

The remaining unstable methods covered here are:

• Arc::strong_count
• Arc::weak_count
• Rc::strong_count
• Rc::weak_count
• Rc::would_unwrap
• Rc::is_unique

They have been #[unstable] for some time now, and I the discussion hasn’t seen new arguments for a while. Based on reading this thread and #27718, the concerns that I know of are:

• The meaning of is_unique is not entirely obvious based only on its name, in presence of two (weak and strong) counts.
• Use cases for is_unique and would_unwrap are niche.
• Arc::*_count is easy to misuse: in the time between calling one of these methods and acting on the result, another thread might have done something to change a count and make that result out of date.

To counter-balance that:

• I haven’t seen a real concern against Rc::*_count, only being conservative on principle when stabilizing things in std.
• Arc::*_count _can_ be used correctly in some cases (such as with external synchronization). And they just return integers, so they’re not too much of a footgun by themselves. To actually have a data race when misusing them, you’d have to also do something else that requires unsafe.
• At least in my case, the use case for would_unwrap, however niche, was enough motivation for forking the std::rc module.
• is_unique and would_unwrap are very easy to implement on top of strong_count and weak_count.

Given all this, I’d like to propose FCP to stabilize all four *_count methods, and deprecate is_unique and would_unwrap.

[alexcrichton]

@rfcbot fcp merge

Thanks for the summary @SimonSapin! Sounds reasonable to me to stabilize the count methods and deprecate the others. We've already committed to reference counting and weak pointers for both Rc and Arc so exposing them seems harmless!

[glaebhoerl]

Let's at least strongly emphasize the concurrency hazards in the Arc methods' documentation.

[SimonSapin]

@glaebhoerl I’ve opened https://github.com/rust-lang/rust/pull/37652, what do you think?

[glaebhoerl]

LGTM :)

[rfcbot]

Team member @alexcrichton has proposed to merge this. The next step is review by the rest of the tagged teams:

• [x] @BurntSushi
• [x] @Kimundi
• [x] @alexcrichton
• [x] @aturon
• [x] @brson
• [x] @sfackler

No concerns currently listed.

Once these reviewers reach consensus, this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[rfcbot]

:bell: This is now entering its final comment period, as per the review above. :bell:

_psst @alexcrichton, I wasn't able to add the final-comment-period label, please do so._

[BurntSushi]

@alexcrichton I think you need to tell rfc bot that you applied the label?

[alexcrichton]

@BurntSushi hm I think rfcbot may be missing out on FCP endings... In any case we'll want to leave this until the near the end of the cycle regardless.

[rfcbot]

The final comment period is now complete.