Rust: Tracking issue for Vec::resize_with and resize_default

[joshlf]

Currently, Vec has a resize method which takes a new parameter so that if the resize involves growing the vector, the new parameter is cloned into each newly-created cell in the vector. T must implement Clone in order for this method to be available.

It would be useful to add a resize_default method that instead requires that T: Default, and calls T::default() to fill the newly-created cells. Not only would this be ergonomic, but for certain implementations of T::default, it might allow the compiler to make better optimizations.

[clarfonthey]

This would be an interesting addition for types which support Default but not Clone. I'll try and make a PR for it.

In the future, just FYI, feature requests should go in the RFCs repository.

[joshlf]

Ah OK, noted. That's even for feature requests like this one that don't deserve an RFC?

[clarfonthey]

Yep! The general rule is either PR here or RFC elsewhere.

Also, @joshlf would you mind renaming this issue to "Tracking issue for Vec::resize_default" ?

And @nikomatsakis would you mind tagging it?

[joshlf]

Gotcha; thanks!

[Mark-Simulacrum]

@rust-lang/libs Nominating for stabilization. The only potential concern is that we can generalize this to take a closure (see below). I don't know that it'd be all that useful, though, so I'm inclined to say that we shouldn't do so, and Clone and Default are the two "producing" traits in std, so standardizing around them seems logical.

fn resize_with(&mut self, new_len: usize, f: F)
where F: FnMut() -> T

[clarfonthey]

@Mark-Simulacrum I'd be willing to make a PR that adds resize_with under a separate feature flag. I think that all three are useful; resize_default would simply be documented as equivalent to resize_with(Default::default)

[sfackler]

I might lean more towards just having resize_with, for now at least. It's strictly more powerful and doesn't seem all that more verbose than resize_default?

[joshlf]

My concern is that resize_with hampers discoverability - you have to come across resize_with and then connect the dots to realize that you can do resize_with(Default::default). Especially for beginners (and people who are ctrl+f-ing through the docs), this will probably make it less likely for people to figure out that they can resize using the default value.

[djc]

Discoverability is pretty easily fixed by just having some documentation, though, right?

I like the power of resize_with, but I'm wondering about optimizability. Especially for Copy types (I guess this might depend on specialization), couldn't a resize_default-like API result in much faster code over something that requires calling a closure?

[sfackler]

You'd just use resize for Copy types, but I'd expect resize_with(Default::default) to be literally identical to resize_default. Calling a function isn't really any different from calling a closure.

[sfackler]

The @rust-lang/libs team discussed this today and feel like resize_with would be preferable.

[djc]

I've submitted #49559 implementing Vec::resize_with() in order to help move this forward.

[oberien]

Any updates on this? Is there anything blocking this issue?

[clarfonthey]

Only concern is whether to keep resize_default now that resize_with exists. But it looks like resize_with can go in FCP regardless.

[djc]

@rust-lang/libs so this has been in nightly for two months. What's the next step?

[Shnatsel]

Not only would this be ergonomic, but for certain implementations of T::default, it might allow the compiler to make better optimizations.

I see people frequently use unsafe .set_len() because the safe .resize() is not fast enough. I've already talked about how this has almost resulted in a vulnerability in my Auditing popular crates post, but now I have discovered an _actual_ vulnerability in another crate.

So I am _very_ interested in the "better compiler optimizations" part if it extends to numeric types such as u32. Or, if there is a different safe idiom for writing to a preallocated vector, I would love to hear about it.

[SoniEx2]

I have a Vec<HashSet<&str>>.

Okay, the type is lying a little. I have a Vec of HashSets using the RandomState (default) hasher. The RandomState hasher is Clone, but using Clone to fill in a Vec has a serious caveat: all HashSets would have the same RandomState internal state. This runs a risk of exposing the system to DoS attacks.

This is a very good reason to have resize_default or w/e in addition to resize. Anyway, guess I'll be using a loop.

[scottmcm]

I just wanted this on VecDeque, because resize wants Clone, which I don't have, but .resize_with(n, Vec::new) would work great.

[scottmcm]

@rust-lang/libs Are you still happy with resize_with, as you were in https://github.com/rust-lang/rust/issues/41758#issuecomment-359970673? Is there any additional work people can do to help this along towards FCP?

[sfackler]

I'm happy with resize_with personally!

impl<T> Vec<T> {
    pub fn resize_with<F>(&mut self, new_len: usize, f: F)
    where
        F: FnMut() -> T;
}

impl<T: Clone> VecDeque<T> {
    pub fn resize_with(&mut self, new_len: usize, generator: impl FnMut() -> T);
}

@rfcbot fcp merge

[rfcbot]

Team member @sfackler has proposed to merge this. The next step is review by the rest of the tagged teams:

• [ ] @Kimundi
• [x] @SimonSapin
• [x] @alexcrichton
• [x] @dtolnay
• [x] @sfackler
• [ ] @withoutboats

No concerns currently listed.

Once a majority of reviewers approve (and none object), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[dtolnay]

Why does the VecDeque one require T: Clone?

[sfackler]

Seems like the method ended up in the wrong impl block by accident. We definitely don't want that bound.

[rfcbot]

:bell: This is now entering its final comment period, as per the review above. :bell:

[SoniEx2]

This should add both resize_with and resize_default for the reasons I mentioned above.

[dtolnay]

The FCP is for resize_with only. vec.resize_with(n, HashSet::new) is not affected by the hash DoS you described.

[SoniEx2]

There are usability/discoverability concerns with resize_with

[clarfonthey]

If the FCP is just for resize_with could someone update the issue title?

[scottmcm]

Good catch @dtolnay; _Mea culpa_. Fix up in https://github.com/rust-lang/rust/pull/56401.

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

[scottmcm]

Reactivating because https://doc.rust-lang.org/nightly/std/vec/struct.Vec.html#method.resize_default still points here.

Any thoughts on what should be done with resize_default?

[SimonSapin]

It’s the same as v.resize_with(Default::default) and was added before we considered resize_with, right? Unless someone argues otherwise I’d be inclined to deprecate and remove.

[SoniEx2]

It's equivalent, but not the same.

v.resize_default is more user-friendly!

[clarfonthey]

Is anyone using resize_default currently? Would love to here some experiences from people.

[Kerollmops]

Isn't Vec::resize_default some kind of completion of the Option/Result::unwrap_or_default methods? Don't know if my comment really make sense.

[clarfonthey]

@Kerollmops And resize_with is like unwrap_or_else. We're not sure if we need a default version.

[Kerollmops]

So, if we deprecate the Vec::resize_default method, why not the Option/Result::unwrap_or_default() ones?
I am probably splitting hairs, but...

[orenbenkiki]

If the matter is open to discussion: I find that writing foo.resize_with(size, Default::default) or even foo.resize_with(size, default) is too verbose and less communicative of the intent than writing the shorter and clearer foo.resize_default(size). As @Kerollmops pointed out, there's a reason we have an unwrap_or_default.

[portstrom]

Is there any reason not to stabilize this right now? Offering functions that do the same thing as another function but with fewer arguments is a very common thing, and it's zero-cost in Rust.

[Lucretiel]

I'd like to have resize_default. I'm about to use it in one of my projects, plus I think it's good to have pairity with the other or_default methods, like [Option|Result]::unwrap_or_default and Entry::or_default

EDIT: I forgot one of my favorites, mem::take (which is just mem::replace(thing, Default::default)

[Lokathor]

I like resize_default, it's very clear and more newbie friendly.

[Timmmm]

use of deprecated item 'std::vec::Vec::::resize_default': This is moving towards being removed in favor of .resize_with(Default::default). If you disagree, please comment in the tracking issue.

I disagree! resize_default() is such a common operation it is nice to have a short way of writing it. And as @Lucretiel mentioned there are plenty of other convenience _default methods in Rust. What exactly is the problem with leaving it in?

Edit: When I say "leaving it in", I actually thought it was already stable (haven't compiled code yet). I discovered it through rust-analyzer's autocomplete, thought "resize_default? Yes that's exactly what I want". Some empirical evidence for @joshlf's discoverability argument. No way would autocomplete have suggested resize_with(Default::default) and honestly I'd have had to have googled how to do it if that was the only way. Or realistically I would have just done resize(..., 0) which is rubbish.

[xfix]

From a quick search on grep.app (https://grep.app/search?q=.resize_with&filter[lang][0]=Rust), I decided to categorize every line using resize_with.

Default parameter usages

data_buffer.resize_with(batch_size, T::T::default);
buf.resize_with(batch_size, || 0);
buf.resize_with(batch_size, || 0);
self.characters.resize_with(cell_count, || None);
self.buffer.resize_with(len, Default::default);
self.buffer.resize_with(len, Default::default);
self.buffer.resize_with(len, Default::default);
data.resize_with(len, || None);
this.resize_with(i + 1, || None);
table_entries.resize_with(msix_vectors as usize, Default::default);
pba_entries.resize_with(num_pba_entries, Default::default);
items.resize_with(capacity, Default::default);
empty_items.resize_with(self.items.len(), Vec::default);
handles.resize_with(widgets.len().saturating_sub(1), || DragHandle::new());
.resize_with(self.widgets.len().saturating_sub(1), || DragHandle::new());
detected.resize_with(DETECTED_SLOTS_NUM, || Mutex::new(HashSet::default()));
buffer.resize_with(width * height, Default::default);
palette.resize_with(palette_length * 3, Default::default);
buffer.resize_with(width * 3, Default::default);
self.v.resize_with((idx + 1).max(self.v.len()), || None);
v.resize_with(255, Default::default);
instances.resize_with(max_instances, || Mutex::new(None));
data.resize_with(size, T::default);
vals.resize_with(BUCKETS, Default::default);
.resize_with(fluid.num_particles(), || RwLock::new(Vec::new()))
.resize_with(fluid.num_particles(), || RwLock::new(Vec::new()))
.resize_with(boundary.num_particles(), || RwLock::new(Vec::new()))
pixels_resolved_this_stage.resize_with(n_workers, || Mutex::new(Vec::new()));
rtrees.resize_with((grid_width * grid_height) as usize, || RwLock::new(RTree::new());
handles.resize_with(INSTANCES_PER_RUN, || None);
self.cur_buffer.resize_with(cols * rows, Default::default);
.resize_with(self.cur_buffer.len(), Char::default);
self.0.resize_with(id, Default::default);
target_candidates.resize_with(test.targets(), Default::default);
res.resize_with(n, Point::identity);
vec.resize_with(Self::capacity(), Default::default);
v.resize_with(rounded_size, T::default);
.resize_with((self.width * self.height) as usize, GridStoreCell::default);
.resize_with((self.width * self.height) as usize, GridStoreCell::default);
self.0.resize_with(i + 1, Default::default);
buckets.resize_with(num_buckets, Default::default);
buckets.resize_with(num_buckets, Default::default);
texels.resize_with((w * h) as usize * pf.canals_len(), Default::default);
self.traces.resize_with(TRACE_SIZE, Trace::default);
hotplug_slots.resize_with(HOTPLUG_COUNT, HotPlugState::default);
d.resize_with(pair.0, Default::default);
vcpu_states.resize_with(usize::from(config.max_vcpus), VcpuState::default);
plaintext_buf.resize_with(ci.len(), u8::default);
notoken_packet.resize_with(1200, u8::default);
self.children.resize_with(elements_count, || Vec::new());
envs_with_walked_drivers.resize_with(expected_len, Assoc::new);
entries.resize_with(rows * cols, Default::default);
v.resize_with(size, Default::default);
vals.resize_with(BUCKETS, Default::default);
pixels.resize_with(capacity, Default::default);
enc.resize_with(enc.len() + octects_needed, Default::default);
inputs.resize_with(edge.to.1 + 1, || None);
ids.resize_with(maxlen, || None);
.resize_with(meta_count.0, Default::default);
self.lines.resize_with(rows.len(), Default::default);
out.resize_with(self.h.len(), || None::<Tensor>);
kvm.vec_events.resize_with(new_len, || None);
output.resize_with(output.capacity(), Default::default);
self.obj_vec.resize_with(self.num_world_tiles + 1, || None);
buf.resize_with(512, Default::default);
buf.resize_with(512, Default::default);
blob.resize_with(size as usize, Default::default);
self.threads.resize_with(tid + 1, Default::default);
enc_id_vec.resize_with(32, Default::default);
new_reserved.resize_with(new_len, || AtomicU32::new(0));
.resize_with(meta_count.0, Default::default);
self.children.resize_with(ENTRIES_PER_NODE, || None);
new_data.resize_with(row * new_stride, Default::default);
new_data.resize_with(row * new_stride, Default::default);
self.ifd_list.resize_with(ifd_num + 1, Default::default);
self.data.resize_with(new_size, Default::default);
col_mat.resize_with(sparse_len, || Arc::new(Mutex::new(vec![])));
prod.resize_with(lhs + 1, || Vec::new());
c::WSAEFAULT => buffer.resize_with(1 + (len as usize) / mem::size_of::<usize>(), Default::default),
chart.resize_with(toks.len() + 1, std::default::Default::default);
vals.resize_with(BUCKETS, Default::default);
in_out.resize_with(ct_len + AUTH_TAG_BYTES, || 0);
buf.resize_with(test_str.as_bytes().len(), Default::default);
quant_new_rec.resize_with(new_height, Default::default);
msgs.resize_with(n * MSG_SIZE, Default::default);

Non-default parameter

self.dirty.resize_with(cell_count, || true);
.resize_with((self.width * self.height) as usize, || value);
authorities.resize_with(30, || PeerId::random());
full_nodes.resize_with(30, || PeerId::random());
self.element_yoga_nodes.resize_with(elements_count, || unsafe { YGNodeNew() });
self.text_yoga_nodes.resize_with(texts_count, || unsafe { YGNodeNew() });
shards.resize_with(N_SHARDS, || (AccessQueue::default(), FastLock::new(Shard::new(shard_capacity))));
positions.resize_with(positions.len() + sub_niddle.len(), || { pos += 1; pos - 1 });
isf_data.passes.resize_with(isf.passes.len(), || ...); // https://github.com/nannou-org/nannou/blob/master/nannou_isf/src/pipeline.rs#L948-L958
self.tabs.resize_with(num_cols.0, || { let is_tabstop = index % INITIAL_TABSTOPS == 0; index += 1; is_tabstop });
self.inner.resize_with(new_len, f);
self.pointers.resize_with(stack + 1, || Pointer { start: last_end, end: last_end });
self.layouts.resize_with(texts_count, || TextLayoutState { font_size: 16., line_height: 20., single_line_width: 0., glyph_ids: Vec::new(), xs: Vec::new(), break_hints: Vec::new(), breaks: Vec::new() }});
new_args.resize_with(arg_count, || FlatArg::Default);
texts.resize_with(texts_count, || unsafe { (gpu.create_buffer(), 0, 0.) })
.resize_with(expected_len, || Subtype::underspecified(n("ddd_bit")).0);
*mgr += self.list.inner_mut().resize_with(len, |n| ListEntry::new(n, n == active));
self.pending_pushes.resize_with(WORK_GROUP_WIDTH as usize, || GpuPush { dir_id: [-1.0; 4] });
data.resize_with(length + 1, || rng.gen());
bufs.resize_with(capacity, || Mutex::new(Buf { status: BufStatus::Unused, data: vec![0; 1 << T::BLOCK_SIZE_LOG2 as usize] }));
data.resize_with(capacity + 1, MaybeUninit::uninit);
self.nodes.resize_with(1, || g.gen_leaf(0));
val.resize_with(x, || self.clone());
self.pending.resize_with(offset + 1, || Output { buffer: writer.buffer(), done: false });
alloc.resize_with(alloc_cell_w, || { let mut out = Vec::with_capacity(alloc_cell_w); out.resize(alloc_cell_w, None); out })
bnodes.resize_with(*id, BlankNode::default)
enc_id_vec.resize_with(32, Default::default);
self.data.resize_with(size, || 0);
v.resize_with(opt.k, || unreachable!());
v.resize_with(opt.k, || unreachable!());

No default implementation, but one could be derived

v.resize_with(nports as usize, || PortState::new());
self.entries.resize_with(capacity, || CacheEntry { items: Vec::new(), occupied: false });
fluid_fluid_contacts.resize_with(fluids.len(), || ParticlesContacts::new());
fluid_boundary_contacts.resize_with(fluids.len(), || ParticlesContacts::new());
boundary_boundary_contacts.resize_with(boundaries.len(), || ParticlesContacts::new());
v.resize_with(hub_desc.num_ports as usize, || super::PortState::new());
out.resize_with(r#in.len(), || MaybeUninit::<wasi::Event>::zeroed().assume_init());
out.resize_with(in_.len(), || { MaybeUninit::<wasi_unstable::Event>::zeroed().assume_init() })

UB

There was one usage from glutin crate that I'm pretty sure is an UB. After determining what is the type created by std::mem::zeroed, I noticed it's c_void, which shouldn't be constructed, and another approach should be used to have a zeroed allocation.

config_ids.resize_with(num_configs as usize, || std::mem::zeroed());

[lambda-fairy]

Thanks for the info @xfix.

If the majority of calls can use Default::default then that would support having a specialized method for that case.

[Shnatsel]

For completeness I've scanned crates.io for uses of .resize_with, exact command used was rg --iglob '*.rs' -F '.resize_with'. Here are the results: https://pastebin.com/UgpzzQxT

At a glance the results seem to align with the ones posted above - roughly half of the invocations just use the default value. On the other hand, there are just 207 uses of resize_with across the entire ecosystem, so I'm not sure adding another method for a relatively niche use case is worth it.

[Shnatsel]

For reference, unwrap_or_else occurs 24631 times and unwrap_or_default 6286 times.

[SoniEx2]

for reference, how many unwrap_or with the default value for the type (0, "", etc)? how many unwrap_or_else with Default::default? is it possible that the problem is elsewhere? (bad docs, bad book, we failed our users?)

[Shnatsel]

Here are all occurrences of .resize* and .unwrap_or* methods for your perusal:
resize.gz unwrap_or.gz

My point is, since the use of .resize_with is very rare, I'm not convinced adding further sugar for it is worth it.

[Timmmm]

My point is, since the use of .resize_with is very rare, I'm not convinced adding further sugar for it is worth it.

I think the point is that there are several other _default() methods so it should be present for consistency and discoverability, even if it is rarely used.

I'm pretty sure i128 is very rarely used but it would be weird to omit it when u128 exists.