Scylla: Permanent read/write fails after "bad_alloc"

[gibsn]

This is Scylla's bug tracker, to be used for reporting bugs only.
If you have a question about Scylla, and not a bug, please ask it in
our mailing-list at [email protected] or in our slack channel.

• [*] I have read the disclaimer above, and I am reporting a suspected malfunction in Scylla.

Installation details
Scylla version (or git commit hash): 3.2.4
Cluster size: 5+5 (multi DC)
OS (RHEL/CentOS/Ubuntu/AWS AMI): C7.5

Platform (physical/VM/cloud instance type/docker): bare metal
Hardware: sockets=2 x Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz cores=40 hyperthreading=yes memory= 6x32GB DDR4 2666MHz
Disks: RAID 10 of 10HDDs 14TB each for data, RAID 1 SSD 1TB for clogs

Hi!

The problem started with errors like "exception during mutation write to 10.161.180.24: std::bad_alloc (std::bad_alloc)" and led to one shard constantly failing a lot of (probably all) write/read operations until scylla-server was manually restarted. I guess that this can be due to having large partitions so here is what we have on that (we have 2 CFs):

becca/events histograms
Percentile  SSTables     Write Latency      Read Latency    Partition Size        Cell Count
                              (micros)          (micros)           (bytes)
50%             2.00             16.00          47988.50               770                29
75%             2.00             20.00          79061.50              5722               215
95%             6.00             33.00         185724.05             88148              2759
98%             8.00             36.00         239365.28            182785              5722
99%            10.00             46.73         295955.11            263210              8239
Min             0.00              1.00             20.00                73                 2
Max            24.00          29492.00        2051039.00         464228842           5839588

becca/events_by_ip histograms
Percentile  SSTables     Write Latency      Read Latency    Partition Size        Cell Count
                              (micros)          (micros)           (bytes)
50%             0.00             16.00              0.00              6866               179
75%             0.00             19.75              0.00             29521               770
95%             0.00             33.00              0.00            315852              8239
98%             0.00             41.00              0.00            785939             20501
99%             0.00             48.43              0.00           1629722             42510
Min             0.00              1.00              0.00                73                 0
Max             0.00          19498.00              0.00         386857368           4866323

Anyway if some big query arrived and failed I do not quite understand why all subsequent queries failed until the node was restarted.

Logs: https://cloud.mail.ru/public/C3AZ/RxPZyKUV6

Dashboard (by shard)

[gibsn]

Same problem on another node today

[gibsn]

same problem, the third node. we have 10 nodes, and so far first, second and third have been affected. I think I know what is gonna happen tomorrow

[dorlaor]

It's probably due to your wide partitions, the largest one is ~400MB size.
The subsequent queries fail since some need answers from tokens
on the busy shard and their requests get queued too.
We have a gradual improvement of our memory allocators so maybe things
will be slightly better with 4.0

Other than better data modeling, you can increase the ram or reduce the
amount of cores (run -smp 20), thus each shard will have more ram and can
allocate/free memory faster.

On Mon, May 11, 2020 at 7:06 AM Kirill Alekseev
[e-mail] wrote:
>

same problem, the third node. we have 10 nodes, and so fat first, second and third have been affected. I think I know what is gonna happen tomorrow

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

[gibsn]

@dorlaor thanks for reply

It's probably due to your wide partitions, the largest one is ~400MB size.

Maximum partition size has been constant for almost 6 months since we manually rate limit it. And this problem showed itself only now on 3.2.4 on different nodes. I can also see another similiar issue (#6254) on 3.2.4.

The subsequent queries fail since some need answers from tokens on the busy shard and their requests get queued too.

So after the big query fails and the queue is drained everything should be normal again? In our case it did not until a manual restart.

you can increase the ram or reduce

Can I somehow check what my RAM is used for? I would like to see how much RAM do BF and Index Summaries require but nodetool status output seems to be wrong (#4513)

Maybe it's worth noting that we have been bootstrapping a new node for 2 weeks now, but the problem shows itself both on nodes from and not from the DC the node belongs to. When bootstrapping is finished we can try updating to 4.0

[avikivity]

There are memory metrics in the dashboard. The important ones are LSA memory and non-LSA memory. LSA memory is used for memtables/cache, non-LSA is used for transient allocations (requests) and metadata.

Please share that dashboard (per-shard). If possible please upload your metrics database.

[avikivity]

Please also post nodetool cfstats.

[gibsn]

@avikivity,

Please share that dashboard (per-shard).

Dashboards:
https://cloud.mail.ru/public/BBbP/YKcZjt9kS -- all nodes recently
https://cloud.mail.ru/public/LqQc/YBMmhRkyk -- one node that failed at the time it failed

Unfortunately all the plugins that I have tried that take a screenshot of the whole page do not work properly for me :(

If possible please upload your metrics database.

This is gonna be tricky since we have one prometheus for all our services

Please also post nodetool cfstats.

cfstats_sc1.tar.gz

[gibsn]

Another node has just failed, now the 4th

[avikivity]

Looks at the Replica Memory charts. The orange shard has high non-LSA and low LSA (since their sum is shard memory). So the problem is running out of memory.

[avikivity]

From the cfstats data, bloom filters and summaries are not taking up too much space.

[avikivity]

Please check if all failures are always at the same shard number (and report which one it is). The bad shard is the one that has low LSA memory (near zero), high non-LSA memory, and low number of partitions in cache.

[avikivity]

Also, check if there is some way to filter out the unrelated metrics from Prometheus. It's likely I can dig something out of it if I have the full database.

[gibsn]

@avikivity

From the cfstats data, bloom filters and summaries are not taking up too much space.

I dont think nodetool status reports right info on BF and index summary size, I reported it in #4513.

Check this:

We create millions of new partitions every week and never remove nor TTL data, but BF size does not really grow

[avikivity]

It's in nodetool cfstats:

Keyspace : becca
        Read Count: 859262
        Read Latency: 0.04868451880800036 ms
        Write Count: 4867988419
        Write Latency: 1.6183266314381098E-5 ms
        Pending Flushes: 0
                Table: events
...
                Bloom filter off heap memory used: 132845488
                Index summary off heap memory used: 198411067

[avikivity]

Ref #2363

[gibsn]

@avikivity

It's in nodetool cfstats:

I know, we parse nodetool cfstats output and build graphs from it like the one I showed above. If we simply count sum of all Filter files sizes in Scylla directory it will be much bigger than the one provided by cfstats. I believe there is a bug

[avikivity]

Can you provide numbers?

Please provide sums per shard. You can figure out an sstable's shard by taking its generation number % number_of_shards.

[gibsn]

@avikivity
Ok, but it will take some time since I will have to ask another person who has access to production servers

[avikivity]

You haven't installed your ssh key there yet?

[gibsn]

I am developer and developers do not have access to production, only admins

[avikivity]

Sure but you can sneak the key in via the code.

[avikivity]

Do you have metrics of foreground/background writes and reads, per shard, meanwhile? Also connections per shard.

[gibsn]

And then sneak out for a new job I guess

[avikivity]

Sorry, the foreground/background reads/writes were there, and quite low. So it's something else.

[avikivity]

Please see if you can get the metrics database filtered, it is the easiest path to get some understanding.

[gibsn]

Please check if all failures are always at the same shard number (and report which one it is)

We had 4 fails on four different nodes by now, so I can not answer that

[avikivity]

Yes, but is it on the same shard (even if on a different node)?

[gibsn]

Yes, but is it on the same shard (even if on a different node)?

hosts 1,3: shard 7
host 2: shard 26
hosts 4: shard 22

[gibsn]

However I see that even though shard are different, all of the shards mentioned tend to have LSA smaller than most other shards on every node. Do you think those can be the shards responsible for IO queues? We have 4 of them

[avikivity]

It's possible. How many shards do you have? We can then see which shards have io_queues. @glommer please look too.

[gibsn]

host 1: 2 io queues, 30 shards
host 2,3,4: 4 io queues, 30 shards

[avikivity]

Unfortunately, the algorithm calls into the hwloc library to make the decision, so the algorithm isn't visible.

You can query via the prometheus interface for counters named scylla_reactor_aio_... that should which shards actually perform aio; these are the io queues.

[gibsn]

You can query via the prometheus interface for counters named scylla_reactor_aio_

Does not look like we can distinguish io queues :(

[avikivity]

Ah! I see there is an ioshard label. So if you look at those labels, you'll get the shards that have io queues.

[gibsn]

Please provide sums per shard.

As seen by ls -l:

Shard   Bloom Filter    Summary File
0   114.39 MB   96.03 MB
1   119.52 MB   96.08 MB
2   111.02 MB   95.31 MB
3   116.24 MB   95.51 MB
4   121.30 MB   95.62 MB
5   121.83 MB   95.40 MB
6   121.09 MB   95.39 MB
7   126.93 MB   96.08 MB
8   115.89 MB   95.53 MB
9   122.62 MB   94.63 MB
10  122.40 MB   94.50 MB
11  117.53 MB   94.63 MB
12  121.30 MB   94.14 MB
13  116.83 MB   94.49 MB
14  119.55 MB   93.73 MB
15  127.94 MB   93.71 MB
16  123.23 MB   94.12 MB
17  126.13 MB   94.62 MB
18  119.07 MB   94.35 MB
19  123.24 MB   94.67 MB
20  118.13 MB   95.05 MB
21  118.64 MB   94.36 MB
22  122.86 MB   94.16 MB
23  119.97 MB   94.90 MB
24  123.28 MB   95.10 MB
25  118.91 MB   95.05 MB
26  121.33 MB   95.52 MB
27  126.40 MB   94.80 MB
28  126.08 MB   95.08 MB
29  120.95 MB   95.36 MB

While cfstats for this node gives us:

                Bloom filter off heap memory used: 141708400
                Index summary off heap memory used: 233749582

[avikivity]

This is pretty bad. Please file a separate issue with this data.

[avikivity]

But don't miss my comment about the ioshard label.

[gibsn]

This is pretty bad. Please file a separate issue with this data.

I did 😄

4513

[gibsn]

Ah! I see there is an ioshard label. So if you look at those labels, you'll get the shards that have io queues.

host1 - 0, 14
host2 - 0, 4, 11, 17
host3 - 0, 6, 14, 22
host4 - 0, 6, 14, 22

[avikivity]

So it looks like it's not io_queue related.

[avikivity]

You can start a node with --abort-on-bad-alloc and then from the core we'll know what the problem is.

[gibsn]

Will it attempt to 'abort' gracefully or will it just crush probably corrupting clogs?

[avikivity]

It will crash, but the commitlogs will be fine. Better to enable it on just one node.

[gibsn]

--abort-on-seastar-bad-alloc or --abort-on-lsa-bad-alloc or both? 😄

[avikivity]

--abort-on-seastar-bad-alloc

[avikivity]

ref #6449

[slivne]

@avikivity / @gibsn - are we waiting for a reproducer with the flag ?

[gibsn]

@slivne we started a couple of nodes with flag --abort-on-seastar-bad-alloc but of course since that moment we had no fails, so waiting

[gibsn]

Ok, so another node has just failed and exited, though we did not start it with --abort-on-seastar-bad-alloc. We have BT and coredump

BTW this node has 1.5x more memory and 1.25x more shards

[gibsn]

we have this bt in scylla logs:

2020-05-18T17:18:15.001373+03:00 beccasc7 scylla[113779]: terminate called after throwing an instance of 'std::bad_alloc'
2020-05-18T17:18:15.002020+03:00 beccasc7 scylla[113779]: what():  std::bad_alloc
2020-05-18T17:18:15.002854+03:00 beccasc7 scylla[113779]: Aborting on shard 4.
2020-05-18T17:18:15.002861+03:00 beccasc7 scylla[113779]: Backtrace:
2020-05-18T17:18:15.002866+03:00 beccasc7 scylla[113779]: 0x0000000002ade972
2020-05-18T17:18:15.002871+03:00 beccasc7 scylla[113779]: 0x0000000002a5756b
2020-05-18T17:18:15.002876+03:00 beccasc7 scylla[113779]: 0x0000000002a57865
2020-05-18T17:18:15.002880+03:00 beccasc7 scylla[113779]: 0x0000000002a57900
2020-05-18T17:18:15.002885+03:00 beccasc7 scylla[113779]: 0x00007f4c54b46e7f
2020-05-18T17:18:15.002890+03:00 beccasc7 scylla[113779]: /opt/scylladb/libreloc/libc.so.6+0x0000000000037e74
2020-05-18T17:18:15.002895+03:00 beccasc7 scylla[113779]: /opt/scylladb/libreloc/libc.so.6+0x0000000000022894
2020-05-18T17:18:15.002900+03:00 beccasc7 scylla[113779]: /opt/scylladb/libreloc/libstdc++.so.6+0x000000000009e6b5
2020-05-18T17:18:15.002904+03:00 beccasc7 scylla[113779]: /opt/scylladb/libreloc/libstdc++.so.6+0x00000000000aa86b
2020-05-18T17:18:15.002909+03:00 beccasc7 scylla[113779]: /opt/scylladb/libreloc/libstdc++.so.6+0x00000000000a9878
2020-05-18T17:18:15.002913+03:00 beccasc7 scylla[113779]: /opt/scylladb/libreloc/libstdc++.so.6+0x00000000000aa223
2020-05-18T17:18:15.002918+03:00 beccasc7 scylla[113779]: 0x00007f4c54301c62
2020-05-18T17:18:15.002922+03:00 beccasc7 scylla[113779]: 0x00007f4c543021b0
2020-05-18T17:18:15.002927+03:00 beccasc7 scylla[113779]: /opt/scylladb/libreloc/libstdc++.so.6+0x00000000000aab1b
2020-05-18T17:18:15.002931+03:00 beccasc7 scylla[113779]: 0x00000000006ce0e5
2020-05-18T17:18:15.002936+03:00 beccasc7 scylla[113779]: 0x00000000006ceeb9
2020-05-18T17:18:15.002940+03:00 beccasc7 scylla[113779]: 0x0000000000a641a4
2020-05-18T17:18:15.002945+03:00 beccasc7 scylla[113779]: 0x0000000000a643fb
2020-05-18T17:18:15.002950+03:00 beccasc7 scylla[113779]: 0x0000000000d28da3
2020-05-18T17:18:15.002954+03:00 beccasc7 scylla[113779]: 0x0000000000d293ba
2020-05-18T17:18:15.002959+03:00 beccasc7 scylla[113779]: 0x0000000002a53ca1
2020-05-18T17:18:15.002964+03:00 beccasc7 scylla[113779]: 0x0000000002a53eaf
2020-05-18T17:18:15.002969+03:00 beccasc7 scylla[113779]: 0x0000000002abde05
2020-05-18T17:18:15.002974+03:00 beccasc7 scylla[113779]: 0x0000000002ad259c
2020-05-18T17:18:15.002978+03:00 beccasc7 scylla[113779]: 0x0000000002a368fd
2020-05-18T17:18:15.002983+03:00 beccasc7 scylla[113779]: /opt/scylladb/libreloc/libpthread.so.0+0x00000000000085a1
2020-05-18T17:18:15.002987+03:00 beccasc7 scylla[113779]: /opt/scylladb/libreloc/libc.so.6+0x00000000000fb302

and this manually taken from coredump:

(gdb) bt
#0  0x00007f4c54162e75 in ?? ()
#1  0xfffffff87ffbfadf in ?? ()
#2  0xffffffffffffffff in ?? ()
#3  0xffffffffffffffff in ?? ()
#4  0xffffffffffffffff in ?? ()
#5  0xffffffffffffffff in ?? ()
#6  0xffffffffffffffff in ?? ()
#7  0xffffffffffffffff in ?? ()
#8  0xffffffffffffffff in ?? ()
#9  0xffffffffffffffff in ?? ()
#10 0xffffffffffffffff in ?? ()
#11 0xffffffffffffffff in ?? ()
#12 0xffffffffffffffff in ?? ()
#13 0xffffffffffffffff in ?? ()
#14 0x00007f4c509e6e50 in ?? ()
#15 0x0000000000000000 in ?? ()

[gibsn]

Also we are seeing this message in our app logs:

Operation timed out for becca.events - received only 1 responses from 2 CL=ONE.

I can see in scylla code that this string is formed by scylla, not our connector (gocql). So if coordinator received one response with CL=ONE, why did it fail?

[gibsn]

@avikivity do you need anything else from us?

[avikivity]

Sorry, I missed the notification. Can you upload the core dump?

[gibsn]

@avikivity we can not since it contains user data

[avikivity]

Ho hum. Then please open the core dump in a container based on the image docker.io/scylladb/scylla-toolchain:fedora-30-20191022:

1. create the container, with bind mounts to some directory where you have the core
2. install the scylla-server and scylla-debuginfo rpms (same version)
3. run gdb /opt/scylladb/libexec/scylla /path/to/core
4. check out that backtrace etc. work
5. find scylla-gdb.py in /usr/src/debug
6. execute source /path/to/scylla-gdb.py
7. execute scylla memory and scylla task_histogram and post the results.

Please keep the container open in case we need more info.

[gibsn]

@avikivity sry for the delay. I could not launch fedora-30-20191022 (it would just exit) so I am taking the closest version fedora-31-20191030. Also I could not find fedora repos for 3.2x (naively changing "centos" to "fedora" in centos repos does not help) so now I am downloading rpms to manually install it. Should I go any other direction?

I also tried running gdb not in container but on a regular centos 7 machine (like we have in prod), it did output some meaningful bt, however it is missing some libs like boost etc. But if that is enough I can provide it

[avikivity]

@tgrabiec do you remember if we have a receipe for debugging relocatable packages? if not I will write one.

[gibsn]

@avikivity

run gdb /opt/scylladb/libexec/scylla /path/to/core

https://pastebin.com/6uqVPYR8

find scylla-gdb.py in /usr/src/debug
execute source /path/to/scylla-gdb.py
execute scylla memory and scylla task_histogram and post the results.

(gdb) source /opt/scylladb/scripts/scylla-gdb.py
Traceback (most recent call last):
  File "/opt/scylladb/scripts/scylla-gdb.py", line 3120, in <module>
    scylla_smp_queues()
  File "/opt/scylladb/scripts/scylla-gdb.py", line 2961, in __init__
    qs = std_unique_ptr(gdb.parse_and_eval('seastar::smp::_qs')).get()
gdb.error: No type "smp" within class or namespace "seastar".
(gdb) scylla memory
Python Exception <class 'gdb.error'> Cannot find thread-local storage for LWP 113784, executable file /opt/scylladb/libexec/scylla:
Cannot find thread-local variables on this target:
Error occurred in Python: Cannot find thread-local storage for LWP 113784, executable file /opt/scylladb/libexec/scylla:
Cannot find thread-local variables on this target
(gdb) scylla task_histogram
Python Exception <class 'gdb.error'> Cannot find thread-local storage for LWP 113784, executable file /opt/scylladb/libexec/scylla:
Cannot find thread-local variables on this target:
Error occurred in Python: Cannot find thread-local storage for LWP 113784, executable file /opt/scylladb/libexec/scylla:
Cannot find thread-local variables on this target

[tgrabiec]

@avikivity There's something here: https://github.com/scylladb/scylla/blob/master/docs/debugging.md#relocatable-binaries

[gibsn]

Now we are getting two nodes failing at the same every day, from different DCs :( so our cluster becomes unresponsive

[gibsn]

Also the node that we started with --abort-on-seastar-bad-alloc did not actually abort

scylla   32744 1025 87.4 17180598224 115141380 ? Rsl May13 112121:58 /usr/bin/scylla --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --collectd-address=127.0.0.1:25826 --collectd=1 --collectd-poll-period 3000 --network-stack posix --memory 110G --abort-on-seastar-bad-alloc --num-io-queues=2 --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-15,17-31

[gibsn]

Also do you know why we might be getting this message?

Operation timed out for becca.events - received only 1 responses from 2 CL=ONE.

Lots of our retries fail with this message

[gibsn]

(gdb) source /opt/scylladb/scripts/scylla-gdb.py
Traceback (most recent call last):
  File "/opt/scylladb/scripts/scylla-gdb.py", line 3120, in <module>
    scylla_smp_queues()
  File "/opt/scylladb/scripts/scylla-gdb.py", line 2961, in __init__
    qs = std_unique_ptr(gdb.parse_and_eval('seastar::smp::_qs')).get()
gdb.error: No type "smp" within class or namespace "seastar".

This helps:
https://github.com/scylladb/scylla/blob/master/docs/debugging.md#namespace-issues

Error occurred in Python: Cannot find thread-local storage for LWP 113784, executable file /opt/scylladb/libexec/scylla:

I tried running info sharedlibrary and seems like all the libraries have been found. However I am getting lots of warnings on GDB startup:
https://pastebin.com/PpDju8WC

UPD: echo "set auto-load safe-path /" > /root/.gdbinit helped

scylla memory:
https://pastebin.com/rVPByMku

scylla task_histogram:
still waiting for output

[avikivity]

Copying scylla memory output here for reference

Used memory:    4802478080
Free memory:             0
Total memory:   4802478080

LSA:
  allocated:     650248192
  used:          650117120
  free:             131072

Cache:
  total:            131072
  used:             131072
  free:                  0

Memtables:
 total:           650117120
 Regular:
  real dirty:     649723904
  virt dirty:     649723904
 System:
  real dirty:        262144
  virt dirty:        262144
 Streaming:
  real dirty:             0
  virt dirty:             0

[avikivity]

We really need to package all the hacks used to start gdb into a nice scripts.

[avikivity]

Well, it's a real OOM. Are you sure you didn't miss some scylla memory output? It should say a lot more.

[gibsn]

scylla task_histogram has been running for half an hour with no output

[gibsn]

Well, it's a real OOM. Are you sure you didn't miss some scylla memory output? It should say a lot more.

Sry, I did miss. This's the missing part:

Coordinator:
  fg writes:              0
  bg writes:              0, 0 B
  fg reads:               0
  bg reads:               0
  hints:                  0 B
  view hints:             0 B

Replica:
  Read Concurrency Semaphores:
    user sstable reads:        0/100, remaining mem:      96049561 B, queued: 0
    streaming sstable reads:   1/ 10, remaining mem:      96049561 B, queued: 0
    system sstable reads:      0/ 10, remaining mem:      96049561 B, queued: 0
  Execution Stages:
    data query stage:
Python Exception <class 'gdb.error'> There is no member named dummy.:
Error occurred in Python: There is no member named dummy.

[avikivity]

There should be much more. Please pick up the latest scylla-gdb.py from master and try to use it instead.

[gibsn]

version c29ccdea7e74dd5a8894a6f3186592eb56d7ec2a:

scylla memory: https://pastebin.com/828hnSh7
scylla task_histogram: https://pastebin.com/JM1nGFSe

[avikivity]

scylla memory:

Replica:
  Read Concurrency Semaphores:
    user sstable reads:        0/100, remaining mem:      96049561 B, queued: 0
    streaming sstable reads:   1/ 10, remaining mem:      96049561 B, queued: 0
    system sstable reads:      0/ 10, remaining mem:      96049561 B, queued: 0
  Execution Stages:
    data query stage:
         Total                            0
    mutation query stage:
         Total                            0
    apply stage:
         Total                            0
  Tables - Ongoing Operations:
    pending writes phaser (top 10):
              0 Total (all)
    pending reads phaser (top 10):
              0 Total (all)
    pending streams phaser (top 10):
              0 Total (all)

Small pools:
objsz spansz    usedobj       memory       unused  wst%
    1   4096          0            0            0   0.0
    1   4096          0            0            0   0.0
    1   4096          0            0            0   0.0
    1   4096          0            0            0   0.0
    2   4096          0            0            0   0.0
    2   4096          0            0            0   0.0
    3   4096          0            0            0   0.0
    3   4096          0            0            0   0.0
    4   4096          0            0            0   0.0
    5   4096          0            0            0   0.0
    6   4096          0            0            0   0.0
    7   4096          0            0            0   0.0
    8   4096       1275        16384         6184  37.7
   10   4096          0         8192         8192  99.9
   12   4096        171         8192         6140  74.9
   14   4096          0         8192         8192  99.8
   16   4096     370938      5939200         4192   0.1
   20   4096       2021        40960          540   0.9
   24   4096      95057      2293760        12392   0.1
   28   4096       1229        36864         2452   6.5
   32   4096     188441      6037504         7392   0.1
   40   4096     175698      7061504        33584   0.1
   48   4096       9793       479232         9168   1.5
   56   4096     164980      9256960        18080   0.0
   64   4096     944904     60477440         3584   0.0
   80   4096      95906      7708672        36192   0.1
   96   4096       3818       380928        14400   2.2
  112   4096       2443       286720        13104   3.0
  128   4096       1000       139264        11264   8.1
  160   4096       5013       835584        33504   1.7
  192   4096        525       114688        13888  10.5
  224   4096      92685     21110784       349344   0.1
  256   4096        625       180224        20224  11.2
  320   8192        460       180224        33024  16.0
  384   8192        399       188416        35200  17.1
  448   4096        310       184320        45440  23.1
  512   4096     184829     94674944        42496   0.0
  640  12288      92210     60489728      1475328   0.1
  768  12288       1383      1146880        84736   5.8
  896   8192        244       311296        92672  28.2
 1024   4096        651       761856        95232  12.5
 1280  20480      91909    120520704      2877184   0.0
 1536  12288          3       131072       126464  90.2
 1792  16384        102       294912       112128  36.5
 2048   8192        535      1196032       100352   8.4
 2560  20480        105       458752       189952  35.2
 3072  12288         27       245760       162816  60.0
 3584  28672          7       294912       269824  89.9
 4096  16384      14162     58294272       286720   0.5
 5120  20480          9       425984       379904  82.9
 6144  24576         41       491520       239616  42.5
 7168  28672         20       622592       479232  64.5
 8192  32768        766      7077888       802816  11.3
10240  40960          7       851968       780288  85.3
12288  49152         20       983040       737280  68.8
14336  57344          7      1245184      1144832  79.4
16384  65536        251      5570560      1458176  26.2
Small allocations: 479064064 [B]
Page spans:
index      size [B]      free [B]     large [B] [spans]
    0          4096             0             0       0
    1          8192             0             0       0
    2         16384             0             0       0
    3         32768             0    3014000640   91980
    4         65536             0      11272192     172
    5        131072             0    1228931072    9376
    6        262144             0             0       0
    7        524288             0             0       0
    8       1048576             0             0       0
    9       2097152             0       2097152       1
   10       4194304             0             0       0
   11       8388608             0             0       0
   12      16777216             0             0       0
   13      33554432             0             0       0
   14      67108864             0      67108864       1
   15     134217728             0             0       0
   16     268435456             0             0       0
   17     536870912             0             0       0
   18    1073741824             0             0       0
   19    2147483648             0             0       0
   20    4294967296             0             0       0
   21    8589934592             0             0       0
   22   17179869184             0             0       0
   23   34359738368             0             0       0
   24   68719476736             0             0       0
   25  137438953472             0             0       0
   26  274877906944             0             0       0
   27  549755813888             0             0       0
   28 1099511627776             0             0       0
   29 2199023255552             0             0       0
   30 4398046511104             0             0       0
   31 8796093022208             0             0       0
Large allocations: 4323409920 [B]

[avikivity]

task_histogram:

(gdb) scylla task_histogram
     51746: 0x3010950 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::handle_exception<dirty_memory_manager::flush_one(memtable_list&, flush_permit&&)::{lambda(std::__exception_ptr::exception_ptr)#1}>(dirty_memory_manager::flush_one(memtable_list&, flush_permit&&)::{lambda(std::__exception_ptr::exception_ptr)#1}&&)::{lambda(auto:1)#1}, seastar::future<> >(seastar::future<>::handle_exception<dirty_memory_manager::flush_one(memtable_list&, flush_permit&&)::{lambda(std::__exception_ptr::exception_ptr)#1}>(dirty_memory_manager::flush_one(memtable_list&, flush_permit&&)::{lambda(std::__exception_ptr::exception_ptr)#1}&&)::{lambda(auto:1)#1})::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16
     51729: 0x30144e8 vtable for seastar::continuation<seastar::future<>::then_impl<utils::phased_barrier::advance_and_await()::{lambda()#1}, seastar::future<> >(utils::phased_barrier::advance_and_await()::{lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16
     28919: 0x30269b0 vtable for seastar::shared_ptr_count_for<checked_file_impl> + 16
     28277: 0x313d030 vtable for db::commitlog::segment_manager + 16
     27759: 0x313d050 vtable for db::commitlog::segment + 16
     27360: 0x332c5f8 vtable for seastar::append_challenged_posix_file_impl + 16
     26499: 0x33307b8 vtable for seastar::metrics::impl::metric_groups_impl + 16
       346: 0x3028d38 vtable for seastar::shared_ptr_count_for<cql3::column_specification> + 16
       344: 0x3028d58 vtable for seastar::shared_ptr_count_for<cql3::column_identifier> + 16
       307: 0x3330850 vtable for seastar::shared_ptr_count_for<seastar::metrics::impl::registered_metric> + 16
       118: 0x3344ad8 vtable for seastar::httpd::function_handler + 16
        66: 0x3157488 vtable for utils::config_file::named_value<unsigned int>::the_value_type + 16
        46: 0x3185878 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::finally_body<seastar::smp::submit_to<seastar::rpc::sink_impl<netw::serializer, frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>::operator()(frozen_mutation_fragment const&, streaming::stream_mutation_fragments_cmd const&)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)#1}::operator()(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)::{lambda()#1}>(unsigned int, seastar::smp_service_group, seastar::futurize&&)::{lambda()#1}, false>, seastar::future<> >(seastar::futurize<std::result_of<seastar::future<>::finally_body<seastar::smp::submit_to<seastar::rpc::sink_impl<netw::serializer, frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>::operator()(frozen_mutation_fragment const&, streaming::stream_mutation_fragments_cmd const&)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)#1}::operator()(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)::{lambda()#1}>(unsigned int, seastar::smp_service_group, seastar::futurize&&)::{lambda()#1}, false> ()>::type>::type)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16
        44: 0x306d0f8 vtable for sstables::bag_sstable_set + 16
        40: 0x3031848 vtable for seastar::shared_ptr_count_for<logalloc::region_impl> + 16
        39: 0x332c828 vtable for seastar::shared_ptr_count_for<seastar::posix_file_impl> + 16
        37: 0x3060af8 vtable for sstables::compaction_metadata + 16
        35: 0x3026420 vtable for seastar::free_deleter_impl + 16
        35: 0x3185058 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::rpc::sink_impl<netw::serializer, frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>::operator()(frozen_mutation_fragment const&, streaming::stream_mutation_fragments_cmd const&)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)#1}::operator()(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)::{lambda(seastar::future<>)#2}, seastar::future<> >(seastar::rpc::sink_impl<netw::serializer, frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>::operator()(frozen_mutation_fragment const&, streaming::stream_mutation_fragments_cmd const&)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)#1}::operator()(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)::{lambda(seastar::future<>)#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16
        33: 0x32495f0 vtable for seastar::lambda_deleter_impl<std::_Bind<void (reader_concurrency_semaphore::reader_permit::*(seastar::lw_share--Type <RET> for more, q to quit, c to continue without paging--scylla task_histogramc
d_ptr<reader_concurrency_semaphore::reader_permit>, unsigned long))(unsigned long)> > + 16
        31: 0x30318a8 vtable for buffer_data_source_impl + 16
        30: 0x3028d78 vtable for seastar::shared_ptr_count_for<query::partition_slice> + 16
        29: 0x3060b28 vtable for sstables::stats_metadata + 16
        27: 0x30327d8 vtable for seastar::internal::do_until_state<mutation_cleaner_impl::start_worker()::{lambda()#1}::operator()()::{lambda()#1}::operator()() const::{lambda()#1}, seastar::condition_variable::wait<{lambda()#1}>({lambda()#1}&&)::{lambda()#1}> + 16
        26: 0x3032760 vtable for seastar::internal::repeater<mutation_cleaner_impl::start_worker()::{lambda()#1}> + 16
        22: 0x3163b50 vtable for utils::filter::murmur3_bloom_filter + 16
        22: 0x3344c08 vtable for seastar::httpd::param_matcher + 16
        21: 0x3157458 vtable for utils::config_file::named_value<bool>::the_value_type + 16
        20: 0x3060ac8 vtable for sstables::validation_metadata + 16
        20: 0x3344c30 vtable for seastar::httpd::str_matcher + 16

[avikivity]

Looks like lots of memory in 32k spans.

[avikivity]

The huge number of metric_group_impls is suspicious. You don't have a huge number of tables, do you?

[gibsn]

I have 2 tables, others are system

[avikivity]

Very odd.

[avikivity]

Please run scylla task_histogram --all, maybe we'll get more information.

[gibsn]

definitely more information:
https://pastebin.com/5jUY4k9N

[avikivity]

    162487: 0x3010950 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::handle_exception<dirty_memory_manager::flush_one(memtable_list&, flush_permit&&)::{lambda(std::__exception_ptr::exception_ptr)#1}>(dirty_memory_manager::flush_one(memtable_list&, flush_permit&&)::{lambda(std::__exception_ptr::exception_ptr)#1}&&)::{lambda(auto:1)#1}, seastar::future<> >(seastar::future<>::handle_exception<dirty_memory_manager::flush_one(memtable_list&, flush_permit&&)::{lambda(std::__exception_ptr::exception_ptr)#1}>(dirty_memory_manager::flush_one(memtable_list&, flush_permit&&)::{lambda(std::__exception_ptr::exception_ptr)#1}&&)::{lambda(auto:1)#1})::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
    162485: 0x30144e8 vtable for seastar::continuation<seastar::future<>::then_impl<utils::phased_barrier::advance_and_await()::{lambda()#1}, seastar::future<> >(utils::phased_barrier::advance_and_await()::{lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
     91986: 0x33307b8 vtable for seastar::metrics::impl::metric_groups_impl + 16 
     91964: 0x30269b0 vtable for seastar::shared_ptr_count_for<checked_file_impl> + 16 
     91824: 0x332c5f8 vtable for seastar::append_challenged_posix_file_impl + 16 
     91824: 0x313d050 vtable for db::commitlog::segment + 16 
     91653: 0x313d030 vtable for db::commitlog::segment_manager + 16 
      1152: 0x3028d38 vtable for seastar::shared_ptr_count_for<cql3::column_specification> + 16 
      1152: 0x3028d58 vtable for seastar::shared_ptr_count_for<cql3::column_identifier> + 16 
       982: 0x3330850 vtable for seastar::shared_ptr_count_for<seastar::metrics::impl::registered_metric> + 16 
       372: 0x3344ad8 vtable for seastar::httpd::function_handler + 16 
       182: 0x3185878 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::finally_body<seastar::smp::submit_to<seastar::rpc::sink_impl<netw::serializer, frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>::operator()(frozen_mutation_--Type <RET> for more, q to quit, c to continue without paging--c
fragment const&, streaming::stream_mutation_fragments_cmd const&)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)#1}::operator()(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)::{lambda()#1}>(unsigned int, seastar::smp_service_group, seastar::futurize&&)::{lambda()#1}, false>, seastar::future<> >(seastar::futurize<std::result_of<seastar::future<>::finally_body<seastar::smp::submit_to<seastar::rpc::sink_impl<netw::serializer, frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>::operator()(frozen_mutation_fragment const&, streaming::stream_mutation_fragments_cmd const&)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)#1}::operator()(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)::{lambda()#1}>(unsigned int, seastar::smp_service_group, seastar::futurize&&)::{lambda()#1}, false> ()>::type>::type)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
       182: 0x3185058 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::rpc::sink_impl<netw::serializer, frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>::operator()(frozen_mutation_fragment const&, streaming::stream_mutation_fragments_cmd const&)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)#1}::operator()(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)::{lambda(seastar::future<>)#2}, seastar::future<> >(seastar::rpc::sink_impl<netw::serializer, frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>::operator()(frozen_mutation_fragment const&, streaming::stream_mutation_fragments_cmd const&)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)#1}::operator()(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)::{lambda(seastar::future<>)#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
       148: 0x32495f0 vtable for seastar::lambda_deleter_impl<std::_Bind<void (reader_concurrency_semaphore::reader_permit::*(seastar::lw_shared_ptr<reader_concurrency_semaphore::reader_permit>, unsigned long))(unsigned long)> > + 16 
       140: 0x332c828 vtable for seastar::shared_ptr_count_for<seastar::posix_file_impl> + 16 
       114: 0x3026420 vtable for seastar::free_deleter_impl + 16 
        99: 0x3344c08 vtable for seastar::httpd::param_matcher + 16 
        98: 0x3344c30 vtable for seastar::httpd::str_matcher + 16 
        91: 0x30327d8 vtable for seastar::internal::do_until_state<mutation_cleaner_impl::start_worker()::{lambda()#1}::operator()()::{lambda()#1}::operator()() const::{lambda()#1}, seastar::condition_variable::wait<{lambda()#1}>({lambda()#1}&&)::{lambda()#1}> + 16 
        91: 0x3032760 vtable for seastar::internal::repeater<mutation_cleaner_impl::start_worker()::{lambda()#1}> + 16 
        91: 0x3032878 vtable for seastar::continuation<seastar::future<>::then_impl<mutation_cleaner_impl::start_worker()::{lambda()#1}::operator()()::{lambda()#1}::operator()() const::{lambda()#2}, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >(mutation_cleaner_impl::start_worker()::{lambda()#1}::operator()()::{lambda()#1}::operator()() const::{lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
        90: 0x3031848 vtable for seastar::shared_ptr_count_for<logalloc::region_impl> + 16 
        84: 0x3028d78 vtable for seastar::shared_ptr_count_for<query::partition_slice> + 16 
        83: 0x306d0f8 vtable for sstables::bag_sstable_set + 16 
        77: 0x3157488 vtable for utils::config_file::named_value<unsigned int>::the_value_type + 16 
        69: 0x3060af8 vtable for sstables::compaction_metadata + 16 
        69: 0x3163b50 vtable for utils::filter::murmur3_bloom_filter + 16 
        69: 0x3060b58 vtable for sstables::serialization_header + 16 
        69: 0x3060b28 vtable for sstables::stats_metadata + 16 
        69: 0x3060ac8 vtable for sstables::validation_metadata + 16 
        66: 0x30318a8 vtable for buffer_data_source_impl + 16 
        52: 0x3079ff8 vtable for cql3::constants::marker + 16 
        47: 0x3157458 vtable for utils::config_file::named_value<bool>::the_value_type + 16 
        40: 0x306d430 vtable for seastar::shared_ptr_count_for<sstables::size_tiered_compaction_strategy> + 16 
        40: 0x306d098 vtable for size_tiered_backlog_tracker + 16 
        39: 0x3157730 vtable for utils::config_file::named_value<seastar::basic_sstring<char, unsigned int, 15u, true> >::the_value_type + 16 
        37: 0x30bee20 vtable for seastar::shared_ptr_count_for<cql3::restrictions::single_column_restriction::EQ> + 16 
        36: 0x30bf030 vtable for seastar::shared_ptr_count_for<cql3::restrictions::single_column_restrictions> + 16 
        34: 0x332edd0 vtable for seastar::file_data_source_impl + 16 
        33: 0x334e040 vtable for seastar::net::posix_data_source_impl + 16 
        33: 0x334dee8 vtable for seastar::net::posix_connected_socket_impl + 16 
        33: 0x334e078 vtable for seastar::net::posix_data_sink_impl + 16 
        33: 0x30654a0 vtable for compressed_file_data_source_impl<crc32_utils> + 16 
        33: 0x334cbd8 vtable for seastar::continuation<seastar::future<unsigned long>::then_impl<seastar::net::posix_data_source_impl::get()::{lambda(unsigned long)#1}, seastar::future<seastar::temporary_buffer<char> > >(seastar::net::posix_data_source_impl::get()::{lambda(unsigned long)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<unsigned long>&&)#1}, unsigned long> + 16 
        33: 0x3339248 vtable for seastar::continuation<seastar::future<>::then_impl<seastar::reactor::read_some(seastar::pollable_fd_state&, void*, unsigned long)::{lambda()#1}, seastar::future<unsigned long> >(seastar::reactor::read_some(seastar::pollable_fd_state&, void*, unsigned long)::{lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
        33: 0x306ae30 vtable for sstables::sstable_mutation_reader<sstables::data_consume_rows_context_m, sstables::mp_row_consumer_m> + 16 
        33: 0x3249610 vtable for seastar::shared_ptr_count_for<tracking_file_impl> + 16 
        32: 0x3061370 vtable for seastar::continuation<seastar::future<seastar::temporary_buffer<char> >::then_impl<seastar::input_stream<char>::read_exactly(unsigned long)::{lambda(auto:1)#1}, seastar::future<seastar::temporary_buffer<char> > >(seastar::input_stream<char>::read_exactly(unsigned long)::{lambda(auto:1)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::temporary_buffer<char> >&&)#1}, seastar::temporary_buffer<char> > + 16 
        28: 0x3061fd0 vtable for boost::detail::sp_counted_impl_p<boost::signals2::dummy_mutex> + 16 
        28: 0x3061f88 vtable for boost::detail::sp_counted_impl_p<boost::signals2::detail::signal_impl<void (), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void ()>, boost::function<void (boost::signals2::connection const&)>, boost::signals2::dummy_mutex>::invocation_state> + 16 
        28: 0x30623a8 vtable for boost::detail::sp_counted_impl_p<boost::signals2::optional_last_value<void> > + 16 
        28: 0x3062360 vtable for boost::detail::sp_counted_impl_p<boost::signals2::detail::grouped_list<int, std::less<int>, boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (), boost::function<void ()> >, boost::signals2::dummy_mutex> > > > + 16 
        28: 0x3061410 vtable for boost::detail::sp_counted_impl_p<boost::signals2::detail::signal_impl<void (), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void ()>, boost::function<void (boost::signals2::connection const&)>, boost::signals2::dummy_mutex> > + 16 
        17: 0x3345cd0 vtable for seastar::httpd::content_replace + 16 
        16: 0x3342298 vtable for seastar::httpd::file_handler + 16 
        16: 0x30b5de8 vtable for seastar::shared_ptr_count_for<cql3::restrictions::statement_restrictions> + 16 
        15: 0x335e3d8 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::handle_exception<seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)1>()::{lambda(std::__exception_ptr::exception_ptr)#3}>(seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)1>()::{lambda(std::__exception_ptr::exception_ptr)#3}&&)::{lambda(auto:1)#1}, seastar::future<> >(seastar::future<>::handle_exception<seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)1>()::{lambda(std::__exception_ptr::exception_ptr)#3}>(seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)1>()::{lambda(std::__exception_ptr::exception_ptr)#3}&&)::{lambda(auto:1)#1})::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
        15: 0x335a938 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::rpc::client::client(seastar::rpc::logger const&, void*, seastar::rpc::client_options, seastar::socket, seastar::socket_address const&, seastar::socket_address const&)::{lambda(seastar::future<>)#2}, seastar::future<> >(seastar::rpc::client::client(seastar::rpc::logger const&, void*, seastar::rpc::client_options, seastar::socket, seastar::socket_address const&, seastar::socket_address const&)::{lambda(seastar::future<>)#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
        15: 0x334df58 vtable for seastar::net::posix_socket_impl + 16 
        15: 0x335dfd8 vtable for seastar::internal::do_until_state<seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)1>()::{lambda()#1}, seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)1>()::{lambda()#2}> + 16 
        15: 0x335a6b8 vtable for seastar::internal::do_until_state<seastar::rpc::client::client(seastar::rpc::logger const&, void*, seastar::rpc::client_options, seastar::socket, seastar::socket_address const&, seastar::socket_address const&)::{lambda(seastar::connected_socket)#1}::operator()(seastar::connected_socket) const::{lambda()#2}::operator()() const::{lambda()#1}, seastar::rpc::client::connected_socket(seastar::rpc::logger const&, void*, seastar::rpc::client_options, seastar::socket, seastar::socket_address const&, seastar::socket_address const&)::{lambda(seastar::connected_socket)#1}::operator()(seastar::connected_socket) const::{lambda(seastar::connected_socket)#1}::operator()() const::{lambda()#2}> + 16 
        14: 0x335e298 vtable for seastar::continuation<seastar::future<seastar::temporary_buffer<char> >::then_impl<seastar::rpc::connection::read_frame<seastar::rpc::response_frame>(seastar::socket_address, seastar::input_stream<char>&)::{lambda(seastar::temporary_buffer<char>)#1}, seastar::future<std::tuple<long, std::optional<seastar::rpc::rcv_buf> > > >(seastar::rpc::connection::read_frame<seastar::rpc::response_frame>(seastar::socket_address, seastar::input_stream<char>&)::{lambda(seastar::temporary_buffer<char>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::temporary_buffer<char> >&&)#1}, seastar::temporary_buffer<char> > + 16 
        14: 0x31c9a58 vtable for seastar::rpc::protocol<netw::serializer, netw::messaging_verb>::client + 16 
        14: 0x335e428 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::handle_exception<seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)0>()::{lambda(std::__exception_ptr::exception_ptr)#3}>(seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)0>()::{lambda(std::__exception_ptr::exception_ptr)#3}&&)::{lambda(auto:1)#1}, seastar::future<> >(seastar::future<>::handle_exception<seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)0>()::{lambda(std::__exception_ptr::exception_ptr)#3}>(seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)0>()::{lambda(std::__exception_ptr::exception_ptr)#3}&&)::{lambda(auto:1)#1})::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
        14: 0x335e1d0 vtable for seastar::continuation<seastar::future<>::then_impl<seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)0>()::{lambda()#2}::operator()() const::{lambda()#2}, seastar::future<> >(seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)0>()::{lambda()#2}::operator()() const::{lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
        14: 0x335aad0 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::rpc::server::connection::process()::{lambda(seastar::future<>)#2}, seastar::future<> >(seastar::rpc::server::connection::process()::{lambda(seastar::future<>)#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
        14: 0x335e180 vtable for seastar::continuation<seastar::future<>::then_impl<seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)1>()::{lambda()#2}::operator()() const::{lambda()#2}, seastar::future<> >(seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)1>()::{lambda()#2}::operator()() const::{lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
        14: 0x335a6e0 vtable for seastar::internal::do_until_state<seastar::rpc::server::connection::process()::{lambda()#1}::operator()()::{lambda()#1}::operator()() const::{lambda()#1}, seastar::rpc::server::connection::process()::{lambda()#1}::operator()()::{lambda()#1}::operator()() const::{lambda()#2}> + 16 
        14: 0x335a898 vtable for seastar::continuation<seastar::future<std::tuple<long, std::optional<seastar::rpc::rcv_buf> > >::then_impl<seastar::rpc::client::client(seastar::rpc::logger const&, void*, seastar::rpc::client_options, seastar::socket, seastar::socket_address const&, seastar::socket_address const&)::{lambda(seastar::connected_socket)#1}::operator()(seastar::connected_socket) const::{lambda()#2}::operator()() const::{lambda()#2}::operator()()::{lambda(std::tuple<long, std::optional<seastar::rpc::rcv_buf> >)#1}, seastar::future<> >(seastar::rpc::client::client(seastar::rpc::logger const&, void*, seastar::rpc::client_options, seastar::socket, seastar::socket_address const&, seastar::socket_address const&)::{lambda(seastar::connected_socket)#1}::operator()(seastar::connected_socket) const::{lambda()#2}::operator()() const::{lambda()#2}::operator()()::{lambda(std::tuple<long, std::optional<seastar::rpc::rcv_buf> >)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<std::tuple<long, std::optional<seastar::rpc::rcv_buf> > >&&)#1}, std::tuple<long, std::optional<seastar::rpc::rcv_buf> > > + 16 
        14: 0x31ca9f8 vtable for seastar::shared_ptr_count_for<netw::messaging_service::rpc_protocol_client_wrapper> + 16 
        14: 0x335ddc8 vtable for seastar::rpc::server::connection + 16 
        14: 0x3151f60 vtable for boost::any::holder<unsigned int> + 16 
        14: 0x335e090 vtable for seastar::internal::do_until_state<seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)0>()::{lambda()#2}::operator()() const::{lambda()#1}, seastar::condition_variable::wait<{lambda()#2}>({lambda()#2}&&)::{lambda()#1}> + 16 
        14: 0x3152020 vtable for boost::any::holder<unsigned long> + 16 
        14: 0x335ad08 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::finally_body<seastar::rpc::server::connection::process()::{lambda()#3}, false>, seastar::future<> >(seastar::future<>::finally_body<seastar::rpc::server::connection::process()::{lambda()#3}, false>&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
        14: 0x335e000 vtable for seastar::internal::do_until_state<seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)0>()::{lambda()#1}, seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)0>()::{lambda()#2}> + 16 
        14: 0x335e068 vtable for seastar::internal::do_until_state<seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)1>()::{lambda()#2}::operator()() const::{lambda()#1}, seastar::condition_variable::wait<{lambda()#2}>({lambda()#2}&&)::{lambda()#1}> + 16 
        14: 0x31ca9c8 vtable for boost::any::holder<gms::inet_address> + 16 
        14: 0x335aa58 vtable for seastar::continuation<seastar::future<std::tuple<std::optional<unsigned long>, unsigned long, long, std::optional<seastar::rpc::rcv_buf> > >::then_impl<seastar::rpc::server::connection::process()::{lambda()#1}::operator()()::{lambda()#1}::operator()() const::{lambda()#2}::operator()()::{lambda(std::tuple<std::optional<unsigned long>, unsigned long, long, std::optional<seastar::rpc::rcv_buf> >)#1}, seastar::future<> >(seastar::rpc::server::connection::process()::{lambda()#1}::operator()()::{lambda()#1}::operator()() const::{lambda()#2}::operator()()::{lambda(std::tuple<std::optional<unsigned long>, unsigned long, long, std::optional<seastar::rpc::rcv_buf> >)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<std::tuple<std::optional<unsigned long>, unsigned long, long, std::optional<seastar::rpc::rcv_buf> > >&&)#1}, std::tuple<std::optional<unsigned long>, unsigned long, long, std::optional<seastar::rpc::rcv_buf> > > + 16 
        13: 0x335e4c8 vtable for seastar::continuation<seastar::future<seastar::temporary_buffer<char> >::then_impl<seastar::rpc::connection::read_frame<seastar::rpc::request_frame_with_timeout>(seastar::socket_address, seastar::input_stream<char>&)::{lambda(seastar::temporary_buffer<char>)#1}, seastar::future<std::tuple<std::optional<unsigned long>, unsigned long, long, std::tuple<seastar::rpc::rcv_buf> > > >(seastar::rpc::connection::read_frame<seastar::rpc::request_frame_with_timeout>(seastar::socket_address, seastar::input_stream<char>&)::{lambda(seastar::temporary_buffer<char>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::temporary_buffer<char> >&&)#1}, seastar::temporary_buffer<char> > + 16 
        13: 0x307bde0 vtable for seastar::shared_ptr_count_for<cql3::constants::setter> + 16 
        12: 0x311ecf0 vtable for cql3::restrictions::single_column_primary_key_restrictions<partition_key> + 16 
        12: 0x333ffb8 vtable for seastar::thread_context + 16 
        10: 0x3138c58 vtable for seastar::continuation<seastar::future<>::then_impl<db::commitlog::segment_manager::replenish_reserve()::{lambda()#2}::operator()() const::{lambda()#1}, seastar::future<> >(db::commitlog::segment_manager::replenish_reserve()::{lambda()#2}::operator()() const::{lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
        10: 0x3138b00 vtable for seastar::internal::do_until_state<db::commitlog::segment_manager::replenish_reserve()::{lambda()#1}, db::commitlog::segment_manager::replenish_reserve()::{lambda()#2}> + 16 
         9: 0x314d1f0 vtable for seastar::internal::do_with_state<seastar::async<db::hints::manager::end_point_hints_manager::sender::start()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<db::hints::manager::end_point_hints_manager::sender::start()::{lambda()#1}>::type&&)...)::work, seastar::future<> > + 16 
         9: 0x332fba8 vtable for seastar::continuation<seastar::future<>::then_impl<seastar::sleep_abortable<std::chrono::_V2::steady_clock>(std::chrono::_V2::steady_clock::duration)::{lambda()#1}, seastar::future<> >(seastar::sleep_abortable<std::chrono::_V2::steady_clock>(std::chrono::_V2::steady_clock::duration)::{lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         9: 0x332fc20 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::handle_exception<seastar::sleep_abortable<std::chrono::_V2::steady_clock>(std::chrono::_V2::steady_clock::duration)::{lambda(std::__exception_ptr::exception_ptr)#2}>(seastar::sleep_abortable<std::chrono::_V2::steady_clock>(std::chrono::_V2::steady_clock::duration)::{lambda(std::__exception_ptr::exception_ptr)#2}&&)::{lambda(seastar::sleep_abortable<std::chrono::_V2::steady_clock>(auto:1::duration)::{lambda(std::__exception_ptr::exception_ptr)#2})#1}, seastar::future<> >(seastar::sleep_abortable<std::chrono::_V2::steady_clock>(std::chrono::_V2::steady_clock::duration)::{lambda(std::__exception_ptr::exception_ptr)#2})::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         9: 0x314d9c0 vtable for seastar::continuation<seastar::future<>::then_impl<seastar::async<db::hints::manager::end_point_hints_manager::sender::start()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<db::hints::manager::end_point_hints_manager::sender::start()::{lambda()#1}>::type&&)...)::{lambda(seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<auto:1>::type ()>::type>::type, seastar::thread_attributes, std::decay<auto:1>::type&&)::work&)#1}::operator()(seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, seastar::thread_attributes, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#2}, seastar::future<> >(seastar::thread_attributes)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         9: 0x332fbf8 vtable for seastar::internal::do_until_state<seastar::reactor::wait_for_stop<long, std::ratio<1l, 1000000000l> >(std::chrono::duration<long, std::ratio<1l, 1000000000l> >)::{lambda()#1}, seastar::condition_variable::wait<{lambda()#1}>(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, {lambda()#1}&&)::{lambda()#1}> + 16 
         8: 0x31577f0 vtable for utils::config_file::named_value<unsigned short>::the_value_type + 16 
         8: 0x30beab0 vtable for cql3::restrictions::single_column_primary_key_restrictions<clustering_key_prefix> + 16 
         7: 0x31576d0 vtable for utils::config_file::named_value<double>::the_value_type + 16 
         7: 0x30ec568 vtable for map_type_impl + 16 
         7: 0x30b6788 vtable for seastar::shared_ptr_count_for<cql3::statements::update_statement> + 16 
         7: 0x30b25d8 vtable for seastar::shared_ptr_count_for<cql3::statements::raw::select_statement::parameters> + 16 
         7: 0x30b5d88 vtable for seastar::shared_ptr_count_for<cql3::metadata> + 16 
         7: 0x30f48c8 vtable for boost::detail::sp_counted_impl_p<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (), boost::function<void ()> >, boost::signals2::dummy_mutex> > + 16 
         7: 0x30f4720 vtable for boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (), boost::function<void ()> >, boost::signals2::dummy_mutex> + 16 
         6: 0x30315e0 vtable for seastar::continuation<seastar::future<>::then_impl<logalloc::region_group::start_releaser(seastar::scheduling_group)::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >(logalloc::region_group::start_releaser(seastar::scheduling_group)::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         6: 0x30bef20 vtable for seastar::shared_ptr_count_for<cql3::statements::primary_key_select_statement> + 16 
         6: 0x311e7f0 vtable for seastar::shared_ptr_count_for<cql3::selection::simple_selection> + 16 
         6: 0x30bef00 vtable for seastar::shared_ptr_count_for<std::vector<unsigned long, std::allocator<unsigned long> > > + 16 
         6: 0x3031590 vtable for seastar::internal::repeater<logalloc::region_group::start_releaser(seastar::scheduling_group)::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}> + 16 
         5: 0x3003090 vtable for seastar::internal::when_all_state<seastar::internal::identity_futures_tuple<seastar::future<>, seastar::future<> >, seastar::future<>, seastar::future<> > + 16 
         5: 0x334dfd0 vtable for seastar::net::posix_ap_server_socket_impl + 16 
         4: 0x30037d0 vtable for seastar::continuation<seastar::future<std::tuple<seastar::future<>, seastar::future<> > >::then_impl<seastar::future<std::tuple<seastar::future<>, seastar::future<> > >::discard_result()::{lambda(std::tuple<seastar::future<>, seastar::future<> >&&)#1}, seastar::future<> >(seastar::future<std::tuple<seastar::future<>, seastar::future<> > >::discard_result()::{lambda(std::tuple<seastar::future<>, seastar::future<> >&&)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<std::tuple<seastar::future<>, seastar::future<> > >&&)#1}, std::tuple<seastar::future<>, seastar::future<> > > + 16 
         4: 0x30769f8 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::finally_body<cql_transport::cql_server::connection::process()::{lambda()#2}, true>, seastar::future<> >(seastar::future<>::finally_body<cql_transport::cql_server::connection::process()::{lambda()#2}, true>&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         4: 0x3079fa0 vtable for cql3::constants::value + 16 
         4: 0x3076638 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<cql_transport::cql_server::connection::process()::{lambda()#1}::operator()() const::{lambda(seastar::future<>)#3}, seastar::future<> >(cql_transport::cql_server::connection::process()::{lambda()#1}::operator()() const::{lambda(seastar::future<>)#3}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         4: 0x307d518 vtable for cql3::maps::marker + 16 
         4: 0x3078468 vtable for seastar::shared_ptr_count_for<cql_transport::cql_server::connection> + 16 
         4: 0x30f4910 vtable for boost::detail::sp_counted_impl_p<boost::signals2::slot<void (), boost::function<void ()> > > + 16 
         4: 0x3076548 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<cql_transport::cql_server::do_accepts(int, bool, seastar::socket_address)::{lambda()#1}::operator()() const::{lambda(seastar::future<seastar::accept_result>)#1}::operator()(seastar::accept_result)::{lambda(seastar::future<>)#1}, seastar::future<> >(cql_transport::cql_server::do_accepts(int, bool, seastar::socket_address)::{lambda()#1}::operator()() const::{lambda(seastar::future<seastar::accept_result>)#1}::operator()(seastar::accept_result)::{lambda(seastar::future<>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         4: 0x30769a8 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::finally_body<seastar::with_gate<cql_transport::cql_server::connection::process()::{lambda()#1}>(seastar::gate&, cql_transport::cql_server::connection::process()::{lambda()#1}&&)::{lambda()#1}, false>, seastar::future<> >(seastar::future<>::finally_body<seastar::with_gate<cql_transport::cql_server::connection::process()::{lambda()#1}>(seastar::gate&, cql_transport::cql_server::connection::process()::{lambda()#1}&&)::{lambda()#1}, false>)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         4: 0x3078448 vtable for seastar::shared_ptr_count_for<auth::authenticated_user> + 16 
         4: 0x30764d0 vtable for seastar::internal::do_until_state<cql_transport::cql_server::connection::process()::{lambda()#1}::operator()() const::{lambda()#1}, cql_transport::cql_server::connection::process()::{lambda()#1}::operator()() const::{lambda()#2}> + 16 
         4: 0x30765e8 vtable for seastar::continuation<seastar::future<seastar::temporary_buffer<char> >::then_impl<cql_transport::cql_server::connection::read_frame()::{lambda(seastar::temporary_buffer<char>)#2}, seastar::future<std::optional<cql_transport::cql_binary_frame_v3> > >(cql_transport::cql_server::connection::read_frame()::{lambda(seastar::temporary_buffer<char>)#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::temporary_buffer<char> >&&)#1}, seastar::temporary_buffer<char> > + 16 
         4: 0x3076700 vtable for seastar::continuation<seastar::future<std::optional<cql_transport::cql_binary_frame_v3> >::then_wrapped_impl<cql_transport::cql_server::connection::process_request()::{lambda(seastar::future<std::optional<cql_transport::cql_binary_frame_v3> >&&)#1}, seastar::future<> >(cql_transport::cql_server::connection::process_request()::{lambda(seastar::future<std::optional<cql_transport::cql_binary_frame_v3> >&&)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<std::optional<cql_transport::cql_binary_frame_v3> >&&)#1}, std::optional<cql_transport::cql_binary_frame_v3> > + 16 
         3: 0x3157820 vtable for utils::config_file::named_value<std::unordered_map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<std::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > >::the_value_type + 16 
         3: 0x30ec5c8 vtable for set_type_impl + 16 
         3: 0x3027090 vtable for seastar::continuation<seastar::future<>::then_impl<seastar::sleep<std::chrono::_V2::steady_clock, long, std::ratio<1l, 1l> >(std::chrono::duration<long, std::ratio<1l, 1l> >)::{lambda()#1}, seastar::future<> >(seastar::sleep<std::chrono::_V2::steady_clock, long, std::ratio<1l, 1l> >(std::chrono::duration<long, std::ratio<1l, 1l> >)::{lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         3: 0x3173428 vtable for locator::simple_strategy + 16 
         3: 0x3020708 vtable for seastar::internal::repeater<table::seal_active_memtable(flush_permit&&)::{lambda(auto:1&)#1}::operator()<flush_permit>(flush_permit&) const::{lambda()#1}> + 16 
         3: 0x3020b18 vtable for seastar::continuation<seastar::future<>::then_impl<table::seal_active_memtable(flush_permit&&)::{lambda()#2}, seastar::future<> >(table::seal_active_memtable(flush_permit&&)::{lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         3: 0x307bda0 vtable for seastar::shared_ptr_count_for<cql3::maps::setter> + 16 
         3: 0x30b6a20 vtable for seastar::shared_ptr_count_for<cql3::statements::delete_statement> + 16 
         3: 0x3026788 vtable for seastar::internal::do_with_state<flush_permit, seastar::future<> > + 16 
         3: 0x3020ac8 vtable for seastar::continuation<seastar::future<>::then_impl<table::seal_active_memtable(flush_permit&&)::{lambda(auto:1&)#1}::operator()<flush_permit>(flush_permit&) const::{lambda()#1}::operator()()::{lambda(auto:1)#2}::operator()<seastar::bool_class<seastar::stop_iteration_tag> >(seastar::bool_class<seastar::stop_iteration_tag>)::{lambda()#1}, seastar::future<{lambda(auto:1)#2}::operator()> >(flush_permit&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         2: 0x3003528 vtable for seastar::continuation<seastar::future<seastar::accept_result>::then_wrapped_impl<seastar::httpd::http_server::do_accepts(int)::{lambda(seastar::future<seastar::accept_result>)#1}, seastar::future<> >(seastar::httpd::http_server::do_accepts(int)::{lambda(seastar::future<seastar::accept_result>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::accept_result>&&)#1}, seastar::accept_result> + 16 
         2: 0x3157700 vtable for utils::config_file::named_value<float>::the_value_type + 16 
         2: 0x30b1db8 vtable for reversed_type_impl + 16 
         2: 0x3010978 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::finally_body<dirty_memory_manager::flush_when_needed()::{lambda()#3}, true>, seastar::future<> >(seastar::future<>::finally_body<dirty_memory_manager::flush_when_needed()::{lambda()#3}, true>&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         2: 0x3010448 vtable for seastar::continuation<seastar::future<>::then_impl<dirty_memory_manager::flush_when_needed()::{lambda()#2}::operator()() const::{lambda()#2}, seastar::future<> >(dirty_memory_manager::flush_when_needed()::{lambda()#2}::operator()() const::{lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         2: 0x3003550 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::httpd::http_server::do_accepts(int)::{lambda(auto:1)#2}, seastar::future<> >(seastar::httpd::http_server::do_accepts(int)::{lambda(auto:1)#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         2: 0x306d490 vtable for seastar::shared_ptr_count_for<sstables::time_window_compaction_strategy> + 16 
         2: 0x31734d0 vtable for locator::local_strategy + 16 
         2: 0x31578b0 vtable for utils::config_file::named_value<unsigned long>::the_value_type + 16 
         2: 0x30ec628 vtable for list_type_impl + 16 
         2: 0x311ef68 vtable for cql3::restrictions::statement_restrictions::initial_key_restrictions<partition_key> + 16 
         2: 0x306d230 vtable for sstables::time_window_backlog_tracker + 16 
         2: 0x311f0a8 vtable for cql3::restrictions::statement_restrictions::initial_key_restrictions<clustering_key_prefix> + 16 
         2: 0x30037f8 vtable for seastar::shared_ptr_count_for<seastar::httpd::http_server> + 16 
         2: 0x300ff80 vtable for seastar::internal::do_until_state<dirty_memory_manager::flush_when_needed()::{lambda()#1}, dirty_memory_manager::flush_when_needed()::{lambda()#2}> + 16 
         2: 0x3010060 vtable for seastar::internal::do_until_state<dirty_memory_manager::flush_when_needed()::{lambda()#2}::operator()() const::{lambda()#1}, seastar::condition_variable::wait<{lambda()#2}>({lambda()#2}&&)::{lambda()#1}> + 16 
         2: 0x303f0f0 vtable for combined_mutation_reader + 16 
         1: 0x3003e30 vtable for seastar::shared_ptr_count_for<cql3::cql_config> + 16 
         1: 0x30053d8 vtable for seastar::shared_ptr_count_for<locator::snitch_ptr> + 16 
         1: 0x3259ab8 vtable for seastar::httpd::api_registry_20 + 16 
         1: 0x30143e0 vtable for seastar::concrete_execution_stage<seastar::future<seastar::lw_shared_ptr<query::result> >, table*, seastar::lw_shared_ptr<schema const>, query::read_command const&, query::result_options, std::vector<nonwrapping_range<dht::ring_position>, std::allocator<nonwrapping_range<dht::ring_position> > > const&, tracing::trace_state_ptr, query::result_memory_limiter&, unsigned long, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, query::querier_cache_context> + 16 
         1: 0x3003948 vtable for seastar::shared_ptr_count_for<seastar::abort_source> + 16 
         1: 0x33ab800 vtable for seastar::reactor_backend_epoll + 16 
         1: 0x3230690 vtable for auth::allow_all_authorizer + 16 
         1: 0x3230448 vtable for auth::allow_all_authenticator + 16 
         1: 0x306d2d8 vtable for sstables::null_backlog_tracker + 16 
         1: 0x333bd88 vtable for seastar::reactor::drain_cross_cpu_freelist_pollfn + 16 
         1: 0x3259a90 vtable for seastar::httpd::api_registry + 16 
         1: 0x30ec0b8 vtable for empty_type_impl + 16 
         1: 0x30ec298 vtable for duration_type_impl + 16 
         1: 0x30ec0e8 vtable for counter_type_impl + 16 
         1: 0x30ec268 vtable for decimal_type_impl + 16 
         1: 0x30ec478 vtable for varint_type_impl + 16 
         1: 0x30ec208 vtable for double_type_impl + 16 
         1: 0x30ec238 vtable for float_type_impl + 16 
         1: 0x30ec4a8 vtable for inet_addr_type_impl + 16 
         1: 0x30ec4d8 vtable for uuid_type_impl + 16 
         1: 0x30ec328 vtable for time_type_impl + 16 
         1: 0x30ec2f8 vtable for simple_date_type_impl + 16 
         1: 0x30ec2c8 vtable for timestamp_type_impl + 16 
         1: 0x30ec448 vtable for timeuuid_type_impl + 16 
         1: 0x30ec418 vtable for date_type_impl + 16 
         1: 0x30ec1d8 vtable for boolean_type_impl + 16 
         1: 0x30ec3b8 vtable for utf8_type_impl + 16 
         1: 0x30ec178 vtable for int32_type_impl + 16 
         1: 0x30ec3e8 vtable for bytes_type_impl + 16 
         1: 0x30ec148 vtable for short_type_impl + 16 
         1: 0x30ec388 vtable for ascii_type_impl + 16 
         1: 0x30ec1a8 vtable for long_type_impl + 16 
         1: 0x30ec118 vtable for byte_type_impl + 16 
         1: 0x31578e0 vtable for utils::config_file::named_value<std::unordered_map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::log_level, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<std::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::log_level> > > >::the_value_type + 16 
         1: 0x333bdc8 vtable for seastar::reactor::lowres_timer_pollfn + 16 
         1: 0x3003d68 vtable for seastar::shared_ptr_count_for<gms::feature_service> + 16 
         1: 0x3157880 vtable for utils::config_file::named_value<std::vector<enum_option<db::experimental_features_t>, std::allocator<enum_option<db::experimental_features_t> > > >::the_value_type + 16 
         1: 0x3157760 vtable for utils::config_file::named_value<std::vector<seastar::basic_sstring<char, unsigned int, 15u, true>, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > > >::the_value_type + 16 
         1: 0x3014258 vtable for db::nop_large_data_handler + 16 
         1: 0x31588c0 vtable for db::cql_table_large_data_handler + 16 
         1: 0x3157790 vtable for utils::config_file::named_value<db::seed_provider_type>::the_value_type + 16 
         1: 0x33307f0 vtable for seastar::shared_ptr_count_for<seastar::metrics::impl::impl> + 16 
         1: 0x33ab880 vtable for std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (seastar::reactor::*)(), seastar::reactor*> > > + 16 
         1: 0x334e120 vtable for seastar::net::posix_ap_network_stack + 16 
         1: 0x311b3a0 vtable for cql3::query_processor::migration_subscriber + 16 
         1: 0x333bc88 vtable for seastar::reactor::io_pollfn + 16 
         1: 0x333be08 vtable for seastar::reactor::smp_pollfn + 16 
         1: 0x333bd08 vtable for seastar::reactor::batch_flush_pollfn + 16 
         1: 0x333be88 vtable for seastar::reactor::syscall_pollfn + 16 
         1: 0x333be48 vtable for seastar::reactor::execution_stage_pollfn + 16 
         1: 0x333bd48 vtable for seastar::reactor::aio_batch_submit_pollfn + 16 
         1: 0x333bcc8 vtable for seastar::reactor::signal_pollfn + 16 
         1: 0x333bec8 vtable for seastar::reactor::epoll_pollfn + 16 
         1: 0x32599b0 vtable for seastar::httpd::api_doc + 16 
         1: 0x30bedf8 vtable for seastar::concrete_execution_stage<seastar::future<seastar::shared_ptr<cql_transport::messages::result_message> >, cql3::statements::select_statement*, service::storage_proxy&, service::query_state&, cql3::query_options const&> + 16 
         1: 0x30b5d60 vtable for seastar::concrete_execution_stage<seastar::future<seastar::shared_ptr<cql_transport::messages::result_message> >, cql3::statements::modification_statement*, service::storage_proxy&, service::query_state&, cql3::query_options const&> + 16 
         1: 0x310d778 vtable for seastar::concrete_execution_stage<seastar::future<>, service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, tracing::trace_state_ptr, service_permit, bool> + 16 
         1: 0x31cd870 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::finally_body<seastar::with_gate<seastar::rpc::recv_helper<netw::serializer, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, seastar::future<seastar::rpc::no_wait_type>, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point>(seastar::rpc::signature<seastar::future<seastar::rpc::no_wait_type> (frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>&&, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point)::{lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1}::operator()(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)::{lambda(auto:1)#2}::operator()<seastar::semaphore_units<seastar::semaphore_default_exception_factory, seastar::lowres_clock> >({lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1})::{lambda()#1}>(seastar::gate&, {lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1}&&)::{lambda()#1}, false>, seastar::future<> >(seastar::rpc::recv_helper<netw::serializer, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, seastar::future<seastar::rpc::no_wait_type>, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point>(seastar::rpc::signature<seastar::future<seastar::rpc::no_wait_type> (frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>&&, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point)::{lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1}::operator()(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)::{lambda(auto:1)#2}::operator()<seastar::semaphore_units<seastar::semaphore_default_exception_factory, seastar::lowres_clock> >({lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1})::{lambda()#1})::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x332fc98 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::finally_body<seastar::sleep_abortable<std::chrono::_V2::steady_clock>(std::chrono::_V2::steady_clock::duration, seastar::abort_source&)::{lambda()#1}, false>, seastar::future<> >(seastar::future<>::finally_body<seastar::sleep_abortable<std::chrono::_V2::steady_clock>(std::chrono::_V2::steady_clock::duration, seastar::abort_source&)::{lambda()#1}, false>&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x3342248 vtable for seastar::httpd::directory_handler + 16 
         1: 0x3003ed0 vtable for seastar::shared_ptr_count_for<cql_config_updater> + 16 
         1: 0x334e1b8 vtable for seastar::net::posix_udp_channel + 16 
         1: 0x3070728 vtable for seastar::internal::repeater<compaction_manager::postponed_compactions_reevaluation()::{lambda()#1}> + 16 
         1: 0x323f810 vtable for tracing::tracing + 16 
         1: 0x3332178 vtable for seastar::prometheus::metrics_handler + 16 
         1: 0x31577c0 vtable for utils::config_file::named_value<long>::the_value_type + 16 
         1: 0x3042e48 vtable for seastar::shared_ptr_count_for<lz4_processor> + 16 
         1: 0x3042e88 vtable for seastar::shared_ptr_count_for<deflate_processor> + 16 
         1: 0x3042e68 vtable for seastar::shared_ptr_count_for<snappy_processor> + 16 
         1: 0x3004240 vtable for seastar::shared_ptr_count_for<cql3::query_processor> + 16 
         1: 0x3170420 vtable for gms::gossiper + 16 
         1: 0x31f98c8 vtable for service::storage_service + 16 
         1: 0x3156ec8 vtable for utils::config_file::named_value<seastar::log_level>::the_value_type + 16 
         1: 0x3157850 vtable for utils::config_file::named_value<int>::the_value_type + 16 
         1: 0x30040a0 vtable for seastar::shared_ptr_count_for<database> + 16 
         1: 0x310d260 vtable for service::storage_proxy + 16 
         1: 0x31fdbd0 vtable for service::cache_hitrate_calculator + 16 
         1: 0x3150058 vtable for seastar::internal::do_with_state<seastar::async<db::hints::space_watchdog::start()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<db::hints::space_watchdog::start()::{lambda()#1}>::type&&)...)::work, seastar::future<> > + 16 
         1: 0x335dec0 vtable for seastar::internal::repeater<seastar::rpc::read_rcv_buf(seastar::input_stream<char>&, unsigned int)::{lambda(seastar::temporary_buffer<char>)#1}::operator()(seastar::temporary_buffer<char>)::{lambda(seastar::rpc::rcv_buf&, unsigned int&)#1}::operator()(seastar::rpc::rcv_buf, seastar::rpc::rcv_buf&) const::{lambda()#1}> + 16 
         1: 0x30f4610 vtable for service::migration_manager + 16 
         1: 0x3330830 vtable for seastar::shared_ptr_count_for<std::vector<seastar::metrics::impl::metric_family_metadata, std::allocator<seastar::metrics::impl::metric_family_metadata> > > + 16 
         1: 0x335e0e0 vtable for seastar::continuation<seastar::future<>::then_impl<seastar::rpc::read_rcv_buf(seastar::input_stream<char>&, unsigned int)::{lambda(seastar::temporary_buffer<char>)#1}::operator()(seastar::temporary_buffer<char>)::{lambda(seastar::rpc::rcv_buf&, unsigned int&)#1}::operator()(seastar::rpc::rcv_buf, seastar::rpc::rcv_buf&) const::{lambda()#2}, seastar::future<{lambda(seastar::temporary_buffer<char>)#1}> >(seastar::rpc::read_rcv_buf(seastar::input_stream<char>&, unsigned int)::{lambda(seastar::temporary_buffer<char>)#1}::operator()(seastar::temporary_buffer<char>)::{lambda(seastar::rpc::rcv_buf&, unsigned int&)#1}::operator()(seastar::rpc::rcv_buf, seastar::rpc::rcv_buf&) const::{lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x3173590 vtable for locator::network_topology_strategy + 16 
         1: 0x3330700 vtable for seastar::metrics::metric_groups + 16 
         1: 0x31272c8 vtable for seastar::shared_ptr_count_for<db::system_keyspace::local_cache> + 16 
         1: 0x30bee40 vtable for seastar::shared_ptr_count_for<cql3::column_identifier::raw> + 16 
         1: 0x320c058 vtable for streaming::stream_manager + 16 
         1: 0x3004380 vtable for seastar::shared_ptr_count_for<db::view::view_update_generator> + 16 
         1: 0x30042e0 vtable for seastar::shared_ptr_count_for<db::batchlog_manager> + 16 
         1: 0x323a7f8 vtable for auth::auth_migration_listener + 16 
         1: 0x3207ac0 vtable for seastar::continuation<seastar::future<>::then_impl<streaming::send_mutation_fragments(seastar::lw_shared_ptr<streaming::send_info>)::{lambda(mutation_fragment*)#1}::operator()(mutation_fragment*) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}::operator()(frozen_mutation_fragment, seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>)::{lambda()#2}::operator()()::{lambda(frozen_mutation_fragment&)#1}::operator()({lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}) const::{lambda()#2}, seastar::future<> >(streaming::send_mutation_fragments(seastar::lw_shared_ptr<streaming::send_info>)::{lambda(mutation_fragment*)#1}::operator()(mutation_fragment*) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}::operator()(frozen_mutation_fragment, seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>)::{lambda()#2}::operator()()::{lambda(frozen_mutation_fragment&)#1}::operator()({lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}) const::{lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x3207c78 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::handle_exception<streaming::send_mutation_fragments(seastar::lw_shared_ptr<streaming::send_info>)::{lambda(mutation_fragment*)#1}::operator()(mutation_fragment*) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}::operator()(frozen_mutation_fragment, seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>)::{lambda()#2}::operator()()::{lambda(frozen_mutation_fragment&)#1}::operator()({lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}) const::{lambda(std::__exception_ptr::exception_ptr)#3}>(streaming::send_mutation_fragments(seastar::lw_shared_ptr<streaming::send_info>)::{lambda(mutation_fragment*)#1}::operator()(mutation_fragment*) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}::operator()(frozen_mutation_fragment, seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>)::{lambda()#2}::operator()()::{lambda(frozen_mutation_fragment&)#1}::operator()({lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}) const::{lambda(std::__exception_ptr::exception_ptr)#3}&&)::{lambda({lambda(frozen_mutation_fragment&)#1})#1}, seastar::future<> >({lambda(frozen_mutation_fragment&)#1})::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x3207ca0 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::finally_body<streaming::send_mutation_fragments(seastar::lw_shared_ptr<streaming::send_info>)::{lambda(mutation_fragment*)#1}::operator()(mutation_fragment*) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}::operator()(frozen_mutation_fragment, seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>)::{lambda()#2}::operator()()::{lambda(frozen_mutation_fragment&)#1}::operator()({lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}) const::{lambda()#4}, true>, seastar::future<> >(seastar::future<>::finally_body<streaming::send_mutation_fragments(seastar::lw_shared_ptr<streaming::send_info>)::{lambda(mutation_fragment*)#1}::operator()(mutation_fragment*) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}::operator()(frozen_mutation_fragment, seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>)::{lambda()#2}::operator()()::{lambda(frozen_mutation_fragment&)#1}::operator()({lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}) const::{lambda()#4}, true>&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x3111b50 vtable for cql3::constants::null_literal::null_value + 16 
         1: 0x335ad30 vtable for seastar::continuation<seastar::future<>::then_impl<seastar::keep_doing<seastar::rpc::server::accept()::{lambda()#1}>(seastar::rpc::server::accept()::{lambda()#1})::{lambda()#1}::operator()()::{lambda()#1}, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >(seastar::rpc::server::accept()::{lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x335ab50 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::rpc::server::accept()::{lambda(seastar::future<>&&)#2}, seastar::future<> >(seastar::rpc::server::accept()::{lambda(seastar::future<>&&)#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x31501c0 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::handle_exception_type<db::hints::space_watchdog::start()::{lambda(seastar::sleep_aborted const&)#2}>(db::hints::space_watchdog::start()::{lambda(seastar::sleep_aborted const&)#2}&&)::{lambda(auto:1)#1}, seastar::future<> >(seastar::future<>::handle_exception_type<db::hints::space_watchdog::start()::{lambda(seastar::sleep_aborted const&)#2}>(db::hints::space_watchdog::start()::{lambda(seastar::sleep_aborted const&)#2}&&)::{lambda(auto:1)#1})::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x323e040 vtable for auth::standard_role_manager + 16 
         1: 0x31c9a20 vtable for netw::messaging_service::rpc_protocol_wrapper + 16 
         1: 0x3185080 vtable for seastar::continuation<seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock> >::then_impl<seastar::rpc::sink_impl<netw::serializer, frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>::operator()(frozen_mutation_fragment const&, streaming::stream_mutation_fragments_cmd const&)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)#1}, seastar::future<> >(seastar::rpc::sink_impl<netw::serializer, frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>::operator()(frozen_mutation_fragment const&, streaming::stream_mutation_fragments_cmd const&)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock> >&&)#1}, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock> > + 16 
         1: 0x307bcc0 vtable for seastar::shared_ptr_count_for<cql3::maps::putter> + 16 
         1: 0x3111db0 vtable for seastar::shared_ptr_count_for<cql3::constants::null_literal> + 16 
         1: 0x3241188 vtable for tracing::trace_keyspace_helper + 16 
         1: 0x32079a8 vtable for seastar::internal::repeater<streaming::send_mutation_fragments(seastar::lw_shared_ptr<streaming::send_info>)::{lambda(mutation_fragment*)#1}::operator()(mutation_fragment*) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}::operator()(frozen_mutation_fragment, seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>)::{lambda()#2}::operator()()::{lambda(frozen_mutation_fragment&)#1}::operator()({lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}) const::{lambda()#1}> + 16 
         1: 0x31064d8 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::handle_exception<service::storage_proxy::init_messaging_service()::{lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator()<frozen_mutation, service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}, service::storage_proxy::init_messaging_service()::seastar::rpc::optional<std::optional<tracing::trace_info> >::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2}>(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}&&, service::storage_proxy::init_messaging_service()::seastar::rpc::optional<std::optional<tracing::trace_info> >::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2}&&) const::{lambda(auto:1 const&, seastar::shared_ptr, unsigned long&)#1}::operator()<{lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator()>(service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}, seastar::shared_ptr, {lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2})::{lambda(std::__exception_ptr::exception_ptr)#3}>(service::storage_proxy::init_messaging_service()::{lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator()<frozen_mutation, service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}, service::storage_proxy::init_messaging_service()::seastar::rpc::optional<std::optional<tracing::trace_info> >::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2}>(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}&&, service::storage_proxy::init_messaging_service()::seastar::rpc::optional<std::optional<tracing::trace_info> >::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2}&&) const::{lambda(auto:1 const&, seastar::shared_ptr, unsigned long&)#1}::operator()<{lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator()>(service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}, seastar::shared_ptr, {lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2})::{lambda(std::__exception_ptr::exception_ptr)#3}&&)::{lambda({lambda(auto:1 const&, seastar::shared_ptr, unsigned long&)#1})#1}, seastar::future<> >({lambda(auto:1 const&, seastar::shared_ptr, unsigned long&)#1})::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x304e290 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::handle_exception<do_io_check<sstables::fsync_directory(std::function<void (std::__exception_ptr::exception_ptr)> const&, seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda()#1}&, , void>(std::function<void (std::__exception_ptr::exception_ptr)> const&, sstables::fsync_directory(std::function<void (std::__exception_ptr::exception_ptr)> const&, seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda()#1}&)::{lambda(sstables::fsync_directory(std::function<void (std::__exception_ptr::exception_ptr)> const&, seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda()#1}&)#1}>(do_io_check<sstables::fsync_directory(std::function<void (std::__exception_ptr::exception_ptr)> const&, seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda()#1}&, , void>(std::function<void (std::__exception_ptr::exception_ptr)> const&, sstables::fsync_directory(std::function<void (std::__exception_ptr::exception_ptr)> const&, seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda()#1}&)::{lambda(sstables::fsync_directory(std::function<void (std::__exception_ptr::exception_ptr)> const&, seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda()#1}&)#1})::{lambda(auto:1)#1}, seastar::future<> >(seastar::future<>::handle_exception<do_io_check<sstables::fsync_directory(std::function<void (std::__exception_ptr::exception_ptr)> const&, seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda()#1}&, , void>(std::function<void (std::__exception_ptr::exception_ptr)> const&, sstables::fsync_directory(std::function<void (std::__exception_ptr::exception_ptr)> const&, seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda()#1}&)::{lambda(sstables::fsync_directory(std::function<void (std::__exception_ptr::exception_ptr)> const&, seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda()#1}&)#1}>(do_io_check<sstables::fsync_directory(std::function<void (std::__exception_ptr::exception_ptr)> const&, seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda()#1}&, , void>(std::function<void (std::__exception_ptr::exception_ptr)> const&, sstables::fsync_directory(std::function<void (std::__exception_ptr::exception_ptr)> const&, seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda()#1}&)::{lambda(sstables::fsync_directory(std::function<void (std::__exception_ptr::exception_ptr)> const&, seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda()#1}&)#1})::{lambda(auto:1)#1})::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x3339680 vtable for seastar::continuation<seastar::future<seastar::syscall_result<int> >::then_impl<seastar::reactor::fdatasync(int)::{lambda(seastar::syscall_result<int>)#4}, seastar::future<> >(seastar::reactor::fdatasync(int)::{lambda(seastar::syscall_result<int>)#4}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::syscall_result<int> >&&)#1}, seastar::syscall_result<int> > + 16 
         1: 0x3207a48 vtable for seastar::continuation<seastar::future<std::optional<std::tuple<int> > >::then_impl<streaming::send_mutation_fragments(seastar::lw_shared_ptr<streaming::send_info>)::{lambda(mutation_fragment*)#1}::operator()(mutation_fragment*) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}::operator()(frozen_mutation_fragment, seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>)::{lambda()#1}::operator()()::{lambda()#1}::operator()()::{lambda(std::optional<std::tuple<int> >)#1}, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >(streaming::send_mutation_fragments(seastar::lw_shared_ptr<streaming::send_info>)::{lambda(mutation_fragment*)#1}::operator()(mutation_fragment*) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}::operator()(frozen_mutation_fragment, seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>)::{lambda()#1}::operator()()::{lambda()#1}::operator()()::{lambda(std::optional<std::tuple<int> >)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<std::optional<std::tuple<int> > >&&)#1}, std::optional<std::tuple<int> > > + 16 
         1: 0x31c9a80 vtable for netw::messaging_service + 16 
         1: 0x306d410 vtable for seastar::shared_ptr_count_for<sstables::null_compaction_strategy> + 16 
         1: 0x304e268 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::handle_exception<sstables::sstable::~sstable()::{lambda(std::__exception_ptr::exception_ptr)#3}>(sstables::sstable::~sstable()::{lambda(std::__exception_ptr::exception_ptr)#3}&&)::{lambda(auto:1)#1}, seastar::future<> >(seastar::future<>::handle_exception<sstables::sstable::~sstable()::{lambda(std::__exception_ptr::exception_ptr)#3}>(sstables::sstable::~sstable()::{lambda(std::__exception_ptr::exception_ptr)#3}&&)::{lambda(auto:1)#1})::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x3176840 vtable for locator::gossiping_property_file_snitch + 16 
         1: 0x335a758 vtable for seastar::internal::repeater<seastar::keep_doing<seastar::rpc::server::accept()::{lambda()#1}>(seastar::rpc::server::accept()::{lambda()#1})::{lambda()#1}> + 16 
         1: 0x30d53c0 vtable for boost::detail::sp_counted_impl_p<cassandra::CassandraAsyncProcessorFactory> + 16 
         1: 0x30764a8 vtable for seastar::internal::repeater<cql_transport::cql_server::do_accepts(int, bool, seastar::socket_address)::{lambda()#1}> + 16 
         1: 0x30d5378 vtable for boost::detail::sp_counted_impl_p<cassandra::CassandraCobSvIfFactory> + 16 
         1: 0x30d5298 vtable for apache::thrift::protocol::TBinaryProtocolFactoryT<apache::thrift::transport::TMemoryBuffer, apache::thrift::protocol::TNetworkBigEndian> + 16 
         1: 0x3323e10 vtable for cassandra::CassandraAsyncProcessorFactory + 16 
         1: 0x31f9cb0 vtable for seastar::shared_ptr_count_for<db::system_distributed_keyspace> + 16 
         1: 0x311c118 vtable for seastar::shared_ptr_count_for<cql3::restrictions::single_column_restriction::slice> + 16 
         1: 0x3004420 vtable for seastar::shared_ptr_count_for<memory_threshold_guard> + 16 
         1: 0x3163030 vtable for seastar::continuation<seastar::future<>::then_impl<seastar::async<db::view::view_update_generator::start()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<db::view::view_update_generator::start()::{lambda()#1}>::type&&)...)::{lambda(seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<auto:1>::type ()>::type>::type, seastar::thread_attributes, std::decay<auto:1>::type&&)::work&)#1}::operator()(seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, seastar::thread_attributes, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#2}, seastar::future<> >(seastar::thread_attributes)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x3004720 vtable for seastar::shared_ptr_count_for<db::view::view_builder> + 16 
         1: 0x3076980 vtable for seastar::continuation<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_impl<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::handle_exception<cql_transport::cql_server::do_accepts(int, bool, seastar::socket_address)::{lambda()#1}::operator()() const::{lambda(auto:1)#2}>({lambda()#1}&&)::{lambda({lambda(auto:1)#2})#1}, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >({lambda(auto:1)#2})::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}, seastar::bool_class<seastar::stop_iteration_tag> > + 16 
         1: 0x335a7f8 vtable for seastar::continuation<seastar::future<std::optional<seastar::rpc::rcv_buf> >::then_impl<seastar::rpc::connection::handle_stream_frame()::{lambda(std::optional<seastar::rpc::rcv_buf>)#1}, seastar::future<> >(seastar::rpc::connection::handle_stream_frame()::{lambda(std::optional<seastar::rpc::rcv_buf>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<std::optional<seastar::rpc::rcv_buf> >&&)#1}, std::optional<seastar::rpc::rcv_buf> > + 16 
         1: 0x3150260 vtable for seastar::continuation<seastar::future<>::then_impl<seastar::async<db::hints::space_watchdog::start()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<db::hints::space_watchdog::start()::{lambda()#1}>::type&&)...)::{lambda(seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<auto:1>::type ()>::type>::type, seastar::thread_attributes, std::decay<auto:1>::type&&)::work&)#1}::operator()(seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, seastar::thread_attributes, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#2}, seastar::future<> >(seastar::thread_attributes)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x3078610 vtable for seastar::continuation<seastar::future<seastar::temporary_buffer<char> >::then_impl<seastar::input_stream<char>::read_up_to(unsigned long)::{lambda(seastar::temporary_buffer<char>)#1}, seastar::future<seastar::temporary_buffer<char> > >(seastar::input_stream<char>::read_up_to(unsigned long)::{lambda(seastar::temporary_buffer<char>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::temporary_buffer<char> >&&)#1}, seastar::temporary_buffer<char> > + 16 
         1: 0x335ab28 vtable for seastar::continuation<seastar::future<seastar::accept_result>::then_impl<seastar::rpc::server::accept()::{lambda()#1}::operator()()::{lambda(seastar::accept_result)#1}, seastar::future<> >(seastar::rpc::server::accept()::{lambda()#1}::operator()()::{lambda(seastar::accept_result)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::accept_result>&&)#1}, seastar::accept_result> + 16 
         1: 0x31fdc10 vtable for service::view_update_backlog_broker + 16 
         1: 0x3162fe0 vtable for seastar::internal::do_with_state<seastar::async<db::view::view_update_generator::start()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<db::view::view_update_generator::start()::{lambda()#1}>::type&&)...)::work, seastar::future<> > + 16 
         1: 0x30d4098 vtable for seastar::continuation<seastar::future<seastar::accept_result>::then_impl<thrift_server::do_accepts(int, bool)::{lambda()#1}::operator()() const::{lambda(seastar::accept_result)#1}, seastar::future<> >(thrift_server::do_accepts(int, bool)::{lambda()#1}::operator()() const::{lambda(seastar::accept_result)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::accept_result>&&)#1}, seastar::accept_result> + 16 
         1: 0x3072300 vtable for cql_transport::cql_server::event_notifier + 16 
         1: 0x30142f0 vtable for seastar::internal::when_all_state<seastar::internal::extract_values_from_futures_tuple<seastar::future<>, seastar::future<> >, seastar::future<>, seastar::future<> > + 16 
         1: 0x30d4160 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::finally_body<seastar::with_gate<thrift_server::do_accepts(int, bool)::{lambda()#1}>(seastar::gate&, thrift_server::do_accepts(int, bool)::{lambda()#1}&&)::{lambda()#1}, false>, seastar::future<> >(seastar::future<>::finally_body<seastar::with_gate<thrift_server::do_accepts(int, bool)::{lambda()#1}>(seastar::gate&, thrift_server::do_accepts(int, bool)::{lambda()#1}&&)::{lambda()#1}, false>)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x30d4138 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::handle_exception<thrift_server::do_accepts(int, bool)::{lambda()#1}::operator()() const::{lambda(auto:1)#2}>({lambda()#1}&&)::{lambda({lambda(auto:1)#2})#1}, seastar::future<> >({lambda(auto:1)#2})::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x31f9ff0 vtable for seastar::shared_ptr_count_for<thrift_server> + 16 
         1: 0x30d20b0 vtable for handler_factory + 16 
         1: 0x31f9d50 vtable for seastar::shared_ptr_count_for<auth::service> + 16 
         1: 0x31fa140 vtable for seastar::shared_ptr_count_for<cql_transport::cql_server> + 16 
         1: 0x3076570 vtable for seastar::continuation<seastar::future<seastar::accept_result>::then_wrapped_impl<cql_transport::cql_server::do_accepts(int, bool, seastar::socket_address)::{lambda()#1}::operator()() const::{lambda(seastar::future<seastar::accept_result>)#1}, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >(cql_transport::cql_server::do_accepts(int, bool, seastar::socket_address)::{lambda()#1}::operator()() const::{lambda(seastar::future<seastar::accept_result>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::accept_result>&&)#1}, seastar::accept_result> + 16 
         1: 0x335e478 vtable for seastar::continuation<seastar::future<seastar::temporary_buffer<char> >::then_impl<seastar::rpc::connection::read_frame<seastar::rpc::stream_frame>(seastar::socket_address, seastar::input_stream<char>&)::{lambda(seastar::temporary_buffer<char>)#1}, seastar::future<std::optional<seastar::rpc::rcv_buf> > >(seastar::rpc::connection::read_frame<seastar::rpc::stream_frame>(seastar::socket_address, seastar::input_stream<char>&)::{lambda(seastar::temporary_buffer<char>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::temporary_buffer<char> >&&)#1}, seastar::temporary_buffer<char> > + 16 
         1: 0x3207980 vtable for seastar::internal::repeater<streaming::send_mutation_fragments(seastar::lw_shared_ptr<streaming::send_info>)::{lambda(mutation_fragment*)#1}::operator()(mutation_fragment*) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}::operator()(frozen_mutation_fragment, seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>)::{lambda()#1}::operator()()::{lambda()#1}> + 16 
         1: 0x335e0b8 vtable for seastar::continuation<seastar::future<seastar::temporary_buffer<char> >::then_impl<seastar::rpc::read_rcv_buf(seastar::input_stream<char>&, unsigned int)::{lambda(seastar::temporary_buffer<char>)#1}::operator()(seastar::temporary_buffer<char>)::{lambda(seastar::rpc::rcv_buf&, unsigned int&)#1}::operator()(seastar::rpc::rcv_buf, seastar::rpc::rcv_buf&) const::{lambda()#1}::operator()() const::{lambda(seastar::temporary_buffer<char>)#1}, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >(seastar::rpc::read_rcv_buf(seastar::input_stream<char>&, unsigned int)::{lambda(seastar::temporary_buffer<char>)#1}::operator()(seastar::temporary_buffer<char>)::{lambda(seastar::rpc::rcv_buf&, unsigned int&)#1}::operator()(seastar::rpc::rcv_buf, seastar::rpc::rcv_buf&) const::{lambda()#1}::operator()() const::{lambda(seastar::temporary_buffer<char>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::temporary_buffer<char> >&&)#1}, seastar::temporary_buffer<char> > + 16 
         1: 0x3105fc0 vtable for seastar::continuation<seastar::future<std::tuple<seastar::future<>, seastar::future<> > >::then_wrapped_impl<service::storage_proxy::init_messaging_service()::{lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator()<frozen_mutation, service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}, service::storage_proxy::init_messaging_service()::seastar::rpc::optional<std::optional<tracing::trace_info> >::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2}>(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}&&, service::storage_proxy::init_messaging_service()::seastar::rpc::optional<std::optional<tracing::trace_info> >::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2}&&) const::{lambda(auto:1 const&, seastar::shared_ptr, unsigned long&)#1}::operator()<{lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator()>(service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}, seastar::shared_ptr, {lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2})::{lambda(seastar::future<std::tuple<seastar::future<>, seastar::future<> > >&&)#5}, seastar::future<seastar::rpc::no_wait_type> >(service::storage_proxy::init_messaging_service()::{lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator()<frozen_mutation, service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}, service::storage_proxy::init_messaging_service()::seastar::rpc::optional<std::optional<tracing::trace_info> >::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2}>(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}&&, service::storage_proxy::init_messaging_service()::seastar::rpc::optional<std::optional<tracing::trace_info> >::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2}&&) const::{lambda(auto:1 const&, seastar::shared_ptr, unsigned long&)#1}::operator()<{lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator()>(service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}, seastar::shared_ptr, {lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2})::{lambda(seastar::future<std::tuple<seastar::future<>, seastar::future<> > >&&)#5}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<std::tuple<seastar::future<>, seastar::future<> > >&&)#1}, std::tuple<seastar::future<>, seastar::future<> > > + 16 
         1: 0x303f138 vtable for restricting_mutation_reader + 16 
         1: 0x335e158 vtable for seastar::continuation<seastar::future<>::then_impl<seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)1>()::{lambda()#2}::operator()() const::{lambda()#2}::operator()() const::{lambda()#1}, seastar::future<> >(seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)1>()::{lambda()#2}::operator()() const::{lambda()#2}::operator()() const::{lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x335e4a0 vtable for seastar::continuation<seastar::future<seastar::rpc::rcv_buf>::then_impl<seastar::rpc::connection::read_frame<seastar::rpc::request_frame_with_timeout>(seastar::socket_address, seastar::input_stream<char>&)::{lambda(seastar::temporary_buffer<char>)#1}::operator()(seastar::temporary_buffer<char>) const::{lambda(seastar::rpc::rcv_buf)#1}, seastar::future<std::tuple<std::optional<unsigned long>, unsigned long, long, seastar::rpc::connection::read_frame<seastar::rpc::request_frame_with_timeout>(seastar::socket_address, seastar::input_stream<char>&)::{lambda(seastar::temporary_buffer<char>)#1}::operator()(seastar::temporary_buffer<char>) const::{lambda(seastar::rpc::rcv_buf)#1}<seastar::rpc::rcv_buf> > > >(seastar::rpc::connection::read_frame<seastar::rpc::request_frame_with_timeout>(seastar::socket_address, seastar::input_stream<char>&)::{lambda(seastar::temporary_buffer<char>)#1}::operator()(seastar::temporary_buffer<char>) const::{lambda(seastar::rpc::rcv_buf)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::rpc::rcv_buf>&&)#1}, seastar::rpc::rcv_buf> + 16 
         1: 0x30709d0 vtable for seastar::continuation<seastar::future<>::then_impl<compaction_manager::postponed_compactions_reevaluation()::{lambda()#1}::operator()() const::{lambda()#1}, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >(compaction_manager::postponed_compactions_reevaluation()::{lambda()#1}::operator()() const::{lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x3207a70 vtable for seastar::continuation<seastar::future<>::then_impl<streaming::send_mutation_fragments(seastar::lw_shared_ptr<streaming::send_info>)::{lambda(mutation_fragment*)#1}::operator()(mutation_fragment*) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}::operator()(frozen_mutation_fragment, seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>)::{lambda()#2}::operator()()::{lambda(frozen_mutation_fragment&)#1}::operator()({lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}) const::{lambda()#1}::operator()()::{lambda(seastar::optimized_optional<mutation_fragment>)#1}::operator()(frozen_mutation_fragment&)::{lambda()#1}, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >(streaming::send_mutation_fragments(seastar::lw_shared_ptr<streaming::send_info>)::{lambda(mutation_fragment*)#1}::operator()(mutation_fragment*) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}::operator()(frozen_mutation_fragment, seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>)::{lambda()#2}::operator()()::{lambda(frozen_mutation_fragment&)#1}::operator()({lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}) const::{lambda()#1}::operator()()::{lambda(seastar::optimized_optional<mutation_fragment>)#1}::operator()(frozen_mutation_fragment&)::{lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x335de98 vtable for seastar::internal::do_with_state<std::tuple<seastar::rpc::rcv_buf, unsigned int>, seastar::future<seastar::rpc::rcv_buf> > + 16 
         1: 0x2ff06c8 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::smp_message_queue::async_work_item<seastar::sharded<database>::invoke_on_all(seastar::smp_service_group, std::function<seastar::future<> (database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>::process()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, seastar::future<> >(seastar::smp_message_queue::async_work_item<seastar::sharded<database>::invoke_on_all(seastar::smp_service_group, std::function<seastar::future<> (database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x3185850 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::finally_body<seastar::smp::submit_to<seastar::rpc::source_impl<netw::serializer, int>::operator()()::{lambda()#2}>(unsigned int, seastar::smp_service_group, std::result_of&&)::{lambda()#1}, false>, seastar::future<> >(seastar::smp_service_group)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x334cc28 vtable for seastar::continuation<seastar::future<>::then_impl<seastar::net::posix_data_sink_impl::put(seastar::net::packet)::{lambda()#1}, seastar::future<> >(seastar::net::posix_data_sink_impl::put(seastar::net::packet)::{lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x31cbd70 vtable for seastar::shared_ptr_count_for<seastar::rpc::sink_impl<netw::serializer, frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd> > + 16 
         1: 0x3041930 vtable for flat_multi_range_mutation_reader<make_flat_multi_range_reader(seastar::lw_shared_ptr<schema const>, mutation_source, std::vector<nonwrapping_range<dht::ring_position>, std::allocator<nonwrapping_range<dht::ring_position> > > const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<flat_mutation_reader::partition_range_forwarding_tag>)::adapter> + 16 
         1: 0x31cc628 vtable for seastar::continuation<seastar::future<seastar::rpc::no_wait_type>::then_wrapped_impl<seastar::rpc::recv_helper<netw::serializer, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, seastar::future<seastar::rpc::no_wait_type>, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point>(seastar::rpc::signature<seastar::future<seastar::rpc::no_wait_type> (frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>, std::function<seastar::future<seastar::rpc::no_wait_type> (seastar::rpc::client_info const&, seastar::rpc::opt_time_point, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)>&&, seastar::rpc::do_want_client_info, seastar::rpc::do_want_time_point)::{lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1}::operator()(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)::{lambda(auto:1)#2}::operator()<seastar::semaphore_units<seastar::semaphore_default_exception_factory, seastar::lowres_clock> >({lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1})::{lambda()#1}::operator()()::{lambda(seastar::future<seastar::rpc::no_wait_type>)#1}, seastar::future<> >({lambda(seastar::shared_ptr<seastar::rpc::server::connection>, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, long, seastar::rpc::rcv_buf)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::rpc::no_wait_type>&&)#1}, seastar::rpc::no_wait_type> + 16 
         1: 0x30265c0 vtable for incremental_reader_selector + 16 
         1: 0x332c798 vtable for seastar::syscall_work_queue::work_item_returning<seastar::syscall_result<int> > + 16 
         1: 0x31059b0 vtable for seastar::smp_message_queue::async_work_item<seastar::sharded<database>::invoke_on<service::storage_proxy::mutate_locally(seastar::lw_shared_ptr<schema const> const&, frozen_mutation const&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(database&)#1}, seastar::future<> >(unsigned int, seastar::smp_service_group, service::storage_proxy::mutate_locally(seastar::lw_shared_ptr<schema const> const&, frozen_mutation const&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(database&)#1}&&)::{lambda()#1}> + 16 
         1: 0x3105f48 vtable for seastar::continuation<seastar::future<>::then_impl<service::storage_proxy::init_messaging_service()::{lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator()<frozen_mutation, service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}, service::storage_proxy::init_messaging_service()::seastar::rpc::optional<std::optional<tracing::trace_info> >::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2}>(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}&&, service::storage_proxy::init_messaging_service()::seastar::rpc::optional<std::optional<tracing::trace_info> >::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2}&&) const::{lambda(auto:1 const&, seastar::shared_ptr, unsigned long&)#1}::operator()<{lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator()>(service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}, seastar::shared_ptr, {lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2})::{lambda()#2}, seastar::future<> >(service::storage_proxy::init_messaging_service()::{lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator()<frozen_mutation, service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}, service::storage_proxy::init_messaging_service()::seastar::rpc::optional<std::optional<tracing::trace_info> >::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2}>(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, frozen_mutation, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}&&, service::storage_proxy::init_messaging_service()::seastar::rpc::optional<std::optional<tracing::trace_info> >::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2}&&) const::{lambda(auto:1 const&, seastar::shared_ptr, unsigned long&)#1}::operator()<{lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator()>(service::storage_proxy::init_messaging_service()::{lambda(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional<std::optional<tracing::trace_info> >)#3}::operator()(seastar::rpc::client_info const, seastar::rpc::opt_time_point, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator(), std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, seastar::rpc::optional) const::{lambda(seastar::shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const&, std::chrono::time_point<seastar::lowres_clock, {lambda(netw::msg_addr, seastar::rpc::opt_time_point, utils::UUID, auto:1, std::vector<gms::inet_address, std::allocator<gms::inet_address> >, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>, auto:2&&, auto:3&&)#2}::operator() const::duration<long, std::ratio<1l, 1000l> > >)#1}, seastar::shared_ptr, {lambda(netw::msg_addr, std::ratio<1l, 1000l>, seastar::lw_shared_ptr<schema const>, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info>)#2})::{lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x3014880 vtable for seastar::continuation<seastar::future<>::then_impl<seastar::get_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>&, unsigned long)::{lambda()#1}, seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock> > >(seastar::get_units<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::chrono::_V2::steady_clock>&, unsigned long)::{lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x3060e80 vtable for seastar::internal::do_with_state<seastar::file, seastar::future<> > + 16 
         1: 0x31ce7c0 vtable for seastar::shared_ptr_count_for<seastar::rpc::source_impl<netw::serializer, int> > + 16 
         1: 0x335a820 vtable for seastar::continuation<seastar::future<>::then_impl<seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&)::{lambda()#1}, seastar::future<> >(seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&)::{lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x31cbd50 vtable for seastar::shared_ptr_count_for<seastar::rpc::client> + 16 
         1: 0x303f0c0 vtable for list_reader_selector + 16 
         1: 0x3184fa8 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::rpc::source_impl<netw::serializer, int>::operator()()::{lambda()#2}::operator()() const::{lambda(seastar::future<>&&)#1}, seastar::future<> >(seastar::rpc::source_impl<netw::serializer, int>::operator()()::{lambda()#2}::operator()() const::{lambda(seastar::future<>&&)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x3185000 vtable for seastar::continuation<seastar::future<>::then_impl<seastar::rpc::source_impl<netw::serializer, int>::operator()()::{lambda()#3}, seastar::future<std::optional<std::tuple<int> > > >(seastar::rpc::source_impl<netw::serializer, int>::operator()()::{lambda()#3}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x3207b10 vtable for seastar::continuation<seastar::future<>::then_impl<streaming::send_mutation_fragments(seastar::lw_shared_ptr<streaming::send_info>)::{lambda(mutation_fragment*)#1}::operator()(mutation_fragment*) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}::operator()(frozen_mutation_fragment, seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>)::{lambda()#3}, seastar::future<> >(streaming::send_mutation_fragments(seastar::lw_shared_ptr<streaming::send_info>)::{lambda(mutation_fragment*)#1}::operator()(mutation_fragment*) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, {lambda(unsigned long)#1}::source<int>)#1}::operator()(frozen_mutation_fragment, seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>)::{lambda()#3}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x3030338 vtable for scanning_reader + 16 
         1: 0x32085f0 vtable for seastar::internal::do_with_state<seastar::rpc::sink<frozen_mutation_fragment, streaming::stream_mutation_fragments_cmd>, seastar::future<> > + 16 
         1: 0x304de78 vtable for seastar::continuation<seastar::future<seastar::file>::then_impl<sstables::fsync_directory(std::function<void (std::__exception_ptr::exception_ptr)> const&, seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda()#1}::operator()() const::{lambda(seastar::file)#1}, seastar::future<> >(sstables::fsync_directory(std::function<void (std::__exception_ptr::exception_ptr)> const&, seastar::basic_sstring<char, unsigned int, 15u, true>)::{lambda()#1}::operator()() const::{lambda(seastar::file)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<seastar::file>&&)#1}, seastar::file> + 16 
         1: 0x304e388 vtable for seastar::continuation<seastar::future<>::then_impl<seastar::async<sstables::remove_by_toc_name(seastar::basic_sstring<char, unsigned int, 15u, true>, std::function<void (std::__exception_ptr::exception_ptr)> const&)::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<sstables::remove_by_toc_name(seastar::basic_sstring<char, unsigned int, 15u, true>, std::function<void (std::__exception_ptr::exception_ptr)> const&)::{lambda()#1}>::type&&)...)::{lambda(seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<auto:1>::type ()>::type>::type, seastar::thread_attributes, std::decay<auto:1>::type&&)::work&)#1}::operator()(seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, seastar::thread_attributes, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#2}, seastar::future<> >(seastar::thread_attributes)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x33392e8 vtable for seastar::continuation<seastar::future<unsigned long>::then_impl<seastar::pollable_fd::write_all(seastar::net::packet&)::{lambda(unsigned long)#1}, seastar::future<> >(seastar::pollable_fd::write_all(seastar::net::packet&)::{lambda(unsigned long)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<unsigned long>&&)#1}, unsigned long> + 16 
         1: 0x333cb68 vtable for seastar::continuation<seastar::future<>::then_impl<seastar::pollable_fd::write_some(seastar::net::packet&)::{lambda()#1}, seastar::future<unsigned long> >(seastar::pollable_fd::write_some(seastar::net::packet&)::{lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x304def0 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<sstables::sstable::unlink()::{lambda(seastar::future<>)#1}, seastar::future<> >(sstables::sstable::unlink()::{lambda(seastar::future<>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
         1: 0x310d5c0 vtable for seastar::internal::do_with_state<std::tuple<frozen_mutation, seastar::shared_ptr<service::storage_proxy>, unsigned long>, seastar::future<seastar::rpc::no_wait_type> > + 16 
         1: 0x304d2c0 vtable for seastar::internal::do_with_state<seastar::async<sstables::remove_by_toc_name(seastar::basic_sstring<char, unsigned int, 15u, true>, std::function<void (std::__exception_ptr::exception_ptr)> const&)::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<sstables::remove_by_toc_name(seastar::basic_sstring<char, unsigned int, 15u, true>, std::function<void (std::__exception_ptr::exception_ptr)> const&)::{lambda()#1}>::type&&)...)::work, seastar::future<> > + 16 
         1: 0x335e3b0 vtable for seastar::continuation<seastar::future<>::then_wrapped_impl<seastar::future<>::finally_body<seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)1>()::{lambda()#2}::operator()() const::{lambda()#2}::operator()() const::{lambda()#2}, false>, seastar::future<> >(seastar::future<>::finally_body<seastar::rpc::connection::send_loop<(seastar::rpc::connection::outgoing_queue_type)1>()::{lambda()#2}::operator()() const::{lambda()#2}::operator()() const::{lambda()#2}, false>&&)::{lambda()#1}::operator()() const::{lambda(seastar::future_state<>&&)#1}> + 16 
(gdb) quit

[avikivity]

@tgrabiec @gleb-cloudius @elcallio please help it looks like there are 91k commitlog segments, and 91k commitlog segment managers? how can this be?

[glommer]

Maybe flushes are falling behind? We used to have metrics (not in the default dashboard) for pending flushes. According to the code we still do

should be something like scylla_commitlog_pending_flushes (granana should auto-complete)

@gibsn maybe try plotting that ?

[gibsn]

@glommer,

beccasc7 failed at ~17:18

[tgrabiec]

Maybe those were created by the hints manager.

[glommer]

hi @gibsn the first two metrics you plotted are not interesting the way they are. They are monotonic counters so you have to irate them. The last one is indeed a gauge and it does show that we have many pending flushes.

There is another metric of interest (I am having to read a lot of that code again, rusty on it):

++totals.flush_limit_exceeded (should be scylla_commitlog_flush_limit_exceeded)

But overall, if we can't flush (disk too busy?) we will not be able to get rid of the commitlog segments and that could explain the memory issue at the surface.

[glommer]

@gibsn let me know what does commitlog_total_space_in_mb in your yaml says

[denesb]

@glommer flushes can fail for other reason as well, I've seen a case where memtable flushes failed because memory was fragmented and a large collection couldn't be linearized for flushing. This can be a vicious cycle if the failed flushes trigger more allocations.

[tgrabiec]

@gibsn Do you see 91k files in your hints directory?

[gibsn]

@glommer,

The last one is indeed a gauge and it does show that we have many pending flushes.

But at the time the node failed metrics are 0. Am I missing something? The next spike is the node coming online and receiving hints I guess

should be scylla_commitlog_flush_limit_exceeded

This is the one on the first screenshot, here is the irate of it:

But overall, if we can't flush (disk too busy?)

We have separate drives for commitlogs and data. I can provide some info on our throughput or clogs drive metrics if it can be useful

let me know what does commitlog_total_space_in_mb in your yaml says

256000

[gibsn]

@tgrabiec,

Do you see 91k files in your hints directory?

Nope, ~10k. Keep in mind that fail happened a couple of days ago and since then the node has been running for some time

[tgrabiec]

To sum up, the OOM is due to large number of commitlog segments (91k * 32KB =~ 3GB), one per segment manager. If I read the code right, we should have one segment manager per commitlog, so one per hints endpoint.

@gibsn I assume you don't have thousands of endpoints? From grafana it looks like you don't. Then this 91k doesn't make sense.

Maybe we hit some kind of memory corruption which caused loading_shared_values inside hints::manager to always create a new entry for hints instead of reusing the existing one. \cc @vladzcloudius

@gibsn Would it be possible for you to upload that core dump?

[gibsn]

@tgrabiec,

I assume you don't have thousands of endpoints?

I am not sure what you mean by endpoint, we have 10 Scylla nodes and 4 client nodes

Would it be possible for you to upload that core dump?

Unfortunately I can not since it contains user data. But I can do what you need and send the results

[avikivity]

endpoint = scylla node

As a workaround, you can disable hints. Please keep that core dump though, we'll want to extract more information.

[gibsn]

@avikivity ok, ty

Do you happen to know what this means:

Operation timed out for becca.events - received only 1 responses from 2 CL=ONE.

We are getting lots of those when a node fails. Should not one response with CL=ONE be enough?

[glommer]

@gibsn that's likely #6123

[gibsn]

Another node has just failed. Before restarting:

[root@beccasc5 ~]# ls -l /clogs/scylla/ | wc -l
8088

[gibsn]

We will now diasble hints and update to 3.3.2

[gibsn]

@glommer @avikivity can we update?

[vladzcloudius]

To sum up, the OOM is due to large number of commitlog segments (91k * 32KB =~ 3GB), one per segment manager. If I read the code right, we should have one segment manager per commitlog, so one per hints endpoint.

@gibsn I assume you don't have thousands of endpoints? From grafana it looks like you don't. Then this 91k doesn't make sense.

Maybe we hit some kind of memory corruption which caused loading_shared_values inside hints::manager to always create a new entry for hints instead of reusing the existing one. \cc @vladzcloudius

@tgrabiec That would be an interesting one. For this either std::hash or std::equal_to or both should be returning a random value all the time.

There is another possibility however (connecting to @glommer's slow backend theory): hints manager "flushes" hints segments to the disk every 10s (see sender::flush_maybe() -> end_point_hints_manager::flush_current_hints()) where we will force the commitlog object re-creation.

If for any reason the future returned by commitlog::shutdown() resolves before the corresponding commitlog infrastructure and most importantly files are closed then the situation above is going to be possible if I/O backend is slow and we know that their backend, at least where hints are stored, IS slow (spinning disks).

@elcallio Could you, please, take a look?

[gibsn]

@vladzcloudius we use SSD for clogs and, therefore, for hints. We use HDDs for data

[vladzcloudius]

@gibsn Hints are written into the different directory: hints_directory configuration parameter.
Does it point to a location on SSD?

[gibsn]

@vladzcloudius, it points to a location on SSD, right next to commitlogs

[vladzcloudius]

@vladzcloudius, it points to a location on SSD, right next to commitlogs

Got it. Thanks.
This still doesn't change my point.

[avikivity]

@tgrabiec do you know what those 32k allocations are?

Of course that's a minor point compared to those 91k segment managers.

[tgrabiec]

@avikivity Maybe those are wait lists of semaphores, https://github.com/scylladb/seastar/issues/629

[avikivity]

I think you're right. So we're back to trying to understand if it is indeed hints (seems likely) and if so what caused the problem.

[gibsn]

we disabled HH on Friday and since then we had no issues

[avikivity]

@vladzcloudius so it's hints.

I can't see how loading_shared_values can fail here, but something did break.

[slivne]

@piodul / @haaawk please look at the issue - we may have resolved this already (3.2 is old and not supported).

• if you think we have handled it - please comment with which commit / version should have the fix.

[vladzcloudius]

@avikivity I posted a theory here how this could have happened and we need an @elcallio 's opinion.

The code in manager.cc doesn't keep more than a single commitlog instance per destination at any given time.

[piodul]

It seems that this is caused by improper disposal of commitlog by hints manager, more details here: https://github.com/scylladb/scylla/issues/6776

The large number of 128k spans probably correspond to the buffers allocated to commitlog segments: link1, link2

And, as @tgrabiec pointed out, it looks like 32k spans might come from the wait list of a semaphore - probably the flush semaphore of segment_manager: link

@slivne this problem is not fixed yet; we observed similar behavior in #4728, #4843 and #6086 - there were lots of bad_allocs that started to happen after some time - but we only fixed a crash that happened some time after bad_allocs started to appear. We did not investigate bad allocs themselves there, but I believe the reason was the same as here.

@elcallio @gleb-cloudius @vladzcloudius

[vladzcloudius]

It seems that this is caused by improper disposal of commitlog by hints manager, more details here: #6776

The large number of 128k spans probably correspond to the buffers allocated to commitlog segments: link1, link2

And, as @tgrabiec pointed out, it looks like 32k spans might come from the wait list of a semaphore - probably the flush semaphore of segment_manager: link

@slivne this problem is not fixed yet; we observed similar behavior in #4728, #4843 and #6086 - there were lots of bad_allocs that started to happen after some time - but we only fixed a crash that happened some time after bad_allocs started to appear. We did not investigate bad allocs themselves there, but I believe the reason was the same as here.

@elcallio @gleb-cloudius @vladzcloudius

@piodul @gleb-cloudius @elcallio
I'm not sure that the approach taken in this series is correct.
I've already described in comments why I think the problem in question should be handled on the commitlog level and not on the HH level.

If for some reason commitlog is unable to complete those writes and/or keep track of them and limit its own memory consumption - then this should be addressed on the commitlog level.
In particular, if the commitlog's memory consumption goes beyond a particular limit it should apply a back pressure on the caller and not allow itself to cause the OOM situation as in the issue.

If we see that commitlog is too hard to fix - then we should reconsider the way HH uses it and make sure that we always wait till all current commitlog writes are complete every time we re-create the commitlog object.

It's very WRONG to do that by calling the commitlog::release() because it technically aborts all current writes to the corresponding object and as a result we will loose some of hints with no good reason.

I wonder if this patch series passed HH related dtest tests - if it did, then we should probably re-adjust these tests because I'd expect them to catch this.

One option for the proper solution if we give up on fixing the commitlog would be to change _store_gate to be a semaphore instead of a gate and then we should wait till it drops to zero and "lock it" every time before we call "commitlog::shutdown".

We already limit the maximum amount of outstanding writes (see the usage of size_of_hints_in_progress). So, you can introduce a semaphore instead of the counter, initialize it to max_size_of_hints_in_progress and take max_size_of_hints_in_progress before you call a commitlog::shutdown().
And then you should chain a finally clause with a commitlog::release() call.

However, again, I think this would be a very dirty hack which would indicate the fact that we are unable to fix commitlog::shutdown() as described above.

[vladzcloudius]

@piodul After looking at #4728, #4843 and #6086 I don't see why you think they are related to this - in the OOM case, which this issue is about, the backtrace usually looks differently than what was reported in issues above.

[piodul]

@vladzcloudius

Yes, that's true that #4728, #4843 and #6086 described a segfault, not an OOM. I know that this backtrace looks differently - in fact, I don't have an explanation for it. My point is different.

Some context: together with @haaawk, we investigated and found out how the segfault described in those issues is triggered (full explanation here). In a nutshell, if an exception would occur in a particular place in the commitlog, it would cause a crash.

I just think that a low-memory situation could cause the crash. If you look at descriptions of those issues you will see that there were lots of bad_allocs reported in the logs - one such bad_alloc could be thrown in the vulnerable place inside commitlog and cause a segfault.

In those three issues, we provided an explanation on why a bad_alloc could cause the crash, but did not investigate why those bad_allocs were happening in the first place. While I don't have enough information to be 100% sure, I think this memory leak could explain it.

[avikivity]

Already backported to 4.0, 4.1, 4.2.