Docs: Update performance page with latest data for 20.2

[johnrk]

Hey @rmloveland , to confirm, we're committed from a docs perspective update the performance page if @aayushshah15 and @nvanbenschoten evaluate our existing performance benchmarks? They have already committed to re-running TPC-C 110k.

Here's a breakdown of what is expected to be specifically updated:

For our performance page, it would helpful for our users to have the latest performance numbers for TPC-C and other workloads:

TPC-C 150k (scheduled to be done by @aayushshah15 )

• Max Throughput 1,245,462 tpmC
• Max Warehouses with Max Efficiency 100,000 Warehouses
• Max Number of Rows
• Max Unreplicated Data
• Number of Nodes
• Node Machine Type

Linear Scaling

• Number of nodes Cockroach can achieve linear scaling (in 20.1 it was 256 nodes) - update this sentence if needed: "CockroachDB has no theoretical scaling limit and, in practice, can achieve near-linear performance at X nodes."

• KV 95 Throughput and latency table needs latest data:
  

• Sysbench Throughput Report (QPS) needs to be updated with the latest test results as well
  

• Lastly, Sysbench Average Latency needs to be updated with the latest:
  

Related: https://github.com/cockroachdb/docs/issues/8023

[rmloveland]

@johnrk sounds good to me, thank you for filing this!

[johnrk]

@rmloveland , we're now doing TPC-C 150k

[rmloveland]

John Kendall [e-mail] writes:

@rmloveland , we're now doing TPC-C 150k

OK!

Hey @adityamaru I heard that you will be doing the big TPC-C run soon
(this week?).

As part of the work to update our performance docs I will also need to
update the 20.2 docs with instructions for running TPC-C benchmarks,
specifically:

• TPC-C 1k:
  https://www.cockroachlabs.com/docs/v20.2/performance-benchmarking-with-tpc-c-1k-warehouses

• TPC-C 10k:
  https://www.cockroachlabs.com/docs/v20.1/performance-benchmarking-with-tpc-c-10k-warehouses

• TPC-C 100k (or whatever you end up with: 150k, etc.):
  https://www.cockroachlabs.com/docs/v20.1/performance-benchmarking-with-tpc-c-100k-warehouses

What is the best way to coordinate with you to get the new versions of
this information?

Is there a doc with your benchmarking instructions written somewhere?

Or perhaps a script you are using to do the TPC-C runs?

Let me know if I should set up a quick sync with you to discuss.

(PS sorry if this is very bad, late timing, I am working across a couple
of areas and this fell off my radar for a bit.)

[johnrk]

cc @aayushshah15

[aayushshah15]

Hey @rmloveland, what we've been doing so far is mostly following the set of instructions we have on the docs page for 19.2, while keeping track of where we've deviated from those instructions. We also have a benchmarking script we've been using, but it relies on CRL-specific tools like roachprod, would that still be helpful for you? After the run, we could send you a google doc that contains the updated instructions for running this stuff on 20.2 on non-roachprod clusters. How does that sound? If you have any suggestions / things you'd like us to keep in mind while doing the 150K run, we'll make sure to do so.

[aayushshah15]

In general, we haven't had to deviate too much from the instructions we already have for 19.2 (aside from a few launch flags & cluster settings). Thus, I expect the structure of the new TPCC perf page to be almost identical to the current one.

[rmloveland]

Aayush Shah [e-mail] writes:

Hey @rmloveland, what we've been doing so far is mostly following the
set of instructions we have on the docs page for 19.2, while keeping
track of where we've deviated from those instructions. We also have a
benchmarking script we've been using, but it relies on CRL-specific
tools like roachprod, would that still be helpful for you?

It might be, it depends on what ends up in the google doc you mention
below. From your description, sounds like I should start with the
google doc as the source of truth for updating our perf docs, and ask
questions as needed. Does that sound reasonable?

After the run, we could send you a google doc that contains the
updated instructions for running this stuff on 20.2 on non-roachprod
clusters. How does that sound?

That sounds amazing! Thanks for offering to do that, I know it's extra
work.

If you have any suggestions / things you'd like
us to keep in mind while doing the 150K run, we'll make sure to do so.

Mainly that the more detailed and specific the google doc is, the
better. :-)

In general, we haven't had to deviate too much from the instructions
we already have for 19.2 (aside from a few launch flags & cluster
settings).

That's really good to know!

Thus, I expect the structure of the new TPCC perf page to be almost
identical to the current one.

That would be an excellent outcome! Again, thank you for your work on
this. Look forward to seeing the results (also, 150k, wow :-} ).

[jseldess]

Nit to consider while updating the benchmarking docs, @rmloveland: https://github.com/cockroachdb/docs/issues/6564

[jseldess]

Another benchmarking instructions update to consider, @rmloveland: https://github.com/cockroachdb/docs/issues/6325

[rmloveland]

Update (2020-11-03):

I have updated information for the big TPC-C run and am working on updating the instructions for TPC-C 140k now.

I do NOT have updated data for the following tasks described in this issue:

• Number of nodes Cockroach can achieve linear scaling (in 20.1 it was 256 nodes)
• KV 95 Throughput and latency data for 20.2 (for making a new chart)
• Sysbench throughput data for 20.2 (for making a new chart)
• Sysbench average latency for 20.2 (for making a new chart)

@aayushshah15 and @adityamaru is this information something y'all have or can generate? If you can provide data files I can generate the charts.

[rmloveland]

Fixed in #8825, save for the KV95 and Sysbench charts.

Have filed #8950 to track updating perf results for 21.1 (including either updating or deleting kv95 and Sysbench bits).