Openblas: performance in sched_yield (multisocket)
Hi!!!
I'm using mumps with openblas 0.2.18 compiled from sources with this options:
- gcc 5.3.0
- USE_THREAD = 1
- NUM_THREADS = 128
- NO_WARMUP = 1
- #NO_AFFINITY = 1
- #BIGNUMA = 1
- MAX_STACK_ALLOC = 8128
I'm running my code in some machines multisocket (2 or 4) with 24 until 128 cores
The problem is that most of the time is executed by the system. To be more acurate is doing not I/O is doing Context switch. You can see it in the below pictures
This doesn't happend in 1 socket machine, but if I force to use just one CPU (taskset -c 0-31 MyAPP) the performance is also poor.
What can I do to give you more information and try to help¿¿¿
phirus2
All 43 comments
How big is data sample? What CPU? Maybe all the time is spent ripping tiny data sample between CPU caches or NUMA nodes?
brada4
on 31 May 2016
sched_yield is that descheduling your flawed automated debugger is recommending.
man sched_yield ?
brada4
on 31 May 2016
@phirus2 What is are the size of the GEMM ? Could you get a sample of them with adding a trace in interface/gemm.c or with a debugger ?
MUMPS is a sparse solver so he is probably working on small blocks on which you can't get any benefits from the OpenBLAS multi-threading. Anyway if blocks are small we should not even enter in blas_thread_server so that's odd.
jeromerobert
on 31 May 2016
There is no 32-core intel CPU being manufactured
Results are fake at best. There is no issue.
brada4
on 1 Jun 2016
depth 0: 1 Machine (type #1)
depth 1: 4 NUMANode (type #2)
depth 2: 4 Socket (type #3)
depth 3: 4 L3Cache (type #4)
depth 4: 64 L2Cache (type #4)
depth 5: 64 L1dCache (type #4)
depth 6: 64 L1iCache (type #4)
depth 7: 64 Core (type #5)
depth 8: 128 PU (type #6)
cat /proc/cpuinfo | grep -i "model name" -B4 | tail -n5
processor : 127
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E7-8860 v3 @ 2.20GHz
@brada4 As you can see it is not a fake CPU.
@jeromerobert Ok i'm execute a bigger problem and give you the sizes of my sistem to analize what is happening
phirus2
on 1 Jun 2016
I'm resolving an sparse matriz equation with this sizes:
- A ==> [(50_50_50) x (50_50_50) ] == [125k x 125k]. For each row there is 7 nz value. So the matriz has aprox 7MB
- B /X ==> [50_50_50] == [125k] ~= 1MB
But internally is doing LU decomposition and I guess it' is using dens matriz. because the total ram used by the program is ~1GB. Changing the size to 100x100x100 the total size of the problem went up to ~15GB. I think is enough big to share avoid that context switch
The profiler info of openBlas is
====== BLAS Profiling Result =======
Function No. of Calls Time Consumption Efficiency Bytes/cycle Wall Time(Cycles)
idamax : 5306040 0.19% 0.000% 2.75 875257291
dgemm : 9533360 91.66% 2.334% 0.85 353009864
dtrsm : 6395160 8.15% 3.402% 0.75 -425599732
--------------------------------------------------------------------
Total : 21234560 2.416% 0.85
The extra info of mumps is ....
=================================================
MUMPS compiled with option -Dmetis
=================================================
L U Solver for unsymmetric matrices
Type of parallelism: Working host
****** ANALYSIS STEP ********
Resetting candidate strategy to 0 because NSLAVES=1
... Structural symmetry (in percent)= 98
... No column permutation
Ordering based on METIS
ELAPSED TIME SPENT IN METIS reordering = 0.9891
Leaving analysis phase with ...
INFOG(1) = 1
INFOG(2) = 208155
-- (20) Number of entries in factors (estim.) = 85425349
-- (3) Storage of factors (REAL, estimated) = 85425349
-- (4) Storage of factors (INT , estimated) = 2406061
-- (5) Maximum frontal size (estimated) = 3644
-- (6) Number of nodes in the tree = 52849
-- (32) Type of analysis effectively used = 1
-- (7) Ordering option effectively used = 5
ICNTL(6) Maximum transversal option = 0
ICNTL(7) Pivot order option = 7
Percentage of memory relaxation (effective) = 20
Number of level 2 nodes = 0
Number of split nodes = 0
RINFOG(1) Operations during elimination (estim)= 1.449D+11
** Rank of proc needing largest memory in IC facto : 0
** Estimated corresponding MBYTES for IC facto : 964
** Estimated avg. MBYTES per work. proc at facto (IC) : 964
** TOTAL space in MBYTES for IC factorization : 964
** Rank of proc needing largest memory for OOC facto : 0
** Estimated corresponding MBYTES for OOC facto : 278
** Estimated avg. MBYTES per work. proc at facto (OOC) : 278
** TOTAL space in MBYTES for OOC factorization : 278
ELAPSED TIME IN ANALYSIS DRIVER= 1.2585
Entering DMUMPS 5.0.1 driver with JOB, N, NZ = 5 132651 1061208
****** FACTORIZATION STEP ********
GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ...
NUMBER OF WORKING PROCESSES = 1
OUT-OF-CORE OPTION (ICNTL(22)) = 0
REAL SPACE FOR FACTORS = 85425349
INTEGER SPACE FOR FACTORS = 2406061
MAXIMUM FRONTAL SIZE (ESTIMATED) = 3644
NUMBER OF NODES IN THE TREE = 52849
MEMORY ALLOWED (MB -- 0: N/A ) = 0
Convergence error after scaling for ONE-NORM (option 7/8) = 0.85D+00
Maximum effective relaxed size of S = 116410103
Average effective relaxed size of S = 116410103
GLOBAL TIME FOR MATRIX DISTRIBUTION = 0.0203
** Memory relaxation parameter ( ICNTL(14) ) : 20
** Rank of processor needing largest memory in facto : 0
** Space in MBYTES used by this processor for facto : 964
** Avg. Space in MBYTES per working proc during facto : 964
ELAPSED TIME FOR FACTORIZATION = 7.0425
Maximum effective space used in S (KEEP8(67)) = 97008403
Average effective space used in S (KEEP8(67)) = 97008403
** EFF Min: Rank of processor needing largest memory : 0
** EFF Min: Space in MBYTES used by this processor : 805
** EFF Min: Avg. Space in MBYTES per working proc : 805
GLOBAL STATISTICS
RINFOG(2) OPERATIONS IN NODE ASSEMBLY = 1.642D+08
------(3) OPERATIONS IN NODE ELIMINATION= 1.449D+11
INFOG (9) REAL SPACE FOR FACTORS = 85425349
INFOG(10) INTEGER SPACE FOR FACTORS = 2406061
INFOG(11) MAXIMUM FRONT SIZE = 3644
INFOG(29) NUMBER OF ENTRIES IN FACTORS = 85425349
INFOG(12) NUMBER OF OFF DIAGONAL PIVOTS = 15
INFOG(13) NUMBER OF DELAYED PIVOTS = 0
INFOG(14) NUMBER OF MEMORY COMPRESS = 0
ELAPSED TIME IN FACTORIZATION DRIVER= 7.0875
****** SOLVE & CHECK STEP ********
STATISTICS PRIOR SOLVE PHASE ...........
NUMBER OF RIGHT-HAND-SIDES = 1
BLOCKING FACTOR FOR MULTIPLE RHS = 1
ICNTL (9) = 1
--- (10) = 0
--- (11) = 0
--- (20) = 0
--- (21) = 0
--- (30) = 0
** Rank of processor needing largest memory in solve : 0
** Space in MBYTES used by this processor for solve : 961
** Avg. Space in MBYTES per working proc during solve : 961
TIME to scatter right-hand-sides = 0.002222
TIME in solution step (fwd/bwd) = 0.383480
.. TIME in forward (fwd) step = 0.237609
.. TIME in backward (bwd) step = 0.145850
TIME to gather solution = 0.001802
ELAPSED TIME IN SOLVE DRIVER= 0.4491
Entering DMUMPS 5.0.1 driver with JOB, N, NZ = 5 132651 1061208
.
phirus2
on 1 Jun 2016
@phirus2 I'm not sure how to deduce the size (m x n) of the gemm blas calls from this information. I (we ?) don't the MUMPS internal and don't know how it use GEMM. You have to get this information from OpenBLAS it self with adding trace or using gdb.
jeromerobert
on 1 Jun 2016
Do you see an improvement when you limit the number of threads to the count of full-featured cores ? Hyperthreading is not going to help for compute-intensive tasks, what you see could be each two threads competing for access to the floating-point hardware.
martin-frbg
on 1 Jun 2016
I thought that but decreasing the number of threads (64, 32, 16) the performance efficiency is the same. Even fixing to same cores using taskset to avoid the context switch. But I still have the poor performance efficiency.
@jeromerobert have modified the src to print the headers of dgemm. First of all... there are soooo many calls in just one iteration of mumps solver (110k) and seems that many of them are matrix of ¿1x1?
TRANSA:N TRANSB:N M:1 N:1 K:1 alpha:-1.000000 a:-0.140344 ldA:2 b:1.572134 ldb:2 beta:1.000000 c:0.000000 ldc:2
TRANSA:N TRANSB:N M:1 N:1 K:1 alpha:-1.000000 a:-0.141675 ldA:2 b:1.583998 ldb:2 beta:1.000000 c:0.000000 ldc:2
TRANSA:N TRANSB:N M:1 N:1 K:1 alpha:-1.000000 a:-0.142773 ldA:2 b:1.593849 ldb:2 beta:1.000000 c:0.000000 ldc:2
TRANSA:N TRANSB:N M:1 N:1 K:1 alpha:-1.000000 a:-0.142272 ldA:2 b:1.589344 ldb:2 beta:1.000000 c:0.000000 ldc:2
TRANSA:N TRANSB:N M:1 N:1 K:1 alpha:-1.000000 a:-0.140708 ldA:2 b:1.575375 ldb:2 beta:1.000000 c:0.000000 ldc:2
TRANSA:N TRANSB:N M:1 N:1 K:1 alpha:-1.000000 a:-0.140055 ldA:2 b:1.569573 ldb:2 beta:1.000000 c:0.000000 ldc:2
TRANSA:N TRANSB:N M:1 N:1 K:1 alpha:-1.000000 a:-0.142308 ldA:2 b:1.589665 ldb:2 beta:1.000000 c:0.000000 ldc:2
Actually there are:
- ~45k calls with N * M == 1
- ~71k calls with N * M < 36
- ~100k calls with N * M < 1000
- ~10k calls with 1000
- ~2k calls with 10000
- ~2k calls with 10000
So the problem is of mumps an his partitioning????
phirus2
on 1 Jun 2016
So the problem is of mumps an his partitioning????
Yes and that was expected. I don't think any sparse solver could rely on BLAS for parallelization. OpenBLAS dgemm will only enable threading when M* N * K > 8192 * GEMM_MULTITHREAD_THRESHOLD (= 32768 most of the time). So in your case very few of them will be run in parallel.
The problem is that in OpenBLAS does not select the number of threads very smartly. If you have a matrix with M_N_K = 32769 it will run on all the core. In your case it will be very inefficient because 32769 is much to small for 64 cores. I guess that even 244000 is much too small for 64 cores. That's why sched_yield takes so much time in your profiling. Too be more efficient we should select the right number of threads between 1 and 64 depending on the size of the matrix (ex: 4 or 6 for MNK=24400). But this is not implemented.
To get good performance with MUMPS you should use MPI not multi-threading. If you absolutely want a shared memory solution, try PaStiX.
jeromerobert
on 1 Jun 2016
You have 4 10-core cpus
7mb is microscopic, best results are in 10 cores and less
brada4
on 1 Jun 2016
45k calls to m.n=1 should not call gemm but gemv instead
brada4
on 1 Jun 2016
Absolutely agree. Thousands of gemm with 1x1 'matrix' is a mistake.
Thanks for the advise and the explanation of the limits of parallelization. I will take a look to that library pastix and also report the results to mumps
phirus2
on 2 Jun 2016
You can see NUMA stride in sysctl. You need at least a stride per multicore CPU in node, never less
brada4
on 2 Jun 2016
Hi,
I just spoke with some of the mumps developpers : to him, it seems impossible that mumps calls so many GEMMs of size 1x1 (unless you have a _very_ specific matrix). During factorization, a sparse matrix will progressively become dense (because of the fill-in) , so sparse solvers (like mumps or pastix) actually manipulate dense matrices (maybe not big enough to be efficient on large number of cores). Is it possible that these 1x1 GEMM actually come from openblas himself (wrapped from gemv or axpy or...) ?
gsylvand
on 3 Jun 2016
@gsylvand I don't know where come from. I just know is that function was launched such times with these params. The matrix which we work is a tridiagonal with 4 more diagonals (2 upper and 2 lower).
I would appreciate know how to open a bug or an issue in mumps. Because in the website I didn't see it. Just contact info
phirus2
on 3 Jun 2016
AFAIK, there is no bugtracker for mumps, but there is a mailing list [email protected] where developpers are responding quite quickly. This is where I go when I have a bug with mumps...
gsylvand
on 3 Jun 2016
Perhaps put a breakpoint on gemm (or more specific, add a conditional somewhere in gemm that does a printf when matrix size is just 1x1 and put the breakpoint on that printf line) and trace that invocation back "up" to see if it an OpenBLAS function that split it up like that, or if the matrix was 1x1
all the way back to mumps ?
martin-frbg
on 3 Jun 2016
Lets set up some debug code
add 2 lines to interface/gemm.c
Index: interface/gemm.c
===================================================================
--- interface/gemm.c (revision 6436)
+++ interface/gemm.c (working copy)
@@ -38,6 +38,7 @@
#include <stdio.h>
#include <stdlib.h>
+#include <signal.h>
#include "common.h"
#ifdef FUNCTION_PROFILE
#include "functable.h"
@@ -387,6 +388,7 @@
#endif
if ((args.m == 0) || (args.n == 0)) return;
+ if ((args.m==1)&&(args.n==1)&&(args.k==1)) raise(SIGABRT);
#if 0
fprintf(stderr, "m = %4d n = %d k = %d lda = %4d ldb = %4d ldc = %4d\n",
Then run your program from gdb (i.e 'gdb ./yourprogram', inside gdb 'r')
It will bail out when it reaches suboptimal case you pointed out
once gdb catches the break run
gdb> bt
gdb> c
few times to find unique call chains leading there (and check where 1x1 matrix starts)
Once done with that and still with time also check cases where m or k == 1 and it should be vector-to-matrix op, and where n==1 it should be vector-to-vector op
PS Even IF ones come from MUMPS it is likely that there is some optimisation possible by returning from openblas call quicker up the call chain.
brada4
on 4 Jun 2016
Oops. You also need to skip building of 'tests' in toplevel makefile as this code will make them bail out too...
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./sblat3 < ./sblat3.dat
Program received signal SIGABRT: Process abort signal.
Backtrace for this error:
#0 0x7FCEDFF73507
#1 0x7FCEDFF72700
#2 0x7FCEDF26211F
#3 0x7FCEDFA3C73B
#4 0x41C279 in sgemm_
#5 0x410860 in schk1_
#6 0x41B6E7 in MAIN__ at sblat3.f:?
/bin/sh: line 1: 78182 Aborted OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./sblat3 < ./sblat3.dat
Note that sched_yield() creates a set of interesting OS level issues. I've noticed a > 10% throughput increase by doing
--- OpenBLAS-0.2.16/common.h~ 2016-03-15 18:49:10.000000000 +0000
+++ OpenBLAS-0.2.16/common.h 2016-06-12 16:12:41.864694505 +0000
@@ -349,7 +349,7 @@
*/
#ifndef YIELDING
-#define YIELDING sched_yield()
+#define YIELDING usleep(10)
#endif
fenrus75
on 11 Jul 2016
@phirus2 did this ever get resolved ? I looked through the archives of the mumps-users mailing list since june but did not see anything that appeared to match your problem.
martin-frbg
on 9 Jan 2017
This change seems to be a particularly bad for the performance of small and especially medium-sized matrices. Here's a gemm benchmark using sched_yield:
Run on (12 X 3800 MHz CPU s)
CPU Caches:
L1 Data 32K (x6)
L1 Instruction 32K (x6)
L2 Unified 256K (x6)
L3 Unified 15360K (x1)
-----------------------------------------------------
Benchmark Time CPU Iterations
-----------------------------------------------------
BM_SGEMM/4 246 ns 246 ns 17028578
BM_SGEMM/6 321 ns 321 ns 13317257
BM_SGEMM/8 344 ns 344 ns 12427136
BM_SGEMM/10 412 ns 412 ns 10291541
BM_SGEMM/16 606 ns 606 ns 6926760
BM_SGEMM/20 800 ns 800 ns 5264078
BM_SGEMM/32 2050 ns 2050 ns 2063846
BM_SGEMM/40 3086 ns 3086 ns 1353523
BM_SGEMM/64 9476 ns 9476 ns 443674
BM_SGEMM/80 36089 ns 34515 ns 122931
BM_SGEMM/100 40914 ns 38593 ns 97910
BM_SGEMM/128 61242 ns 57638 ns 69829
BM_SGEMM/150 69747 ns 68803 ns 62768
BM_SGEMM/200 88773 ns 88730 ns 47978
BM_SGEMM/256 227258 ns 220578 ns 20418
BM_SGEMM/300 235833 ns 226789 ns 17611
BM_SGEMM/400 472455 ns 470280 ns 8775
BM_SGEMM/500 1123952 ns 1030861 ns 3636
BM_SGEMM/600 1393240 ns 1367583 ns 3183
BM_SGEMM/700 2532205 ns 2493220 ns 1727
BM_SGEMM/800 3821254 ns 3680541 ns 1037
BM_SGEMM/1000 5724859 ns 5713986 ns 785
BM_SGEMM/2000 41394059 ns 41328371 ns 90
BM_DGEMM/4 253 ns 253 ns 16684921
BM_DGEMM/6 294 ns 294 ns 14060050
BM_DGEMM/8 327 ns 327 ns 12852336
BM_DGEMM/10 430 ns 430 ns 9903132
BM_DGEMM/16 688 ns 688 ns 6100390
BM_DGEMM/20 1058 ns 1058 ns 3972796
BM_DGEMM/32 2933 ns 2933 ns 1451017
BM_DGEMM/40 4990 ns 4989 ns 843290
BM_DGEMM/64 16312 ns 16298 ns 256596
BM_DGEMM/80 30987 ns 30388 ns 129788
BM_DGEMM/100 39382 ns 36503 ns 132214
BM_DGEMM/128 62843 ns 60830 ns 71023
BM_DGEMM/150 84489 ns 81598 ns 54367
BM_DGEMM/200 142225 ns 137729 ns 28107
BM_DGEMM/256 241833 ns 241484 ns 16694
BM_DGEMM/300 376695 ns 372174 ns 12520
BM_DGEMM/400 726624 ns 726224 ns 6137
BM_DGEMM/500 1305679 ns 1305189 ns 3309
BM_DGEMM/600 2484410 ns 2432980 ns 1810
BM_DGEMM/700 3682210 ns 3628288 ns 1202
BM_DGEMM/800 5871802 ns 5793628 ns 797
BM_DGEMM/1000 10359337 ns 10254159 ns 420
BM_DGEMM/2000 70603271 ns 70524090 ns 62
and using usleep(10):
Run on (12 X 3800 MHz CPU s)
CPU Caches:
L1 Data 32K (x6)
L1 Instruction 32K (x6)
L2 Unified 256K (x6)
L3 Unified 15360K (x1)
-----------------------------------------------------
Benchmark Time CPU Iterations
-----------------------------------------------------
BM_SGEMM/4 290 ns 290 ns 2379045
BM_SGEMM/6 358 ns 358 ns 1913060
BM_SGEMM/8 381 ns 381 ns 1832770
BM_SGEMM/10 456 ns 456 ns 1549668
BM_SGEMM/16 647 ns 647 ns 1075645
BM_SGEMM/20 845 ns 845 ns 825140
BM_SGEMM/32 2063 ns 2063 ns 338270
BM_SGEMM/40 3139 ns 3139 ns 223982
BM_SGEMM/64 9469 ns 9470 ns 73677
BM_SGEMM/80 215715 ns 32480 ns 21130
BM_SGEMM/100 213620 ns 29254 ns 24238
BM_SGEMM/128 234462 ns 52012 ns 13497
BM_SGEMM/150 243646 ns 61948 ns 11183
BM_SGEMM/200 252119 ns 70189 ns 9675
BM_SGEMM/256 373665 ns 172991 ns 4115
BM_SGEMM/300 375489 ns 192592 ns 3651
BM_SGEMM/400 640518 ns 394721 ns 1774
BM_SGEMM/500 1000332 ns 741130 ns 951
BM_SGEMM/600 1370778 ns 1103237 ns 638
BM_SGEMM/700 2240202 ns 1933774 ns 367
BM_SGEMM/800 3077418 ns 2645025 ns 263
BM_SGEMM/1000 5687645 ns 4276803 ns 163
BM_SGEMM/2000 44240844 ns 35940586 ns 20
BM_DGEMM/4 298 ns 298 ns 2340901
BM_DGEMM/6 336 ns 336 ns 2108827
BM_DGEMM/8 369 ns 369 ns 1908432
BM_DGEMM/10 475 ns 475 ns 1476408
BM_DGEMM/16 720 ns 720 ns 973000
BM_DGEMM/20 1138 ns 1138 ns 619431
BM_DGEMM/32 2946 ns 2946 ns 239137
BM_DGEMM/40 5025 ns 5026 ns 139832
BM_DGEMM/64 16202 ns 16202 ns 43268
BM_DGEMM/80 212307 ns 29382 ns 23685
BM_DGEMM/100 214416 ns 31720 ns 22158
BM_DGEMM/128 236865 ns 54680 ns 12883
BM_DGEMM/150 251692 ns 70811 ns 9742
BM_DGEMM/200 266804 ns 87101 ns 8183
BM_DGEMM/256 423978 ns 211992 ns 3263
BM_DGEMM/300 538503 ns 282711 ns 2475
BM_DGEMM/400 818981 ns 568654 ns 1230
BM_DGEMM/500 1405214 ns 1144277 ns 616
BM_DGEMM/600 2321151 ns 1900062 ns 365
BM_DGEMM/700 3717390 ns 3266041 ns 215
BM_DGEMM/800 5164727 ns 4623332 ns 151
BM_DGEMM/1000 9569988 ns 8354815 ns 84
BM_DGEMM/2000 70862683 ns 61130896 ns 11
I'm not sure that a fixed-size wait is going to do better than sched_yield. While it's true that sched_yield() is a hot spot (in the above benchmark it's taking 20% of the runtme), if you look at traces with usleep it's not doing any better (and in many cases worse). I believe the issue with sched_yield() is that it causes synchronization in the kernel, but I'm not sure how to get around that. At the very least usleep (also pthread_yield) are worse.
I'd strongly recommend reverting this change.
oon3m0oo
on 11 Jun 2018
That is the first time I have seen anybody in favor of sched_yield... in my tests with smaller numbers of cpus, the performance difference seemed negligible - except on one particular laptop where sched_yield led to serious thermal throttling. Intel cpus were the only ones to use sched_yield() while ARM,POWER, and even AMD/x86 all have some variation of "asm(nop);" I believe "fenrus75" who commented above is a senior engineer on Intel's Linux team.
Perhaps there is something peculiar about our GEMM threading or this particular benchmark that skews the numbers ? (BTW; what is your hardware and operating system - if you happen to be on
*BSD or something else non-Linux, perhaps the scheduler behaves differently there and the change
should have been made for Linux only ?)
martin-frbg
on 11 Jun 2018
This is a debian-like linux distro that we use here at Google (so not BSD-based) with Intel CPUs (Xeon E5-1650, to be precise). It's possible that this benchmark skews things, but I'm curious why OpenBLAS doesn't use something like select, (e)poll, or even a condition to wait rather than yielding. Looping over a yield is generally not a good pattern to use for waiting until there is work to be done, and the kernel is quite good at fd-based triggers (Android, for example, does this extensively via epoll for very high-performance triggering, though I'm running these benchmarks on a desktop).
oon3m0oo
on 11 Jun 2018
openmp does the same (poll) so there's some precedent.
doing a "rep nop" (aka pause) is great for very short delays,
for longer delays it's better to put the cpu into an idle state (so that
the power budget can be re-allocated)
one of the interesting questions is if the matrix blocks for threading are
too small for newer cpu systems
On Mon, Jun 11, 2018 at 7:06 AM oon3m0oo notifications@github.com wrote:
This is a debian-like linux distro that we use here at Google (so not
BSD-based) with Intel CPUs (Xeon E5-1650, to be precise). It's possible
that this benchmark skews things, but I'm curious why OpenBLAS doesn't use
something like select, (e)poll, or even a condition to wait rather than
yielding. Looping over a yield is generally not a good pattern to use for
waiting until there is work to be done, and the kernel is quite good at
fd-based triggers (Android, for example, does this extensively via epoll
for very high-performance triggering, though I'm running these benchmarks
on a desktop).—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/xianyi/OpenBLAS/issues/900#issuecomment-396255686,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABPeFbKX0FneXWn3XZ4bnepWgPTvOfBMks5t7nl0gaJpZM4Iql30
.
fenrus75
on 11 Jun 2018
@oon3m0oo could you try usleep(0) and usleep(1) instead of usleep(10) and tell the result?
It looks like useless sleep gets just added to each call when it is meant to be sched_yield but with less waste.
brada4
on 11 Jun 2018
OpenMP calls sched_yield from kmp_yield (I'm looking at llvm's version), but that is a very small fraction of trace time in what I'm seeing. It's worth noting that OpenBLAS's choice of YIELDING shows up a large fraction of the time for both OpenMP and non-OpenMP builds. I'm actively looking into this now. =)
It's possible the profiles are wrong, They're telling me that the calls to routine() (line 296) in blas_server_omp are causing yielding, and I'm not sure I understand that.
oon3m0oo
on 11 Jun 2018
While I'm doing this, is there a generally preferred method for threading? In my testing OpenMP has outperformed regular pthreads pretty consistently (which is a bit odd to me since OpenMP is built on pthreads). Since they're different builds I'd rather limit my testing to just one to make this go faster.
oon3m0oo
on 11 Jun 2018
I'll readily admit that I had next to zero experience with multithreading and associated topics before I got trapped in OpenBLAS maintenance (hence my lame "appeal to authority" above). "#define YIELDING sched_yield()" goes back to GotoBLAS2-1.13 (the 1.08 version even had "YIELDING" defined to nothing).
OpenMP is "believed to be" slower but "mostly" thread safe, while the pure pthreads build may still house some bugs (although I believe to have fixed the major ones in the past two years with the help of valgrind and thread sanitizer, but this brings us back to the first sentence of this comment). Pure pthreads with USE_SIMPLE_THREADED_LEVEL3 set may be a safe(r) option.
martin-frbg
on 11 Jun 2018
multithreading topics are a few PhD's worth of stuff.
so to be more pragmatic, how about
1) revert the change for now until we collect better data
2) we figure out which benchmarks matter
3) collect data
I will either do this myself or find help in my team with this, hopefully
oon3mm0oo can help as well
On Mon, Jun 11, 2018 at 7:33 AM Martin Kroeker notifications@github.com
wrote:
I'll readily admit that I had next to zero experience with multithreading
and associated topics before I got trapped in OpenBLAS maintenance (hence
my lame "appeal to authority" above). "#define YIELDING sched_yield()" goes
back to GotoBLAS2-1.13 (the 1.08 version even had "YIELDING" defined to
nothing).
OpenMP is "believed to be" slower but "mostly" thread safe, while the pure
pthreads build may still house some bugs (although I believe to have fixed
the major ones in the past two years with the help of valgrind and thread
sanitizer, but this brings us back to the first sentence of this comment).
Pure pthreads with USE_SIMPLE_THREADED_LEVEL3 set may be a safe(r) option.—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/xianyi/OpenBLAS/issues/900#issuecomment-396265115,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABPeFRPDI_henDCSBxwRFSZ2sPJfg-8Qks5t7n_UgaJpZM4Iql30
.
fenrus75
on 11 Jun 2018
@fenrus75 I suspect just fixed delay is added with 10us, maybe even twice since it gets called in main thread too. 0us would hide the issue for some time to come.
Lets wait if it helped (in previous discussion around sched_yield i suspect nobody tested with high precision timers.
brada4
on 11 Jun 2018
On rereading #923 and my earlier PR #1051 I am ashamed to notice that partial results back then already pointed to a performance decrease from using usleep rather than rep(nop). (What brought me back to the topic was seeing the usleep patch in the clearlinux repo...) I'll revert this in a minute.
martin-frbg
on 11 Jun 2018
the hybrid would be an exponential-backoff style... sleep 0 for a bit, then
1 then 2 .. upto 10
putting the cpu into an idle state has system level performance benefits,
the other cores can run faster, so I do not want to discount the value of
that.
(this is not universal across all possible CPUs)
On Mon, Jun 11, 2018 at 7:58 AM Andrew notifications@github.com wrote:
@fenrus75 https://github.com/fenrus75 I suspect just fixed delay is
added with 10us, maybe even twice since it gets called in main thread too. 0us
would hide the issue for some time to come
https://github.com/xianyi/OpenBLAS/issues/900#issuecomment-396259664.
Lets wait if it helped (in previous discussion around sched_yield i
suspect nobody tested with high precision timers.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/xianyi/OpenBLAS/issues/900#issuecomment-396273814,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABPeFTLZyDFv0U7QrmtCKhlO_NlR4zukks5t7oV-gaJpZM4Iql30
.
fenrus75
on 11 Jun 2018
FWIW I see more consistent performance when using __builtin_ia32_pause() rather than usleep() for those mid-sized matrices. @fenrus75, what's your take on that? I recall Intel recommended using that for busy loops a while back, though I have no idea if it's still a good thing on modern architectures (though it removed the bump in medium-sized matrices for me). Performance for smaller matrices is still worse than shed_yield, though (this is without openmp):
Run on (12 X 3800 MHz CPU s)
CPU Caches:
L1 Data 32K (x6)
L1 Instruction 32K (x6)
L2 Unified 256K (x6)
L3 Unified 15360K (x1)
-----------------------------------------------------
Benchmark Time CPU Iterations
-----------------------------------------------------
BM_SGEMM/4 290 ns 290 ns 2355161
BM_SGEMM/6 361 ns 361 ns 1956911
BM_SGEMM/8 382 ns 382 ns 1830327
BM_SGEMM/10 453 ns 453 ns 1552514
BM_SGEMM/16 649 ns 649 ns 1074342
BM_SGEMM/20 832 ns 832 ns 833405
BM_SGEMM/32 2155 ns 2155 ns 334638
BM_SGEMM/40 3153 ns 3153 ns 222982
BM_SGEMM/64 9463 ns 9463 ns 73678
BM_SGEMM/80 61145 ns 60207 ns 11634
BM_SGEMM/100 54527 ns 52781 ns 12753
BM_SGEMM/128 81174 ns 77631 ns 8800
BM_SGEMM/150 93845 ns 87261 ns 8053
BM_SGEMM/200 108452 ns 105637 ns 6497
BM_SGEMM/256 387648 ns 301357 ns 3185
BM_SGEMM/300 376078 ns 300721 ns 2803
BM_SGEMM/400 601002 ns 550217 ns 1000
BM_SGEMM/500 1028824 ns 964436 ns 700
BM_SGEMM/600 1576130 ns 1386449 ns 443
BM_SGEMM/700 2512956 ns 2455080 ns 269
BM_SGEMM/800 3605126 ns 3423020 ns 192
BM_SGEMM/1000 6061129 ns 5952332 ns 137
BM_SGEMM/2000 47175561 ns 46554248 ns 14
BM_DGEMM/4 298 ns 298 ns 2330945
BM_DGEMM/6 334 ns 334 ns 2110857
BM_DGEMM/8 374 ns 374 ns 1892115
BM_DGEMM/10 477 ns 477 ns 1476261
BM_DGEMM/16 739 ns 739 ns 949175
BM_DGEMM/20 1123 ns 1123 ns 589482
BM_DGEMM/32 2938 ns 2938 ns 235881
BM_DGEMM/40 5018 ns 5017 ns 138310
BM_DGEMM/64 16409 ns 16408 ns 42941
BM_DGEMM/80 61732 ns 56806 ns 12545
BM_DGEMM/100 67572 ns 60401 ns 11536
BM_DGEMM/128 89098 ns 84852 ns 7190
BM_DGEMM/150 106198 ns 102878 ns 6858
BM_DGEMM/200 149058 ns 141427 ns 5298
BM_DGEMM/256 319800 ns 299454 ns 2373
BM_DGEMM/300 427313 ns 400634 ns 2074
BM_DGEMM/400 846095 ns 817089 ns 1043
BM_DGEMM/500 1480201 ns 1323197 ns 557
BM_DGEMM/600 2370991 ns 2235729 ns 315
BM_DGEMM/700 3951761 ns 3695940 ns 195
BM_DGEMM/800 6467427 ns 5550365 ns 132
BM_DGEMM/1000 19683714 ns 16225276 ns 73
BM_DGEMM/2000 79131424 ns 76075481 ns 8
I should say, my benchmark is pretty simple. It just uses https://github.com/google/benchmark and runs through various sizes as shown in my comments, generates three random matrices, and calls [sd]gemm_ many times in a loop. I can see about uploading it to OpenBLAS, but we'd need to clean it up a bit (using an internal library for something minor).
oon3m0oo
on 11 Jun 2018
builtin_ia32_pause probably corresponds to the asm "pause" or "rep(nop)" instruction ? With the very small matrices, how many threads are there (or who is yielding to whom ? I assume you do not count the time spent setting up the random matrices)
martin-frbg
on 11 Jun 2018
(I'd love a standardized benchmark)
pause (aka "rep nop") is a good instruction for polling busy wait. It is
basically a "bubble" in the cpu pipeline of between 100 to 500 cpu cycles
(varies a little by cpu model).
This bubble means that a hyperthreading sibling gets to core to itself, but
it also means power consumption is reduced for this period (and reduced
power consumption may lead to other cores being able to go faster)
sched_yield() is sort of the worst, it makes the cpu busy (including cross
cpu locks in the kernel) while not getting any useful work done.
for very small matrixes I am surprised we even get down these code paths..
any synchronization costs likely outweigh the threading benefits
On Mon, Jun 11, 2018 at 8:24 AM oon3m0oo notifications@github.com wrote:
I should say, my benchmark is pretty simple. It just uses
https://github.com/google/benchmark and runs through various sizes as
shown in my comments, generates three random matrices, and calls [sd]gemm_
many times in a loop. I can see about uploading it to OpenBLAS, but we'd
need to clean it up a bit (using an internal library for something minor).—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/xianyi/OpenBLAS/issues/900#issuecomment-396283113,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABPeFXzwn4dkkZ8kwRY8ShVCaq9D9kCHks5t7ovHgaJpZM4Iql30
.
fenrus75
on 11 Jun 2018
What's odd is there should really be _no_ threads for small matrices. I've been wondering about this for a few days now, why performance gets worse for very small matrices when there are no threads (that is, we are definitely under the threshold) but threading is enabled in the build. I'll have to profile just the small matrices, but I'm bogged down by something else right now. Should we open up a new issue just for this or keep it all here?
oon3m0oo
on 11 Jun 2018
I think having a separate issue for "threading is used for too small
matrixes" makes sense, I'd like to be on it tho ;)
On Mon, Jun 11, 2018 at 8:41 AM oon3m0oo notifications@github.com wrote:
What's odd is there should really be no threads for small matrices.
I've been wondering about this for a few days now, why performance gets
worse for very small matrices when there are no threads (that is, we are
definitely under the threshold) but threading is enabled in the build. I'll
have to profile just the small matrices, but I'm bogged down by something
else right now. Should we open up a new issue just for this or keep it all
here?—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/xianyi/OpenBLAS/issues/900#issuecomment-396288791,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABPeFUQewABrH8VNA9VkzvIjFBFAnACxks5t7o-zgaJpZM4Iql30
.
fenrus75
on 11 Jun 2018
I'm building with #define MONITOR in blas_server.c now. Maybe there are lingering threads created during initialization of the library (there is a timeout variable in Makefile.rule for these) that somehow manage to interfere. The curious difference between running with and without OPENBLAS_NUM_THREADS=1 for a workload that should use only one thread is also noted in #1544 (but that issue thread is pretty much frogged, so feel free to start a new one)
martin-frbg
on 11 Jun 2018
Given that this is a closed issue, I am going to go ahead and start a new thread for small matrix performance with threading enabled (@oon3m0oo and I are colleagues).
sandwichmaker
on 11 Jun 2018
sandwichmaker
on 11 Jun 2018
Related issues
hholst80
·
17Comments
marker68
·
8Comments
santo4ul
·
9Comments
aytekinar
·
10Comments
yaswanthsvist
·
9Comments
Most helpful comment
Perhaps put a breakpoint on gemm (or more specific, add a conditional somewhere in gemm that does a printf when matrix size is just 1x1 and put the breakpoint on that printf line) and trace that invocation back "up" to see if it an OpenBLAS function that split it up like that, or if the matrix was 1x1
all the way back to mumps ?