Corefreq: CPU data missing on AMD Epyc 7451
Running on a dual CPU AMD Epyc 7451 machine, with SMT enabled, 96 cores. Data is shown for cores 0-63. Cores 64-96 show no data.
munorc
All 32 comments
Hello, thank you for trying _CoreFreq_
How long time do you give me to enhance the source code to handle more than 64 CPUs ?
This will require several tests from your side.
Edit: as a hint, could you post corefreq-cli -s -m
Regards
Cyril
cyring
on 14 Aug 2019
# ./corefreq-cli -s -m
Processor [AMD EPYC 7451 24-Core Processor ]
|- Architecture [EPYC/Naples]
|- Vendor ID [AuthenticAMD]
|- Microcode [ 134222375]
|- Signature [ 8F_01]
|- Stepping [ 2]
|- Online CPU [ 96/96]
|- Base Clock [ 99.81]
|- Frequency (MHz) Ratio
Min 399.25 < 4 >
Max 2295.70 < 23 >
|- Factory [100.00]
2300 [ 23 ]
|- Performance
|- OSPM
TGT 2295.70 < 23 >
|- Turbo Boost [UNLOCK]
XFR 3194.01 [ 32 ]
CPB 2894.58 [ 29 ]
1C 1796.63 < 18 >
2C 1197.76 < 12 >
|- Uncore [ LOCK]
Instruction Set Extensions
|- 3DNow!/Ext [N,N] ADX [Y] AES [Y] AVX/AVX2 [Y/Y]
|- AVX-512 [N] BMI1/BMI2 [Y/Y] CLFSH [Y] CMOV [Y]
|- CMPXCH8 [Y] CMPXCH16 [Y] F16C [Y] FPU [Y]
|- FXSR [Y] LAHF/SAHF [Y] MMX/Ext [Y/Y] MONITOR [Y]
|- MOVBE [Y] MPX [N] PCLMULDQ [Y] POPCNT [Y]
|- RDRAND [Y] RDSEED [Y] RDTSCP [Y] SEP [Y]
|- SGX [N] SSE [Y] SSE2 [Y] SSE3 [Y]
|- SSSE3 [Y] SSE4.1/4A [Y/Y] SSE4.2 [Y] SYSCALL [Y]
Features
|- 1 GB Pages Support 1GB-PAGES [Present]
|- 100 MHz multiplier Control 100MHzSteps [Missing]
|- Advanced Configuration & Power Interface ACPI [Present]
|- Advanced Programmable Interrupt Controller APIC [Present]
|- Core Multi-Processing CMP Legacy [Present]
|- L1 Data Cache Context ID CNXT-ID [Missing]
|- Direct Cache Access DCA [Missing]
|- Debugging Extension DE [Present]
|- Debug Store & Precise Event Based Sampling DS, PEBS [Missing]
|- CPL Qualified Debug Store DS-CPL [Missing]
|- 64-Bit Debug Store DTES64 [Missing]
|- Fast-String Operation Fast-Strings [Missing]
|- Fused Multiply Add FMA|FMA4 [Present]
|- Hardware Lock Elision HLE [Missing]
|- Long Mode 64 bits IA64|LM [Present]
|- LightWeight Profiling LWP [Missing]
|- Machine-Check Architecture MCA [Present]
|- Model Specific Registers MSR [Present]
|- Memory Type Range Registers MTRR [Present]
|- No-Execute Page Protection NX [Present]
|- OS-Enabled Ext. State Management OSXSAVE [Present]
|- Physical Address Extension PAE [Present]
|- Page Attribute Table PAT [Present]
|- Pending Break Enable PBE [Missing]
|- Process Context Identifiers PCID [Missing]
|- Perfmon and Debug Capability PDCM [Missing]
|- Page Global Enable PGE [Present]
|- Page Size Extension PSE [Present]
|- 36-bit Page Size Extension PSE36 [Present]
|- Processor Serial Number PSN [Missing]
|- Restricted Transactional Memory RTM [Missing]
|- Safer Mode Extensions SMX [Missing]
|- Self-Snoop SS [Missing]
|- Time Stamp Counter TSC [Invariant]
|- Time Stamp Counter Deadline TSC-DEADLINE [Missing]
|- Virtual Mode Extension VME [Present]
|- Virtual Machine Extensions VMX [Missing]
|- Extended xAPIC Support x2APIC [Missing]
|- XSAVE/XSTOR States XSAVE [Present]
|- xTPR Update Control xTPR [Missing]
Technologies
|- System Management Mode SMM-Lock [ ON]
|- Simultaneous Multithreading SMT [ ON]
|- PowerNow! CnQ [OFF]
|- Core Performance Boost CPB < ON>
|- Virtualization SVM [OFF]
|- I/O MMU AMD-V [OFF]
|- Hypervisor [OFF]
Performance Monitoring
|- Version PM [ 0]
|- Counters: General Fixed
| 6 x 64 bits 3 x 64 bits
|- Enhanced Halt State C1E <OFF>
|- Core C6 State CC6 < ON>
|- Package C6 State PC6 <OFF>
|- Frequency ID control FID [ ON]
|- Voltage ID control VID [ ON]
|- P-State Hardware Coordination Feedback MPERF/APERF [ ON]
|- Hardware-Controlled Performance States HWP <OFF>
|- Capabilities (MHz) Ratio
Lowest nan [ 0 ]
Efficient nan [ 0 ]
Guaranteed nan [ 0 ]
Highest nan [ 0 ]
|- Hardware Duty Cycling HDC [OFF]
|- Package C-State
|- Configuration Control CONFIG [ LOCK]
|- Lowest C-State LIMIT [ 0]
|- I/O MWAIT Redirection IOMWAIT [Disable]
|- Max C-State Inclusion RANGE [ 0]
|- MONITOR/MWAIT
|- State index: #0 #1 #2 #3 #4 #5 #6 #7
|- Sub C-State: 1 1 0 0 0 0 0 0
|- Core Cycles [Present]
|- Instructions Retired [Present]
|- Reference Cycles [Present]
|- Last Level Cache References [Missing]
|- Last Level Cache Misses [Missing]
|- Branch Instructions Retired [Missing]
|- Branch Mispredicts Retired [Missing]
Power & Thermal
|- Clock Modulation ODCM [Disable]
|- DutyCycle [ 0.00%]
|- Power Management PWR MGMT [ LOCK]
|- Energy Policy Bias Hint [ 0]
|- Energy Policy HWP EPP [ 0]
|- Junction Temperature TjMax [ 0: 0]
|- Digital Thermal Sensor DTS [Present]
|- Power Limit Notification PLN [Missing]
|- Package Thermal Management PTM [Missing]
|- Thermal Monitor 1 TTP [Present]
|- Thermal Monitor 2 HTC [Present]
|- Units
|- Power watt [ 0.125000000]
|- Energy joule [ 0.000015259]
|- Window second [ 0.000976562]
CPU Pkg Apic Core Thread Caches (w)rite-Back (i)nclusive
# ID ID CCX ID ID L1-Inst Way L1-Data Way L2 Way L3 Way
00: BSP 0 0 0 0 64 4 32 8 512 8 i 32768 64w
01: 0 2 0 1 0 64 4 32 8 512 8 i 32768 64w
02: 0 4 0 2 0 64 4 32 8 512 8 i 32768 64w
03: 0 8 1 4 0 64 4 32 8 512 8 i 32768 64w
04: 0 10 1 5 0 64 4 32 8 512 8 i 32768 64w
05: 0 12 1 6 0 64 4 32 8 512 8 i 32768 64w
06: 1 16 0 8 0 64 4 32 8 512 8 i 32768 64w
07: 1 18 0 9 0 64 4 32 8 512 8 i 32768 64w
08: 1 20 0 10 0 64 4 32 8 512 8 i 32768 64w
09: 1 24 1 12 0 64 4 32 8 512 8 i 32768 64w
10: 1 26 1 13 0 64 4 32 8 512 8 i 32768 64w
11: 1 28 1 14 0 64 4 32 8 512 8 i 32768 64w
12: 2 32 0 16 0 64 4 32 8 512 8 i 32768 64w
13: 2 34 0 17 0 64 4 32 8 512 8 i 32768 64w
14: 2 36 0 18 0 64 4 32 8 512 8 i 32768 64w
15: 2 40 1 20 0 64 4 32 8 512 8 i 32768 64w
16: 2 42 1 21 0 64 4 32 8 512 8 i 32768 64w
17: 2 44 1 22 0 64 4 32 8 512 8 i 32768 64w
18: 3 48 0 24 0 64 4 32 8 512 8 i 32768 64w
19: 3 50 0 25 0 64 4 32 8 512 8 i 32768 64w
20: 3 52 0 26 0 64 4 32 8 512 8 i 32768 64w
21: 3 56 1 28 0 64 4 32 8 512 8 i 32768 64w
22: 3 58 1 29 0 64 4 32 8 512 8 i 32768 64w
23: 3 60 1 30 0 64 4 32 8 512 8 i 32768 64w
24: 4 64 0 0 0 64 4 32 8 512 8 i 32768 64w
25: 4 66 0 1 0 64 4 32 8 512 8 i 32768 64w
26: 4 68 0 2 0 64 4 32 8 512 8 i 32768 64w
27: 4 72 1 4 0 64 4 32 8 512 8 i 32768 64w
28: 4 74 1 5 0 64 4 32 8 512 8 i 32768 64w
29: 4 76 1 6 0 64 4 32 8 512 8 i 32768 64w
30: 5 80 0 8 0 64 4 32 8 512 8 i 32768 64w
31: 5 82 0 9 0 64 4 32 8 512 8 i 32768 64w
32: 5 84 0 10 0 64 4 32 8 512 8 i 32768 64w
33: 5 88 1 12 0 64 4 32 8 512 8 i 32768 64w
34: 5 90 1 13 0 64 4 32 8 512 8 i 32768 64w
35: 5 92 1 14 0 64 4 32 8 512 8 i 32768 64w
36: 6 96 0 16 0 64 4 32 8 512 8 i 32768 64w
37: 6 98 0 17 0 64 4 32 8 512 8 i 32768 64w
38: 6 100 0 18 0 64 4 32 8 512 8 i 32768 64w
39: 6 104 1 20 0 64 4 32 8 512 8 i 32768 64w
40: 6 106 1 21 0 64 4 32 8 512 8 i 32768 64w
41: 6 108 1 22 0 64 4 32 8 512 8 i 32768 64w
42: 7 112 0 24 0 64 4 32 8 512 8 i 32768 64w
43: 7 114 0 25 0 64 4 32 8 512 8 i 32768 64w
44: 7 116 0 26 0 64 4 32 8 512 8 i 32768 64w
45: 7 120 1 28 0 64 4 32 8 512 8 i 32768 64w
46: 7 122 1 29 0 64 4 32 8 512 8 i 32768 64w
47: 7 124 1 30 0 64 4 32 8 512 8 i 32768 64w
48: 0 1 0 0 1 64 4 32 8 512 8 i 32768 64w
49: 0 3 0 1 1 64 4 32 8 512 8 i 32768 64w
50: 0 5 0 2 1 64 4 32 8 512 8 i 32768 64w
51: 0 9 1 4 1 64 4 32 8 512 8 i 32768 64w
52: 0 11 1 5 1 64 4 32 8 512 8 i 32768 64w
53: 0 13 1 6 1 64 4 32 8 512 8 i 32768 64w
54: 1 17 0 8 1 64 4 32 8 512 8 i 32768 64w
55: 1 19 0 9 1 64 4 32 8 512 8 i 32768 64w
56: 1 21 0 10 1 64 4 32 8 512 8 i 32768 64w
57: 1 25 1 12 1 64 4 32 8 512 8 i 32768 64w
58: 1 27 1 13 1 64 4 32 8 512 8 i 32768 64w
59: 1 29 1 14 1 64 4 32 8 512 8 i 32768 64w
60: 2 33 0 16 1 64 4 32 8 512 8 i 32768 64w
61: 2 35 0 17 1 64 4 32 8 512 8 i 32768 64w
62: 2 37 0 18 1 64 4 32 8 512 8 i 32768 64w
63: 2 41 1 20 1 64 4 32 8 512 8 i 32768 64w
64: 2 43 1 21 1 64 4 32 8 512 8 i 32768 64w
65: 2 45 1 22 1 64 4 32 8 512 8 i 32768 64w
66: 3 49 0 24 1 64 4 32 8 512 8 i 32768 64w
67: 3 51 0 25 1 64 4 32 8 512 8 i 32768 64w
68: 3 53 0 26 1 64 4 32 8 512 8 i 32768 64w
69: 3 57 1 28 1 64 4 32 8 512 8 i 32768 64w
70: 3 59 1 29 1 64 4 32 8 512 8 i 32768 64w
71: 3 61 1 30 1 64 4 32 8 512 8 i 32768 64w
72: 4 65 0 0 1 64 4 32 8 512 8 i 32768 64w
73: 4 67 0 1 1 64 4 32 8 512 8 i 32768 64w
74: 4 69 0 2 1 64 4 32 8 512 8 i 32768 64w
75: 4 73 1 4 1 64 4 32 8 512 8 i 32768 64w
76: 4 75 1 5 1 64 4 32 8 512 8 i 32768 64w
77: 4 77 1 6 1 64 4 32 8 512 8 i 32768 64w
78: 5 81 0 8 1 64 4 32 8 512 8 i 32768 64w
79: 5 83 0 9 1 64 4 32 8 512 8 i 32768 64w
80: 5 85 0 10 1 64 4 32 8 512 8 i 32768 64w
81: 5 89 1 12 1 64 4 32 8 512 8 i 32768 64w
82: 5 91 1 13 1 64 4 32 8 512 8 i 32768 64w
83: 5 93 1 14 1 64 4 32 8 512 8 i 32768 64w
84: 6 97 0 16 1 64 4 32 8 512 8 i 32768 64w
85: 6 99 0 17 1 64 4 32 8 512 8 i 32768 64w
86: 6 101 0 18 1 64 4 32 8 512 8 i 32768 64w
87: 6 105 1 20 1 64 4 32 8 512 8 i 32768 64w
88: 6 107 1 21 1 64 4 32 8 512 8 i 32768 64w
89: 6 109 1 22 1 64 4 32 8 512 8 i 32768 64w
90: 7 113 0 24 1 64 4 32 8 512 8 i 32768 64w
91: 7 115 0 25 1 64 4 32 8 512 8 i 32768 64w
92: 7 117 0 26 1 64 4 32 8 512 8 i 32768 64w
93: 7 121 1 28 1 64 4 32 8 512 8 i 32768 64w
94: 7 123 1 29 1 64 4 32 8 512 8 i 32768 64w
95: 7 125 1 30 1 64 4 32 8 512 8 i 32768 64w
The place where data is missing is in the various windows that are updated e.g. Dashboard.
munorc
on 14 Aug 2019
Thank you.
- The Processor data appear OK, please confirm if they match BIOS ?
- The topology offers enough distinct values per CPU.
Fyi, the Package ID is not the Socket as expected with Intel, but the Node ID of Zen
https://github.com/cyring/CoreFreq/blob/e21a6d5480fff6a095edec7b6ea01310ad25c0e9/corefreqk.c#L1352
Before I jump into code refactoring I have to clarify this questions:
What's the max Core count per Socket in the whole x86 architectures range ?
- if less or equal 64 then I can _instantiate_ the API structures per Socket
- if higher than 64 then I preserve the API but I have to refactor any asm atomic operations
cyring
on 14 Aug 2019
Meanwhile, _CoreFreq_ will run fine with your EPYC if you disable SMT to lmit to 48 CPUs
cyring
on 14 Aug 2019
EPYC (Naples) has a max of 32 cores/64 threads per package. The 7451 is 24 core, 48 threads).
EPYC 2 (Rome) has a max of 64 cores /128 threads per package. We should have those in place soon as well (256 threads per server node with SMT enabled).
munorc
on 14 Aug 2019
Huge impacts, please contact me on labs[at]cyring[dot]fr
cyring
on 14 Aug 2019
Hello,
Does this code returns 1 with your EPYC ?
/*
CMPXCHG16B.c
-- Build & Run --
cc CMPXCHG16B.c -o CMPXCHG16B
./CMPXCHG16B; echo $?
*/
#define CMPXCHG16B( _dest, _src ) \
({ \
register unsigned char _ret = 0; \
\
__asm__ volatile \
( \
"movq %[losrc], %%rax" "\n\t" \
"movq %[hisrc], %%rdx" "\n\t" \
"movq %%rax, %%rbx" "\n\t" \
"movq %%rdx, %%rcx" "\n\t" \
"lock rex cmpxchg16b %[dest]" "\n\t" \
"setz %[ret]" \
: [dest] "=m" (_dest), [ret] "+r" (_ret) \
: [losrc] "m" (_src[0]), [hisrc] "m" (_src[1]) \
: "cc", "memory", "%rax", "%rbx", "%rcx", "%rdx" \
); \
\
_ret; \
})
int main(int argc, char *argv[])
{
unsigned long long src[2] __attribute__ ((aligned (128))) = {
0b1111101010101010101010101010101010101010101010101010101010101010,
0b1111111111111111111111111111111100000000000000000000000000000001
}, dest[2] __attribute__ ((aligned (128))) = {
0b1111101010101010101010101010101010101010101010101010101010101010,
0b1111111111111111111111111111111100000000000000000000000000000001
};
unsigned char ret = CMPXCHG16B(src, dest);
return((int) ret);
}
It appears to.
$ cc CMPXCHG16B.c -o CMPXCHG16B
$ ./CMPXCHG16B; echo $?
1
munorc
on 16 Aug 2019
Great, we have our first 128 bits primitive for a 128 CPUs bitmap.
Unfortunately, I still have found no instruction to handle 256 CPUs.
And any FPU extension that can do with 256 bits registers, such as AVX2, is not supported by the Kernel.
Edit : above code fixed
cyring
on 16 Aug 2019
I made mt128.c to launch 128 threads, one per SMT Core. The main thread will join them using a 128 bits room bitmap. After 1000 loops, main returns the remaining counter. It has to echo zero.
I have try to test it in a 128 x CPU VM but it is too slow and couldn't complete. You should have a better result with your hardware.
Btw, the VM shows that CoreFreq can currently display the tasks of 128 CPUs.
cyring
on 17 Aug 2019
mt128 returns 0
munorc
on 17 Aug 2019
Hello,
Can you try this version. It focuses on driver and daemon back-ends only to handle up to 128 Cores.
UI has not been refactored yet, thus you won't read good CPU numbers.
Please provides screenshot of the Client with the last 72 Cores. I would check if we read frequencies for all CPUs.
cyring
on 17 Aug 2019
New optimized development version for testings.
CoreFreq.tar.gz
cyring
on 18 Aug 2019
Which screen are you looking for a a screenshot of? Most data is being displayed as "inf"
munorc
on 18 Aug 2019
default startup screen, showing last CPU rows
cyring
on 18 Aug 2019
munorc
on 18 Aug 2019
Is it virtualized or bare-metal ?
cyring
on 18 Aug 2019
bare metal
munorc
on 18 Aug 2019
Has the module been unload b/c between the 2 versions the API has changed.
cyring
on 18 Aug 2019
Development revision 99
Source code:
CoreFreq.tar.gzDriver & Daemon resized to 256 CPUs
Client, beside Dashboard, UI impacts fixed
What it may look like
cyring
on 18 Aug 2019
adatum
on 18 Aug 2019
@adatum :1st_place_medal: Thank you. Apparently no regression
Now waiting for EPYC results and the high Core Count properly support ?
@munorc: can you test w/ kernel 4 minimum ?
cyring
on 18 Aug 2019
Development revision 97
(_remark: development revisions in descending order_)
Source code:
CoreFreq_163_97.tar.gz- Try to improve the atomic operations to support 256 Cores
- Fixed several memory alignements
- UI: fixed CPUs scrolling
cyring
on 23 Aug 2019
Development revision 96
- Source code:
CoreFreq_163_96.tar.gz
- Daemon: restored bit synchronization of threads
- Driver: fixed OnLine CPU count
cyring
on 23 Aug 2019
Hello,
The 256 SMT-Cores Scaling version is released.
Thank you for your tests.
cyring
on 24 Aug 2019
Just tested on the 2x AMD Epyc 7451 (Naples). I see data for all 96 cores.
These are running CentOS 7.5 so kernel is 3.10.0-862.
munorc
on 24 Aug 2019
Hello,
Can you please try the stress tests of the last CoreFreq version.
If Conics, especially, crash your EPYC processor ?
In the issue #131 , Threadripper hard locks with Conics
cyring
on 16 Sep 2019
Hello,
I've a request to fix the topology with dual EPYC. Can you help ?
cyring
on 22 Oct 2019
Hello,
Among the last changes in issue #195 , an I/O ASM access to the FCH have been added: can you please check if the last _CoreFreq_ version is starting fine ?
Thank you
cyring
on 6 Jul 2020
Hello,
The develop branch brings the UMC timings.
https://github.com/cyring/CoreFreq/issues/196#issuecomment-678599938
Could you please give a try with EPYC and tell if it matches BIOS settings ?
Feel free to add screenshots or Cli output of the _CoreFreq_ Memory Controller
Thank you
cyring
on 22 Aug 2020
Hello,
Can you help with an issue about the inconsistency of the instructions counter with Zen 1st generation architecture.
Based on the latest _CoreFreq_ version, show this view while EPYC is idling:
Regards
CyrIng
cyring
on 29 Oct 2020
Related issues
cyring
路
18Comments
Chlorophytus
路
26Comments
olejon
路
16Comments
vincenzogianfelice
路
17Comments
Arthav24
路
26Comments