Chapel: issue installing chapel (internal error: varying number of cpu cores)

[AlexanderKoshkarov]

I use standart gnu/linux x64 (void linux) and I am trying to install chapel 1.17.0,
following this link https://chapel-lang.org/docs/usingchapel/QUICKSTART.html

$ source util/setchplenv.bash #also tried source util/quickstart/setchplenv.bash
$ make
$ chpl -o hello examples/hello.chpl
$ ./hello
> internal error: varying number of cpu cores
(make check produces the same error)

I am not sure what am I doing wrong. The compilations seems to finish successfully.

[mppf]

Can you share the output of cat /proc/cpuinfo ?

[AlexanderKoshkarov]

```processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 158
model name : Intel(R) Xeon(R) CPU E3-1535M v6 @ 3.10GHz
stepping : 9
microcode : 0x70
cpu MHz : 823.489
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs : cpu_meltdown spectre_v1 spectre_v2
bogomips : 6192.00
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 158
model name : Intel(R) Xeon(R) CPU E3-1535M v6 @ 3.10GHz
stepping : 9
microcode : 0x70
cpu MHz : 832.918
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 1
cpu cores : 8
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs : cpu_meltdown spectre_v1 spectre_v2
bogomips : 6192.00
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 158
model name : Intel(R) Xeon(R) CPU E3-1535M v6 @ 3.10GHz
stepping : 9
microcode : 0x70
cpu MHz : 838.861
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 2
cpu cores : 8
apicid : 4
initial apicid : 4
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs : cpu_meltdown spectre_v1 spectre_v2
bogomips : 6192.00
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 158
model name : Intel(R) Xeon(R) CPU E3-1535M v6 @ 3.10GHz
stepping : 9
microcode : 0x70
cpu MHz : 811.355
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 3
cpu cores : 8
apicid : 6
initial apicid : 6
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs : cpu_meltdown spectre_v1 spectre_v2
bogomips : 6192.00
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 158
model name : Intel(R) Xeon(R) CPU E3-1535M v6 @ 3.10GHz
stepping : 9
microcode : 0x70
cpu MHz : 837.341
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs : cpu_meltdown spectre_v1 spectre_v2
bogomips : 6192.00
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 158
model name : Intel(R) Xeon(R) CPU E3-1535M v6 @ 3.10GHz
stepping : 9
microcode : 0x70
cpu MHz : 824.212
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 1
cpu cores : 3
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs : cpu_meltdown spectre_v1 spectre_v2
bogomips : 6192.00
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 158
model name : Intel(R) Xeon(R) CPU E3-1535M v6 @ 3.10GHz
stepping : 9
microcode : 0x70
cpu MHz : 831.346
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 2
cpu cores : 2
apicid : 5
initial apicid : 5
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs : cpu_meltdown spectre_v1 spectre_v2
bogomips : 6192.00
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 158
model name : Intel(R) Xeon(R) CPU E3-1535M v6 @ 3.10GHz
stepping : 9
microcode : 0x70
cpu MHz : 827.324
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 3
cpu cores : 1
apicid : 7
initial apicid : 7
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs : cpu_meltdown spectre_v1 spectre_v2
bogomips : 6192.00
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:
```

[mppf]

Can you also share your version of the linux kernel & what distribution and release you are using?

[AlexanderKoshkarov]

It is recent Void linux (rolling release) with glib

-> % uname -a
Linux koshlaptop 4.15.15_1 #1 SMP PREEMPT Wed Apr 4 08:48:06 UTC 2018 x86_64 GNU/Linux

p.s. This distribution is quite vanila GNU/Linux. The one major difference is the absence of systemd

[mppf]

@gbtitus - it looks like our code processing /proc/cpuinfo made an assumption that is wrong on the reporter's system.

[ben-albrecht]

https://github.com/buddha314/Charcoal/issues/6 might be related, where another user received this error message

[gbtitus]

@mppf right, and @ben-albrecht right. I'll take this for now, since I've already looked into it for the earlier instance of this same message.

@AlexanderKoshkarov it's not an issue with Linux as far as we know. It appears simply to be a limitation in what our runtime expects for the values in the 'cpu cores' lines of /proc/cpuinfo. The code expects those all to be the same, and now seemingly suddenly we are encountering systems where they are not.

[bradcray]

@gbtitus: Just troubleshooting out loud, is there any workaround a user like @AlexanderKoshkarov could use to work around this for the time being?

[gbtitus]

@bradcray, @AlexanderKoshkarov: no, I don't believe there is a workaround when /proc/cpuinfo has more than one value in the 'cpu cores' lines. It looks like all tasking layers hit the code in question. The GASNet comm layer does as well, but that's moot given the tasking layer situation.

[AlexanderKoshkarov]

To be honest, the nonconstant number of cpu cores does not make sense to me. I have a regular laptop with fixed number of physical cores (when I look at htop output - I see 8 cores all the time. 4 physical x2 for virtual cores).
Can it be related to some settings of new kernel? For me it still looks like linux related, probably because I do not know much about CPUs and linux.

[gbtitus]

@AlexanderKoshkarov agreed, it certainly looks to me like you should have a straightforward 4-core system with 2 hyperthreads per core. I also think it could be linux-related. The contents of the /proc/cpuinfo file are produced by the kernel. Can you put the result of uname -rv here?

You aren't running on some kind of hypervisor or a virtual machine hosted on this hardware, are you?

[mppf]

@gbtitus - I would think the comment https://github.com/chapel-lang/chapel/issues/9170#issuecomment-380594169 would cover any uname questions.

[gbtitus]

Oops, sorry, yes, it does.

[mppf]

I can't find any real documentation for /proc/cpuinfo (since it seems to be platform dependent). The closest thing I have to an answer is the kernel code here: https://github.com/torvalds/linux/blob/master/arch/x86/kernel/cpu/proc.c but I don't see anything there indicating that the booted_cores value can change. Perhaps the kernel is saving power by turning some cores off? Maybe acpitool can help to establish that?

[AlexanderKoshkarov]

Well, no "hypervisor or a virtual machine" - plain installation of void linux. Before that incident, I thought void linux is vanila distro.... now I do not know. Maybe there is a more robust way to determine number of cores? (maybe even if it depende on /proc/cpuinfo)
e.g.

-> % nproc --all
8

or

-> % lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               158
Model name:          Intel(R) Xeon(R) CPU E3-1535M v6 @ 3.10GHz
Stepping:            9
CPU MHz:             800.104
CPU max MHz:         4200.0000
CPU min MHz:         800.0000
BogoMIPS:            6192.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            8192K
NUMA node0 CPU(s):   0-7
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp

[gbtitus]

@AlexanderKoshkarov: there is PR to fix this now: #9188. If you could possibly try it out for me that would be great. Unfortunately I'm away until Monday now, however, so if you have any difficulties, addressing those will have to wait until then.

[AlexanderKoshkarov]

sure, I will try it out once the PR is closed. Thank you and do not worry about your absence - I am not in much hurry.

[mppf]

@AlexanderKoshkarov - I think @gbtitus was asking if you can try it out before he merges the PR, since we don't have a system that does this, so we don't know if it actually fixes the problem. I can help you do that but it'd go something like this:

git clone https://github.com/gbtitus/chapel.git
git checkout improve-cpuinfo-scanning
make
make check

[AlexanderKoshkarov]

It works :)

-> % source util/setchplenv.bash
-> % make check                 
[Info] Running minimal test script: $CHPL_HOME/util/test/checkChplInstall
[Info] Found executable chpl in /home/kosh/chapel/bin/linux64/chpl.
[Info] Found $CHPL_HOME directory: /home/kosh/chapel
[Info] /home/kosh/.chpl does not exist. Creating it.
[Info] Temporary test job directory: /home/kosh/.chpl/chapel-test-u71BO
[Info] Compiling $CHPL_HOME/test/release/examples/hello6-taskpar-dist.chpl
[Info] Test job compiled into /home/kosh/.chpl/chapel-test-u71BO/hello6-taskpar-dist
[Info] $CHPL_LAUNCHER=none is compatible with test script.
[Info] Running test job.
[Info] Test job complete.
SUCCESS: 'make check' passed!
-> % chpl -o hello examples/hello.chpl 
-> % ./hello 
Hello, world!

[mppf]

great, thanks!