Corefreq: 2x Intel Xeon EC5509 SLBWM

[cyring]

@svmlegacy Thank you very much for adding those results.

Can you please post the output of lspci -nn ?
I will add the PCI device id of the IMC to decode the DRAM data.

Regards,
CyrIng

[svmlegacy]

Xeon EC5509 lspci -nn

[cyring]

We have to experiment with Jasper Forest because I don't find any meaningful datasheet of its registers.
It is based on Nehlalem micro-architecture and according to its CPUID signature, in the class of Lynnfield.

• Can you edit the header file corefreqk.h and replace the structure PCI_Nehalem_DMI_ids with the one bellow.

https://github.com/cyring/CoreFreq/blob/0e8f292b6cce0ff00aeccb9e0654ec0be852e9db/corefreqk.h#L1641

static struct pci_device_id PCI_Nehalem_DMI_ids[] = {
    {   /* Lynnfield IMC                    */
        PCI_VDEVICE(INTEL, DID_INTEL_LYNNFIELD_MCR),
        .driver_data = (kernel_ulong_t) Lynnfield_IMC
    },
    {   /* Lynnfield IMC Test Registers             */
        PCI_VDEVICE(INTEL, DID_INTEL_LYNNFIELD_MC_TEST),
        .driver_data = (kernel_ulong_t) NHM_IMC_TR
    },
    { /* Lynnfield QuickPath Architecture Generic Non-core Registers */
        PCI_VDEVICE(INTEL, DID_INTEL_LYNNFIELD_NON_CORE),
        .driver_data = (kernel_ulong_t) NHM_NON_CORE
    },
    { /* Clarksfield Processor Uncore Device 0, Function 0      */
        PCI_VDEVICE(INTEL, DID_INTEL_CLARKSFIELD_NON_CORE),
        .driver_data = (kernel_ulong_t) NHM_NON_CORE
    },
    { /* Westmere/Clarkdale QuickPath Architecture Non-core Registers */
        PCI_VDEVICE(INTEL, DID_INTEL_CLARKDALE_NON_CORE),
        .driver_data = (kernel_ulong_t) NHM_NON_CORE
    },
    {   /* Jasper Forest IMC                    */
        PCI_VDEVICE(INTEL, 0x2cd8),
        .driver_data = (kernel_ulong_t) Lynnfield_IMC
    },
    {0, }
};

Fyi: The change is about adding the PCI device id [8086:2cd8] and invoke the existing Lynnfield_IMC decoding function (which should be called twice because there are two EC5509)

• EDIT: in the Daemon source code, change also this statement to add 0x2cd8
  https://github.com/cyring/CoreFreq/blob/0e8f292b6cce0ff00aeccb9e0654ec0be852e9db/corefreqd.c#L4884
  ... with:

    case DID_INTEL_LYNNFIELD_MCR:       /*  Lynnfield   */
    case 0x2cd8:
        NHM_IMC(Shm, Proc);
        SET_CHIPSET(IC_IBEXPEAK);
        break;

• Fully rebuild _CoreFreq_
  make clean all

• Save your files and start _CoreFreq_

• Check if we now have more data in the IMC part of the CLI ? like the channel count.

[svmlegacy]

Seems to have made some difference, but still not pulling in more data with ./corefreq-cli -M

I believe these CPU's to be very similar to Lynnfield, but have both DMI and QPI. QPI is used only for CPU -> CPU communication.

Unlike Lynnfield, these have a triple channel memory controller. (Though I believe Lynnfield physically has three controllers.)

I've attached my modified files if you have any questions on them.

2x Intel Xeon EC5509 IMC Test 1

[cyring]

• SLBWM listed in Table 9. Processor Identification of Intel® Xeon® Processor C5500/C3500 Series Specification Update

• Looking for C5500/C3500 Series Datasheet, Volume 2 [323317]
  

[cyring]

Hello,

Commit dbc6ebcfad9cfad11047da17e775806c70a41470 provides the IMC for Nehalem C5500/C3500 Series

Once pulled, don't forget to fully rebuild the source code:

make clean all

IMC, DRAM, VT-d are expected from this change. Two controllers should be counted.

[svmlegacy]

Results here

Both IMC's are now appearing, with the DRAM information only appearing on Controller # 1.

VT-d now shows "ON", correct for my configuration.

[cyring]

Results here

Both IMC's are now appearing, with the DRAM information only appearing on Controller # 1.

VT-d now shows "ON", correct for my configuration.

Not so bad for a first try.

Dual controllers case is not easy to solve. I finally found datasheet volume 2 in which are mentioned two discriminant registers: Primary and Secondary Bus numbers. Those numbers should help in establishing a _hardware_ topology for channel and DIMM queries.
So far I'm relying on kernel bus numbering 0xfe and 0xff as listed in previous lspci report.

[cyring]

If you're interested in, send me an email to my address shown in corefreq-cli -h and I will send you back the two datasheet volumes of your processor family.
Inside, there are registers of interest which need to be dump and study.

[cyring]

Hello,

In commit a6e7bca06b300b79b10b2ffe29a69a823d15f918 you will get a new IMC numbering.
All Nehalem architectures are impacted by this change.

Thanks for any test.

[svmlegacy]

Commit https://github.com/cyring/CoreFreq/commit/a6e7bca06b300b79b10b2ffe29a69a823d15f918 and onward returns the controller to outputting nothing on Jasper Forest:

$ ./corefreq-cli -M
                           P55/Ibex Peak  [2CD8]                           
Controller #0                                                              
 Bus Rate  2500 MT/s      Bus Speed 2493 MT/s          DRAM Speed 1063 MHz 

 Cha    CL  RCD   RP  RAS  RRD  RFC   WR RTPr WTPr  FAW  B2B  CWL CMD  REFI
      ddWR drWR srWR ddRW drRW srRW ddRR drRR srRR ddWW drWW srWW CKE   ECC

[cyring]

Commit a6e7bca and onward returns the controller to outputting nothing on Jasper Forest:

$ ./corefreq-cli -M
                           P55/Ibex Peak  [2CD8]                           
Controller #0                                                              
 Bus Rate  2500 MT/s      Bus Speed 2493 MT/s          DRAM Speed 1063 MHz 

 Cha    CL  RCD   RP  RAS  RRD  RFC   WR RTPr WTPr  FAW  B2B  CWL CMD  REFI
      ddWR drWR srWR ddRW drRW srRW ddRR drRR srRR ddWW drWW srWW CKE   ECC

Can you please try with latest commit c135b2c00f7bb58ccd3d7d48c76bb28cec8ae16c

[svmlegacy]

Can you please try with latest commit c135b2c

Commit c135b2c edited Lynnfield_IMC, and not Nehalem_IMC which is being called by the C5500_C3500_IMC definition, thus had no effect.

I was able to get some output by forcing mc = 0 in corefreqk.c, as follows:

static PCI_CALLBACK Nehalem_IMC(struct pci_dev *dev)
{ /* Arrandale; Beckton; Bloomfield; Clarkdale; Eagleton; Gainestown; Gulftown*/
    kernel_ulong_t rc;
    const unsigned char bus_number = 0xff - dev->bus->number;
    const unsigned short mc = (unsigned short) bus_number % MC_MAX_CTRL;

    if (mc >= PUBLIC(RO(Proc))->Uncore.CtrlCount) {
        PUBLIC(RO(Proc))->Uncore.CtrlCount++;
    }
    /* rc = Query_NHM_IMC(dev, mc); */
    rc = Query_NHM_IMC(dev, 0); 

    return ((PCI_CALLBACK) rc);
}
```` 

Of course, this only output controller # 0, as follows:

$ ./corefreq-cli -M
P55/Ibex Peak [2CD8]
Controller #0 Triple Channel
Bus Rate 2500 MT/s Bus Speed 2493 MT/s DRAM Speed 1063 MHz

Cha CL RCD RP RAS RRD RFC WR RTPr WTPr FAW B2B CWL CMD REFI
#0 7 7 7 20 4 59 8 6 19 20 0 6 1T 509
#1 7 7 7 20 4 59 8 6 19 20 0 6 1T 509
#2 7 7 7 20 4 59 8 6 19 20 0 6 1T 509
ddWR drWR srWR ddRW drRW srRW ddRR drRR srRR ddWW drWW srWW CKE ECC
#0 6 6 14 9 9 9 7 6 4 7 7 4 3 1
#1 6 6 14 9 9 9 7 6 4 7 7 4 3 1
#2 6 6 14 9 9 9 7 6 4 7 7 4 3 1

DIMM Geometry for channel #0
Slot Bank Rank Rows Columns Memory Size (MB)
#0 8 1 16384 1024 1024
DIMM Geometry for channel #1
Slot Bank Rank Rows Columns Memory Size (MB)
#0 8 1 16384 1024 1024
DIMM Geometry for channel #2
Slot Bank Rank Rows Columns Memory Size (MB)
#0 8 1 16384 1024 1024
```

I'll poke around a little more, and see if I can get the 2nd controller up.

[cyring]

Can you please try with latest commit c135b2c

Commit c135b2c edited Lynnfield_IMC, and not Nehalem_IMC which is being called by the C5500_C3500_IMC definition, thus had no effect.

I was able to get some output by forcing mc = 0 in corefreqk.c, as follows:

static PCI_CALLBACK Nehalem_IMC(struct pci_dev *dev)
{ /* Arrandale; Beckton; Bloomfield; Clarkdale; Eagleton; Gainestown; Gulftown*/
  kernel_ulong_t rc;
  const unsigned char bus_number = 0xff - dev->bus->number;
  const unsigned short mc = (unsigned short) bus_number % MC_MAX_CTRL;

  if (mc >= PUBLIC(RO(Proc))->Uncore.CtrlCount) {
      PUBLIC(RO(Proc))->Uncore.CtrlCount++;
  }
  /* rc = Query_NHM_IMC(dev, mc); */
  rc = Query_NHM_IMC(dev, 0); 

  return ((PCI_CALLBACK) rc);
}

Of course, this only output controller # 0, as follows:

$ ./corefreq-cli -M
                           P55/Ibex Peak  [2CD8]                           
Controller #0                                               Triple Channel 
 Bus Rate  2500 MT/s      Bus Speed 2493 MT/s          DRAM Speed 1063 MHz 

 Cha    CL  RCD   RP  RAS  RRD  RFC   WR RTPr WTPr  FAW  B2B  CWL CMD  REFI
  #0     7    7    7   20    4   59    8    6   19   20    0    6  1T   509
  #1     7    7    7   20    4   59    8    6   19   20    0    6  1T   509
  #2     7    7    7   20    4   59    8    6   19   20    0    6  1T   509
      ddWR drWR srWR ddRW drRW srRW ddRR drRR srRR ddWW drWW srWW CKE   ECC
  #0     6    6   14    9    9    9    7    6    4    7    7    4   3    1 
  #1     6    6   14    9    9    9    7    6    4    7    7    4   3    1 
  #2     6    6   14    9    9    9    7    6    4    7    7    4   3    1 

 DIMM Geometry for channel #0                                              
      Slot Bank Rank     Rows   Columns    Memory Size (MB)                
       #0     8    1     16384      1024           1024                    
 DIMM Geometry for channel #1                                              
      Slot Bank Rank     Rows   Columns    Memory Size (MB)                
       #0     8    1     16384      1024           1024                    
 DIMM Geometry for channel #2                                              
      Slot Bank Rank     Rows   Columns    Memory Size (MB)                
       #0     8    1     16384      1024           1024                    

I'll poke around a little more, and see if I can get the 2nd controller up.

Oh nice help. It is hard to imagine debug from here, but indeed the purpose is to iterate mc from 0 to N with 0 as the first controller.

I'm expecting to compute:
Constant minus bus_number

0xff - 0xff = 0 first IMC
0xff - 0xfe = 1 second IMC

But things depend on bus numbering as listed by lspci

[cyring]

...
I'll poke around a little more, and see if I can get the 2nd controller up.

Hello,

Are you getting something ?

[svmlegacy]

Hello,

Are you getting something ?

I'm expecting to compute:
Constant minus bus_number

0xff - 0xff = 0 first IMC
0xff - 0xfe = 1 second IMC

But things depend on bus numbering as listed by lspci

The IMC's are reliably appearing at 0xFF.03 for the first CPU and 0xFE.03 for the 2nd CPU, based on the lspci results.

Haven't had much time to test lately, but still familiarizing myself with the code to see if I can find where this is breaking down.

[svmlegacy]

Is it possible that the Uncore_Update function is running out of PCI devices before querying the 2nd IMC?

I am seeing that NHM_IMC is only being run once when # ./corefreqd is started.
Shouldn't this function be called for each IMC?

I tried bumping up the CHIP_MAX_PCI value from 24, but the daemon doesn't load with values greater than 25.
lspci is showing a total of 44 devices under bus's 0xFF and 0xFE.

Additionally, this issue is also affecting my DP Westmere-EP server. (Xeon X5670's)

[cyring]

Is it possible that the Uncore_Update function is running out of PCI devices before querying the 2nd IMC?

I am seeing that NHM_IMC is only being run once when # ./corefreqd is started.
Shouldn't this function be called for each IMC?

I tried bumping up the CHIP_MAX_PCI value from 24, but the daemon doesn't load with values greater than 25.
lspci is showing a total of 44 devices under bus's 0xFF and 0xFE.

Additionally, this issue is also affecting my DP Westmere-EP server. (Xeon X5670's)

This could explain the issue ! Or be part of it.

At the time of first implementation I didn't have such a _big_ workstation to decide the IDs array dimension.

However, Daemon should be called once for multiple controllers and iterates from the first to the last, aggregating for each IMC, their channels and DIMM data.
Thus Driver has to prepare properly the whole multi-dim structure in prior.

I'm reviewing this part of code...

[cyring]

@svmlegacy

Only the probed Device DID in list will be added in array by the driver. So the whole lspci won't be scan.

Printing the whole structure could help to debug where the values have been stored (or not), depending on bus numbering.

In this function:
https://github.com/cyring/CoreFreq/blob/2200d1dea99c1c95623166e539e319929190fcdf/corefreqd.c#L2778

For exemple, the DIMM presence register.
Dump from 0 to max IMC, to max Channels, to max DIMMs:

for (mc = 0; mc < MC_MAX_CTRL; mc++)
    {
for (cha = 0; cha < MC_MAX_CHA; cha++)
      {
for (slot = 0; slot < MC_MAX_DIMM; slot++)
       {
        printf ("%d\n", Proc->Uncore.MC[mc].Channel[cha].DIMM[slot].DOD.DIMMPRESENT);
       }
     }
   }