Niall’s virtual diary archives – Thursday 10th April 2014

by . Last updated . This page has been accessed 42 times since the 26th March 2019.

Thursday 10th April 2014: 7.47pm. Link shared: http://code.google.com/p/i7z

So, the parts for my replacement cloud server finally arrived - I had to order the Supermicro X10SL7-F motherboard from the US and import it to Ireland as the only places I could find it reasonably priced in Europe were in Germany, and none of those companies would deduct VAT for my company. The X10SL7-F is quite literally the only motherboard in its niche - high end Haswell home servers - so I had little choice in the selection unfortunately (or cost actually, but in fairness it's quarter the cost of other Xeon server solutions of a similar spec).

Anyway, I plugged it all together last night with 16Gb of ECC RAM and an Intel Xeon E3-1230 v3 CPU, fired it up and yeah, what's not to like - Aspeed KVM over IP integrated so I never need plug in a monitor or keyboard ever again (or physically hit the power or reset buttons :) ), two 10Gbit ethernet ports which saves me the expansion card I had before, and no less than fourteen SATA connectors with eight of those hanging from a LSI SAS2308 controller which is Intel VT-d virtualisable i.e. you can pass it through at full performance to a virtualised FreeNAS VM. And of course the thing is an absolute beast compared to the $35 Sandy Bridge Pentium G530 I had in there before, as well it should be given the price difference.

But then came an interesting problem: the cloud box runs Proxmox as the VM hypervisor, and that's a fairly ancient 2.6.32 Linux kernel which of course doesn't understand Haswell CPUs, so as dmesg reports it disables the intel_idle power management driver and falls back onto acpi_idle. Yet, this is what ACPI power states the kernel uses:

ed@milla:~$ cat /proc/acpi/processor/CPU0/power
active state:            C0
max_cstate:              C8
maximum allowed latency: 2000000000 usec
states:
    C1:                  type[C1] promotion[--] demotion[--] latency[001] usage[00494647] duration[00000000000000000000]
    C2:                  type[C2] promotion[--] demotion[--] latency[148] usage[01379614] duration[00000000064459801669]

[Yes, Haswells add a C8 low power state, but you will need a suitable PSU capable of going from hundreds of watts to fractions of watts in microseconds, and mine is too old. Mine is new enough to support C7 though, and that draws only a watt, and rather handily Supermicro have let you choose if C7 is available in the BIOS rather than just disabling it outright for improved compatibilty with older PSUs]

Anyway, the curious thing above is that Linux's default ondemand CPU governor doesn't bother changing the clock speed of the CPU on Haswell because the C-states reported by ACPI above all say to run at the full 3.3Ghz - my big question was whether this means I have misconfigured the BIOS or something? Much trial and error later, I came to the conclusion that no I haven't. So will running this Haswell server with a kernel not capable of power saving result in a big electricity bill or not?

So, here are the empirical results for whole system power draw (which includes four 3Tb WD Red hard drives @ 4w each):

Idle: 49.4w
1x 'cat /dev/urandom > /dev/null &': 66.4w
2x 'cat /dev/urandom > /dev/null &': 75.4w
3x 'cat /dev/urandom > /dev/null &': 82.9w
4x 'cat /dev/urandom > /dev/null &': 87.5w
8x 'cat /dev/urandom > /dev/null &': 91.0w

Well that certainly looks about right ... even with that SAS2308 controller adding 10w and 16w for the hard drives, that leaves 23.4w for the rest of the system, and the internet says it should draw no less than 21w. It looks pretty close to optimal, and that makes no sense!

Anyway, after a lot of head scratching I finally figured out the answer - it turns out that Haswell, unlike Ivy Bridge, does its own ondemand governor internally and doesn't need the OS to do it for it, so while the OS thinks it is running the CPU full belt, the CPU is actually doing its own power management. Witness the following thanks to the i7z tool (http://code.google.com/p/i7z/, you'll need to grab it from trunk for Haswell support):


Socket [0] - [physical cores=4, logical cores=8, max online cores ever=4]
  TURBO ENABLED on 4 Cores, Hyper Threading ON
  Max Frequency without considering Turbo 3390.73 MHz (99.73 x [34])
  Max TURBO Multiplier (if Enabled) with 1/2/3/4 Cores is  37x/37x/36x/35x
  Real Current Frequency 3176.12 MHz [99.73 x 31.85] (Max of below)
        Core [core-id]  :Actual Freq (Mult.)      C0%   Halt(C1)%  C3 %   C6 %   C7 %  Temp      VCore
        Core 1 [0]:       3176.12 (31.85x)         1    2.49       0       0    97.3    25      0.9999
        Core 2 [1]:       3063.24 (30.72x)         1    15.8       0       0    84.1    26      0.9974
        Core 3 [2]:       2825.56 (28.33x)         1    3.26       0       0    96.7    29      0.9974
        Core 4 [3]:       2645.90 (26.53x)         1    4.36       0       0    95.6    26      0.9974

C0 = Processor running without halting
C1 = Processor running with halts (States >C0 are power saver modes with cores idling)
C3 = Cores running with PLL turned off and core cache turned off
C6, C7 = Everything in C3 + core state saved to last level cache, C7 is deeper than C6
  Above values in table are in percentage over the last 1 sec
[core-id] refers to core-id number in /proc/cpuinfo
'Garbage Values' message printed when garbage values are read
  Ctrl+C to exit


Note the reduced voltage (VCore) and that the clocks are all much lower. Let's try one thread of work:


Socket [0] - [physical cores=4, logical cores=8, max online cores ever=4]
  TURBO ENABLED on 4 Cores, Hyper Threading ON
  Max Frequency without considering Turbo 3390.73 MHz (99.73 x [34])
  Max TURBO Multiplier (if Enabled) with 1/2/3/4 Cores is  37x/37x/36x/35x
  Real Current Frequency 3648.67 MHz [99.73 x 36.59] (Max of below)
        Core [core-id]  :Actual Freq (Mult.)      C0%   Halt(C1)%  C3 %   C6 %   C7 %  Temp      VCore
        Core 1 [0]:       3643.07 (36.53x)         1    0.313      0       0    99.4    34      1.0438
        Core 2 [1]:       3609.90 (36.20x)         1     1.1       0       0    98.8    34      1.0461
        Core 3 [2]:       3648.67 (36.59x)         1     100       0       0       0    46      1.0414
        Core 4 [3]:       3607.95 (36.18x)         1    6.01       0       0      94    39      1.0413


Hmm, voltage and clock speed has jumped into turbo land, now it claims that one thread is always being C1 halted which is surely not true whilst everything else remains in C7. Let's try four threads:


Socket [0] - [physical cores=4, logical cores=8, max online cores ever=4]
  TURBO ENABLED on 4 Cores, Hyper Threading ON
  Max Frequency without considering Turbo 3390.73 MHz (99.73 x [34])
  Max TURBO Multiplier (if Enabled) with 1/2/3/4 Cores is  37x/37x/36x/35x
  Real Current Frequency 3490.45 MHz [99.73 x 35.00] (Max of below)
        Core [core-id]  :Actual Freq (Mult.)      C0%   Halt(C1)%  C3 %   C6 %   C7 %  Temp      VCore
        Core 1 [0]:       3490.45 (35.00x)         1    99.5       0       0       0    51      1.0194
        Core 2 [1]:       3490.45 (35.00x)       100       0       0       0       0    49      1.0219
        Core 3 [2]:       3489.02 (34.99x)         1    99.6       0       0       0    54      1.0219
        Core 4 [3]:       3488.69 (34.98x)         1    99.6       0       0       0    48      1.0219


Suddenly the C1 halted state when one core is 100% in use makes sense - it's the Hyperthreading, because a single thread at 100% isn't actually using the whole CPU from the CPU's perspective and is in fact halting. If you run eight threads, you get all four cores at C0, as to be expected.

So what should we learn from this? Well, I did quite a few experiments with manually forcing the clock speed down to 800Mhz, or using usermode CPU governors, or forcing kernel CPU governors - none made a jot of difference to idle power consumption except to raise it, plus they did stop the CPU from utilising turbo frequencies.

I therefore believe that it is actually the case that with Haswell doing no power consumption management at all is in fact ideal for this CPU, or at least until my Linux kernel gains the Intel p-states driver, and even then I understand that is more for reducing power consumption under load not idle.

So there you go eh? Who would have thought we've come full circle after ten years of active CPU power management!


#haswell #powermanagement #intel #server #xeon  

Go back to the archive index Go back to the latest entries

Contact the webmaster: Niall Douglas @ webmaster2<at symbol>nedprod.com (Last updated: 2014-04-10 19:47:18 +0000 UTC)