Welcome to ned
Productions (non-commercial personal website, for commercial company see ned Productions
Limited). Please choose an item you are interested in on the left
hand side, or continue down for Niall's virtual diary.
If you have any comments/questions/criticism of my virtual diary,
you can email me at the address at the bottom of the page.
For a deep, meaningful moment, watch this dialogue
(needs a video player) or for something which plays with your
perception, check out this picture. Try moving your eyes around - are
those circles rotating???
So, the parts for my replacement cloud server finally arrived - I had to order the Supermicro X10SL7-F motherboard from the US and import it to Ireland as the only places I could find it reasonably priced in Europe were in Germany, and none of those companies would deduct VAT for my company. The X10SL7-F is quite literally the only motherboard in its niche - high end Haswell home servers - so I had little choice in the selection unfortunately (or cost actually, but in fairness it's quarter the cost of other Xeon server solutions of a similar spec).
Anyway, I plugged it all together last night with 16Gb of ECC RAM and an Intel Xeon E3-1230 v3 CPU, fired it up and yeah, what's not to like - Aspeed KVM over IP integrated so I never need plug in a monitor or keyboard ever again (or physically hit the power or reset buttons :) ), two 10Gbit ethernet ports which saves me the expansion card I had before, and no less than fourteen SATA connectors with eight of those hanging from a LSI SAS2308 controller which is Intel VT-d virtualisable i.e. you can pass it through at full performance to a virtualised FreeNAS VM. And of course the thing is an absolute beast compared to the $35 Sandy Bridge Pentium G530 I had in there before, as well it should be given the price difference.
But then came an interesting problem: the cloud box runs Proxmox as the VM hypervisor, and that's a fairly ancient 2.6.32 Linux kernel which of course doesn't understand Haswell CPUs, so as dmesg reports it disables the intel_idle power management driver and falls back onto acpi_idle. Yet, this is what ACPI power states the kernel uses:
[Yes, Haswells add a C8 low power state, but you will need a suitable PSU capable of going from hundreds of watts to fractions of watts in microseconds, and mine is too old. Mine is new enough to support C7 though, and that draws only a watt, and rather handily Supermicro have let you choose if C7 is available in the BIOS rather than just disabling it outright for improved compatibilty with older PSUs]
Anyway, the curious thing above is that Linux's default ondemand CPU governor doesn't bother changing the clock speed of the CPU on Haswell because the C-states reported by ACPI above all say to run at the full 3.3Ghz - my big question was whether this means I have misconfigured the BIOS or something? Much trial and error later, I came to the conclusion that no I haven't. So will running this Haswell server with a kernel not capable of power saving result in a big electricity bill or not?
So, here are the empirical results for whole system power draw (which includes four 3Tb WD Red hard drives @ 4w each):
Well that certainly looks about right ... even with that SAS2308 controller adding 10w and 16w for the hard drives, that leaves 23.4w for the rest of the system, and the internet says it should draw no less than 21w. It looks pretty close to optimal, and that makes no sense!
Anyway, after a lot of head scratching I finally figured out the answer - it turns out that Haswell, unlike Ivy Bridge, does its own ondemand governor internally and doesn't need the OS to do it for it, so while the OS thinks it is running the CPU full belt, the CPU is actually doing its own power management. Witness the following thanks to the i7z tool (http://code.google.com/p/i7z/, you'll need to grab it from trunk for Haswell support):
Socket  - [physical cores=4, logical cores=8, max online cores ever=4] TURBO ENABLED on 4 Cores, Hyper Threading ON Max Frequency without considering Turbo 3390.73 MHz (99.73 x ) Max TURBO Multiplier (if Enabled) with 1/2/3/4 Cores is 37x/37x/36x/35x Real Current Frequency 3176.12 MHz [99.73 x 31.85] (Max of below) Core [core-id] :Actual Freq (Mult.) C0% Halt(C1)% C3 % C6 % C7 % Temp VCore Core 1 : 3176.12 (31.85x) 1 2.49 0 0 97.3 25 0.9999 Core 2 : 3063.24 (30.72x) 1 15.8 0 0 84.1 26 0.9974 Core 3 : 2825.56 (28.33x) 1 3.26 0 0 96.7 29 0.9974 Core 4 : 2645.90 (26.53x) 1 4.36 0 0 95.6 26 0.9974
C0 = Processor running without halting C1 = Processor running with halts (States >C0 are power saver modes with cores idling) C3 = Cores running with PLL turned off and core cache turned off C6, C7 = Everything in C3 + core state saved to last level cache, C7 is deeper than C6 Above values in table are in percentage over the last 1 sec [core-id] refers to core-id number in /proc/cpuinfo 'Garbage Values' message printed when garbage values are read Ctrl+C to exit
Note the reduced voltage (VCore) and that the clocks are all much lower. Let's try one thread of work:
Socket  - [physical cores=4, logical cores=8, max online cores ever=4] TURBO ENABLED on 4 Cores, Hyper Threading ON Max Frequency without considering Turbo 3390.73 MHz (99.73 x ) Max TURBO Multiplier (if Enabled) with 1/2/3/4 Cores is 37x/37x/36x/35x Real Current Frequency 3648.67 MHz [99.73 x 36.59] (Max of below) Core [core-id] :Actual Freq (Mult.) C0% Halt(C1)% C3 % C6 % C7 % Temp VCore Core 1 : 3643.07 (36.53x) 1 0.313 0 0 99.4 34 1.0438 Core 2 : 3609.90 (36.20x) 1 1.1 0 0 98.8 34 1.0461 Core 3 : 3648.67 (36.59x) 1 100 0 0 0 46 1.0414 Core 4 : 3607.95 (36.18x) 1 6.01 0 0 94 39 1.0413
Hmm, voltage and clock speed has jumped into turbo land, now it claims that one thread is always being C1 halted which is surely not true whilst everything else remains in C7. Let's try four threads:
Socket  - [physical cores=4, logical cores=8, max online cores ever=4] TURBO ENABLED on 4 Cores, Hyper Threading ON Max Frequency without considering Turbo 3390.73 MHz (99.73 x ) Max TURBO Multiplier (if Enabled) with 1/2/3/4 Cores is 37x/37x/36x/35x Real Current Frequency 3490.45 MHz [99.73 x 35.00] (Max of below) Core [core-id] :Actual Freq (Mult.) C0% Halt(C1)% C3 % C6 % C7 % Temp VCore Core 1 : 3490.45 (35.00x) 1 99.5 0 0 0 51 1.0194 Core 2 : 3490.45 (35.00x) 100 0 0 0 0 49 1.0219 Core 3 : 3489.02 (34.99x) 1 99.6 0 0 0 54 1.0219 Core 4 : 3488.69 (34.98x) 1 99.6 0 0 0 48 1.0219
Suddenly the C1 halted state when one core is 100% in use makes sense - it's the Hyperthreading, because a single thread at 100% isn't actually using the whole CPU from the CPU's perspective and is in fact halting. If you run eight threads, you get all four cores at C0, as to be expected.
So what should we learn from this? Well, I did quite a few experiments with manually forcing the clock speed down to 800Mhz, or using usermode CPU governors, or forcing kernel CPU governors - none made a jot of difference to idle power consumption except to raise it, plus they did stop the CPU from utilising turbo frequencies.
I therefore believe that it is actually the case that with Haswell doing no power consumption management at all is in fact ideal for this CPU, or at least until my Linux kernel gains the Intel p-states driver, and even then I understand that is more for reducing power consumption under load not idle.
So there you go eh? Who would have thought we've come full circle after ten years of active CPU power management!
Just finished upgrading all my servers with the fixed OpenSSL not having the "Heart Bleed" vuln (http://heartbleed.com/) which effectively makes encryption useless - that means all your encryption keys, passwords, identification, all of it is stealable. Next step (after coffee) is regenerating all my server encryption keys, and then paying $100 per SSL certificate to have the old ones revoked and new ones issued. What a complete bloody mess! :(
Wednesday 9th April 2014: 8.51am. Clara has been of the opinion that night time is play time last two nights in a row. So the tiredness I am feeling right now borders on the insane, especially as debugging a new threading primitive object for the Boost libraries (I'm contributing a permit object) is my current work item. Race condition finding is definitely best done after plenty of sleep!
Monday 7th April 2014: 2.45am. As you can tell by the very late hour, been burning the candle at both ends last two weekends. Just finished converting over my ancient workhorse Atom netbook which has served me so well since 2008 to that ex-corporate Dell I got off eBay some weeks ago - yes, it has taken me weeks to get round to the conversion :( The only other to do item I achieved was to start my Canadian tax return so I'd have some idea how much I owe them - it'll be more than a grand which sucks, due to BlackBerry claiming I got over 100k off them last year which makes no sense seeing as they fired me in August. So next weekend definitely need to go through my bank records and see if the numbers add up. The really big item I did NOT get round to is OU maths coursework - I'm now three weeks behind, and given my workload between now and the C++ Now conference next month (I am unofficially keynoting the final day, so I need an academic paper + semi coherent material to present), god knows where I'm going to find the time. Bleh! Sleep!
Monday 31st March 2014: 8.45am. Last night the cloud node died, and with it goes access to our TV, all my company files, in fact pretty much everything which makes for a very bored Megan. Data itself is fine, just inaccessible. Anyway got about four hours of sleep last night after trial and error diagnosis of what was hanging the machine so I know what replacements to buy. It's the motherboard south bridge, rest of the machine is fine. I guess that upgrade to a Xeon server comes now. Best price I can do it for is about €600 ex VAT. Tough price to pay when you have less than €2000 left :(
Finally got round to adding a clang 3.4 ThreadSanitize and UndefinedBehaviorSanitize pass per commit to proposed Boost.AFIO. The reason I put it off for so long was the labour of writing the suppressions file to filter out all the false positives generated by the Boost libraries - I really wish the maintainers of those Boost libraries would get round to fixing up their code with _attribute_((no_sanitize_thread)) where needed :( Anyway, tis done now, and if I should ever accidentally introduce racy code in the future, Travis CI will shout at me :)
Friday 28th March 2014: 3.14pm. Apparently baby Clara is now over 5kg at five weeks old! That puts her into the top 5% of all babies her weight and age!
Friday 28th March 2014: 9.23am. Feeling very groggy this morning. Was doing fifteen months worth of company accounts until 10pm last night. Which is like staying up to 3am in "have a baby" time, not helped that she has a bit of chestiness and freaks out when she can't breathe. Anyway another step closer to earning money again!
It's the weekend, so been trying to get through my list of todos, or at least the ones possible whilst holding a laptop above a sleeping newborn (ouch my wrists!). I got round to hooking up the house power monitor (an inexpensive Owl CM160) to my cloud node, and witness below the pretty web gui which is thanks to this open source project https://github.com/cornetp/eagle-owl. You can clearly tell when we got our stuff from Canada last week and the cloud node started being on 24/7: that doubled our baseload from 100w to 200w, costing me an extra €0.48 per day. Last Tuesday I went back to work and had to start sitting in my rather cold north facing office where I need to keep an electric heater on: daily electricity usage jumped from 8kWh per day to 14kWh which is costing me €1.20 extra per day! You can see this in the hourly chart for the 19th March - I clearly started work around 10am with the oven going on for dinner around 8pm. Anyway, the cloud will now log our house power usage every six seconds from now on - nice!
You can tell it's below freezing outside ... Clara didn't care much for our cold bedroom!
Thursday 20th March 2014: 5.09pm. Currently recompiling a large code base with -fsanitize=thread, and said code base's build system won't parallel build correctly. Which means a lot of waiting around which I'm trying to fill in with reading technical papers, which reminds me how I am back at work once again :(
Monday 17th March 2014: 2.09am. Now I have my cloud node back and once again fully functional after I replaced and resilvered a failing hard drive in my ZFS array, I let it loose on catching up with changes of the past three months i.e. let it go full bore at syncing data over this supposedly 70Mbit vDSL internet which is the fastest I have ever had into my house (the Canadian connection was 54Mbit, St. Andrews just 6.4Mbit, and Kerry Pike was and still is a truly lovely 0.5Mbit). To date, he peaks at 7.9Mb/sec, no more, and in fact oscillates between about 6.5Mb/sec and 7.9Mb/sec. In Mbit terms, that is between 52Mbit and 63Mbit, so we're not quite at the point of saturating the connection yet.
So what's the cause? It's actually the cloud node: its 2.4Ghz dual core Sandy Bridge CPU is maxed out by the Linux KVM doing the downloading and decompression on one core, and the FreeNAS KVM doing the checksumming for ZFS and writing to disc on the other core, so for the first time to date since I set up this cloud node in late 2011 the CPU has become the bottleneck limiting everything else.
The big surprise really is that it hasn't happened until now: that Sandy Bridge CPU cost me €32 new back when it came out - that's right, it was the cheapest possible CPU at the time, and I think it's done marvellously running three fat KVMs and five thin VMs for the years since. Unfortunately, technology has overtaken it, but it just goes to show how linear growth Moore's Law has become: a bottom end €32 CPU from late 2011 is more than capable of running Linux and FreeBSD virtualised and pushing 100Mbit of data through both (it's the ZFS which drops throughput to the 50Mbit range, ZFS is very hard on the CPU caches especially when virtualised as ZFS talks to ~6Gb of RAM each time slice).
I'm very strongly minded to upgrade it to a Haswell 1220v3 Xeon CPU with a C224 chipset server motherboard as soon as financially possible. It should lop 40% off idle power consumption, important when a kWh costs €0.19; it adds ECC memory which is all important for stability under cosmic rays and substantially reducing the bitrot ZFS repairs every two weeks; bumps the CPU up by a third in single core speed and doubles the number of CPUs. Such hardware should carry me to the 200Mbit "vectoring" vDSL of supposedly sometime later this year here in Ireland which is probably as good as home internet will get for some years to come, so it's a future proofed investment. There is also an added side benefit: I sorely need a test machine capable of Intel TSX memory transactions as I am currently deploying TSX-enabled code which I have no idea whether it is fast or not, and that bugs me.
Roll on some financial security, that's what I say!
I've just realised that Tony Benn died yesterday. Like pretty much well ... everyone ... I didn't agree with almost everything Tony Benn said and believed. He was sufficiently competent to have done a great deal of harm to Britain back when he was in power, and his actions nearly brought about a military coup against the British government by a desperate industrial elite, the first since the 17th century. However he was also sufficiently incompetent to be effective without being wise, in a sort of way which was oddly similar to Margaret Thatcher, only that her view won the majority of the people of that time while Benn's views never did. In the way only history does, as a result Thatcher was disgraced after just a decade, whereas Benn was widely respected and admired right up until his death, and probably most especially by those who completely disagreed with him which was pretty much everybody in the past thirty years. I'll put it another way: of all the people who Ali G (https://en.wikipedia.org/wiki/Ali_G) ever made a fool of publicly, the only one he ever apologised immediately afterwards was for Tony Benn with a photo complete with the newly made aware Tony who actually took it very well. As much as you thought he spoke mostly rot, you couldn't help but like him in a way I guess the vast majority also did e.g. fictional characters by Sacha Baron Cohen.
As a product of the same generation, I too am similarly biased. While Thatcher's death didn't particularly move me, Tony Benn's has moved me and for the life of me I am unsure why. Both were great people; both were basically the same sort of thing; both, unlike almost any modern politician, actually believed most of what they said and didn't give a crap if people got upset. Both had a worldview which held together logically, even if no one normal would actually like to live in such a world.
Both were also dinosaurs, and both were destructive rather than constructive people. Benn, unlike Thatcher, was still appearing as a knowledgeable if left/Benn biased commentator on global politics right up until his death - I remember him opining on Irish politics only a few years ago, and I thought him remarkably accurate right up until he started spewing his usual rhetoric.
Perhaps the reason that Tony Benn was so widely loved is exactly that reason: how could a person so remarkably in touch and with such remarkable understanding of the world, be so remarkably wrong? Except, of course, you always have this nasty sneaking suspicion that had history gone slightly differently, Thatcher would be Benn and Benn would be Thatcher. Perhaps our collectiveness fondness for Tony Benn was in fact a recognition that in moderation, his extremist views actually made a ton load of sense just as Thatcher in moderation did.
Saturday 15th March 2014: 3.58am. Just learned the hard way: when replacing a failing drive in ZFS, add the new drive as a third mirror before removing the failing drive. Me, I tried to be clever and tried to dd the old drive to the new in lvm outside the virtualised FreeBSD, but ZFS twigged I'd changed out the drive from beneath the virtualisation somehow and broke the mirror. Now ZFS is resilvering the mirror, but it's from a single copy so I'm quite screwed if some bit flips have happened between earlier today and now :( Note to self: seriously look into OVH Hubic (https://hubic.com/en/), 10Tb for €10/month would make these sorts of risks go away forever.
Friday 14th March 2014: 4.34pm. After three months spent being shipped from Canada to Ireland in some pretty cold weather with no doubt many hard jolts, ZFS found and repaired just 128Kb out of 2.93Tb of data, or 0.004%. As a single bit error is reported by ZFS as 4Kb, let us assume that was 32 bit flips, in which case that was an error rate of 1.242 x 10^-12, or about 1000 times more than I typically see in this configuration. Thank you very much ZFS!
Wednesday 12th March 2014: 10.26pm. Got my third lot of OU graph algorithms coursework submitted today, and only in the nick of time. Man, kinda feel a bit run off my feet recently ... in some ways, being woken up by Clara is great for productivity, but it feels like my todo list has swollen massively recently.
Our stuff turns up from Canada on Friday at long long last - we last saw it before Christmas! With that, I finally get back my cloud node and dev workstation, so I've been experimenting with ways of clearing that noxious smell from the front room of our rented house where I assume I'll be spending a lot of my next few months. I got the landlord to pay for a Rug Doctor on it already, and in fairness that machine did lift the worst of the smell along with an amazing amount of dirt, but it's still there enough to bother me. Right now I have a bowl of activated charcoal in there to see what happens, but if that fails I'm going to have to get an air purifier with at least a HEPA filter to suck the smell from the air in that room.
I may finally be working again soon - some piece work may be turning up, but until I see money you can never be sure. Next major task is surely my academic paper for the conference, it's only two months away now. Phew! Tired!
Sunday 9th March 2014: 1.39am. Unfortunately the one and only Irish company I was interviewing with rejected me on Friday. Still, first time I have ever made it to final stage interviews with any Irish employer - I guess Irish tech employers are finally waking up to the idea that open source developers actually might be comparable to corporate permies rather than being mere eccentric hobbyists with "no real experience", though they're not quite there yet obviously. I'm still taking it as a win!
During the week I received my "new" laptop from eBay imported from the US for a silly low price, that ex-corporate Dell E6410 I was talking about before, and I've been trying out system configs to see what I'll settle on as a system install for the next few years. I had really, really wanted my main install to be a dual boot Windows 8 and FreeBSD 10, mainly because FreeBSD 10 is the first really exciting FreeBSD in at least three years. Why? It's entirely built with clang 3.3 and so a bleeding edge C++11 compiler and STL come bundled without any faffing around, and you generally use ZFS as the root filing system nowadays, two key factors which are significant advantages over Kubuntu LTS which is my normal POSIX work system, especially as AFIO's eventual end goal is a transactional graph database which is particularly optimised for COW filing systems like ZFS or BTRFS. Even better, PC-BSD (PC-BSD is to FreeBSD as Kubuntu is to Linux) is now on FreeBSD 10, comes with latest KDE and a full binary package manager all on ZFS, and is remarkably competitive in features with Kubuntu 14.04 LTS which is scheduled for release in a few months. Some would even say that PC-BSD 10 is actually superior in features to Kubuntu 14.04 LTS!
I am in fact typing this from within PC-BSD 10, and I have to admit it was an absolute cinch to install and incredibly, everything just worked like NVidia graphics drivers or desktop effects or audio without hacking at it for hours like you normally have to with FreeBSD. Thanks to the ZFS and hardware graphics acceleration, the system just absolutely flies on this 2010 era laptop with its clunky old spinning hard drive - it probably won't quite match Kubuntu, but it's closer than any FreeBSD to a Linux than I've seen for many, many years (since before Ubuntu turned up actually).
The only real showstopper is the damn Wifi: FreeBSD has famously partial driver support, and really only what is installed on corporate servers is really well supported. Because corporate servers lag a few generations of technology behind, so generally does FreeBSD driver support - though, this 2010 Westmere era laptop is very well supported, but its peripherals are not so much. I can live without the 3G data card, but the lack of a working Wifi card is a problem. Mine is a Broadcom BCM4313, and getting it working without randomly hanging the system is sadly not possible.
Luckily, I have a latest generation 802.11ac wifi card on the way - mainly because I need the 5Ghz 11n support rather than 11ac itself, and that Broadcom card can't do 5Ghz. It's an Intel 7260 card which is surprisingly affordable for what it is, but being less than six months old FreeBSD support is non-existent. That said, the two main chipsets of Wifi card usually supported on FreeBSD are Atheros and Intel, so Intel 7xxx series wifi card support will turn up eventually, in fact one of the main BSD kernel developers is already on it.
The problem is, can I make do without Wifi until then?
Saturday 1st March 2014: 9.17pm. Just finished my OU coursework despite Clara being clamped onto my shoulder snoring. As much as this lack of sleep thing is a bore, I think so far productivity has increased. Next step: rehash my C++ Now conference presentation as its submission deadline is this week. Step after that: finish the monadic continuations prototype for the Boost C++ libraries and send it in for peer feedback before I finish it. Step after that: write academic paper for C++ Now. But by then I suspect I may have some work and finally will be supporting my family ...