Welcome to ned Productions

Monday 18th February 2013: 6.00pm. This is the first weekend where after fixing the lack of copy & paste in my BEurtle program which has been a problem since July, my todo list has finally became empty for now. The next big ticket item is going to be buying and insuring a car, currently scheduled for April/May as we won't have the cash until then, and we're going to have to pay in cash because no one will lend to us given our lack of credit history in Canada. In fact, I had to bribe a bank to give me a credit card by handing them $5000 as a security after two prior credit card applications had failed. As much as Canada welcomes immigrants more than most countries, the lack of maturity in its financial services industry is a pain (they could after all pull your history from Europe). If we weren't earning enough to be able to save to buy a car for cash outright, I'm not sure what we'd do. Still, it's nice to be finally through that todo list and to take a break from spending every spare moment outside work crossing off todo items.

As you might have guessed from the lack of any articles here about RIM (now BlackBerry), it isn't allowed to talk about anything to do with work publicly, not even anything which is public knowledge. In fact, I can talk much more freely about what our competitors are up to than I can about us, which kinda sucks, but there you go. So, sorry about the lack of articles - I did write three of them, and they were very interesting I think. But it's a no-go.

So, what's happened these last three months? Surprisingly little. We left our BlackBerry-provided temporary accommodation at the start of December, and we now live about fifteen minutes' walk from the University of Waterloo just beyond which is RIM 1, the original BlackBerry office, where I work. So, come the summer I'll be able to walk to work quite pleasantly, though right now in the typical -15C temperatures I must admit to taking the bus to the university and walking ten minutes from there because it's slow going through all the snow and ice. I have learned that snow, when blown by a strong wind, is quite like sand at these temperatures and one can arrive at work with the tears from your eyes dripping having frozen on your cheeks. I also have to admit that I can now see the point of thermal long johns, especially for under your jeans which are relatively unprotected from the chill of a wind. Luckily, so far I've been fairly fortunate in Waterloo where usually the wind isn't more than a light breeze, and I'm actually plenty warm enough outside if the air remains still even if it's snowing heavily.

I wasn't with my family in Cork for the first time ever in my life this Christmas: instead myself and Megan drove down to her parents in Indiana with a rental, a trip taking about ten hours. I can't say I much enjoyed being away from my family, but we spoke by telephone on Christmas day and Megan's family made things better than they would have been otherwise.

Myself and Megan stopped off in Detroit for two nights on the way back to break up the trip, and as a mini-holiday. We did enjoy ourselves, but I think it was despite Detroit rather than anything else, and for the $1200 or so it cost it really wasn't worth it. The hotel, a Hilton, was really not great. The casino we visited, while upmarket, was seedy and depressing, and the $350 we dropped on the meal in the quite pretty top floor restaurant there (shown above) was too expensive for the very limited choice of menu available. It didn't help that there were several murders and rapes on the local news while we were there, and the city as a whole is very obviously deprived and its centre looks like something from those Hollywood movies you see and you think they're dramatising how run down and full of the drug addled poor US cities are - well, judging from city centre Detroit, they aren't (and my sister said very similar things about Los Angeles which she visited during her honeymoon). About the only highlight of our mini-holiday in Detroit really was Detroit Zoo which was better than expected. So, I'm afraid very much a thumbs down to Detroit for vacationing. Thankfully, next Christmas there are plenty of other US-Canada border places like Sarnia, or even Niagra Falls or Buffalo. If Detroit cut its cost of visiting by about a half, it might just be worth visiting as a once off, but otherwise that city has a long way to go in dealing with its decline. Right now it's just not attractive to be in, not helped by the expense to reward ratio.

So, in January I went back to work after the break ... I had spent December implementing a demonstration prototype of some proposed functionality for future BlackBerry phones, and I spent most of January writing a 20,000 word internal white paper on the technology. I finally upgraded my 2007 era PC which had served so well to something fairly cutting edge, and sure it is somewhat snappier than it was even when typing in this HTML, so I am pleased for the money spent, which wasn't much (about $400) as I recycled old parts from former uses. And of course, I turned thirty-five, and you'll note the lack of my usual annual reflection upon the past year of my life: Why? When I think upon this past year, it's very hard to say anything definitive about it. In Christmas 2011 a year ago I decided we were leaving Ireland for wherever in the world would give us a decent chance, which wasn't Ireland and wasn't Britain (their current anti-immigrant obsession makes it very hard for Megan to get a work visa even if she has Masters degrees out the wazoo). So I made the appropriate calls, did the appropriate interviews, made the appropriate choices and enacted the appropriate arrangements including the appropriate goodbyes. And that's 2012: I can't say much about it past that, and I certainly can't reflect upon it usefully at the present time. Maybe I will be able to in a year's time once things have settled.

At the end of January an admin position opening appeared in the Research Institute I'm attached to, namely the Institute for Complexity and Innovation in the University of Waterloo,, so we got Megan into applying for that and she got the job, so now she works for the University which is a great help in so many ways, not least that the Canadian tax system greatly rewards both people in a couple working so her modest additional income has a disproportionate effect on our after-tax income (I think our net marginal rate will actually drop). I also applied to join the executive board for a small local community grant disbursement fund who try to invest money into (currently) youth innovation and leadership (which would eventually lead to stuff like the Nanolight, a startup out of Toronto who raised $200k via Kickstarter), and to which I was accepted which is great as not only does it let me donate my usual 2.5% of charitable time and money per year, it also plugs me into Canada's tech startup community, so I'll have a line on what's going on just outside RIM (most tech startups in this innovation cluster are started by disaffected former RIM employees). Indeed, many of the fund's board members also sit on the local TTEDx Waterloo organising board, so things are looking good on that.

And I think that is about that to report. Not a lot really. The weeks pass, and everything seems on track so far. Our only real constraint now is the lack of a car, as the public transport here is effectively limited to the area surrounding the university which is fine for work and groceries, but useless if say you want to buy a bookcase because all those stores are accessible by car only. And we've also worked through all the nearby restaurants now, so as much as dining is indeed very good if somewhat pricey here, it gets much better with a car. Even simple things like going to cinema are a pain without a car: buses stop early here, so going to a cinema needs to be done in the afternoon which is tricky when you catch up on sleep at weekends, and you have certain things which need doing like chores before you can relax. Anyway, we'll get that solved soon in the next few months. Maybe I'll even post here before then? Anything could happen ... till next time, be happy!

Saturday 27th October 2012: 2.50pm. My first entry from our new life in Canada! Looking back over the past three months, it is striking how both expensive and stressful moving continents is, requiring about €15,000 in temporary bridging money and a good €5000 in non-retrievable costs, and that was with RIM paying directly for about half the total cost and them reimbursing us about €3,000 of expenses. Now, admittedly, we didn't need all of that €15,000 at all - perhaps about €10,000, but I deliberately added on 50% before I began to cover any unexpected surprises. Which I'll come to just shortly.

After the last entry, things of course got hectic. I went to visit about 60% of those I'm still in contact with from Cambridge, Hull, St. Andrews and beyond, and that of course involved a lot of travel and a lot of restaurant meals and drinking. Simultaneously, I had to organise the move itself which involved never ending emails between myself and various others employed by RIM to supposedly make our move easier. I have to admit, looking back, that while they made the bridging cost outlay much lower because they spent RIM's money rather than us having to fork out now for reimbursement later, they did not make things massively easier. Call me a control freak, but I didn't appreciate the extra layer of people between me and our relocation plan, especially when they point blank refused to do what they were told or in some cases, did something different to what they were supposed to without telling me. That added stress, because I had to rejig our schedule to cope with the uncertainty of people not getting back to me with necessary information for scheduling or not doing their jobs in a timely fashion. It wasn't helpful. No wonder I started smoking cigarettes again for the first time in years!

Still, it was great to see everybody and to say goodbye properly, even with the unbelievably tight scheduling constraints (after all, who moves continent and visits most of those they know in the world and starts a new job in exactly one calendar month!). I know from experience we won't see most of those from St. Andrews in particular ever again, and it was especially good to draw a solid line under the St. Andrews experience, four years after graduation. It closed things off especially well as I went to St. Andrews in 2004 at the age of 26, and those whom I knew there who were 18 are now aged exactly 26 as well, so they are now at exactly where I was at back when they first met me. And that did generate quite a lot of reflection: it came up frequently with each who I visited the loneliness and pointlessness of existence that graduates feel post-graduation, because you've had a few years post-uni to do something with your life and you find every worthy avenue closed off to you due to one overriding factor: society doesn't give a shit about you or your opinions or what you want or what is right. You get one binary, dimorphic choice: either join the capitalist machine working some unimportant, probably self-destructive job, have just enough time for acquaintances rather than friends, work till you get sick and we'll give you just enough money to have children and live an okay life, or else choose to be irrelevant and get nothing and be nothing. That realisation, one which I promulgated to 18 year olds who didn't quite believe me back during 2004-2008, is now as stark and as real and as depressing as any trauma a person can experience in a lifetime. The huge and only question remains: now that I understand this, what do I do now?

And of course, I don't have the answer there either, not like I had when they were 18 and answers were so much simpler. I returned to university aged 26 to try something different with the explicit goal of making possible a series of opportunities not normally available to those aged 26. Having mostly succeeded, I then had to pick from those opportunities, which I have done, and hopefully I haven't screwed up too badly. That's "the" answer, and I have no idea if I'm right yet. I guess we'll see. I do agree with my fellow St. Andrews graduates though that life after university isn't much fun, especially if you didn't graduate with a numerate degree. Even with a numerate degree, the drudgery, loneliness and pointlessness of the self-destructive rat race wears you down. Without a numerate degree, add to that constant worries about lack of money and lack of security and most especially, lack of self-worth because good jobs are especially hard to find when society is ambivalent to you about your value to society. There you go: that One Dimensional Man by Marcuse you studied at St. Andrews was actually true. And now it's the sole choice of life they give you. Welcome to adulthood.

So, moving on from depressing truths of our reality, Megan finished up her work at the end of August and almost straight after we went to Canada for our five day house hunting trip (paid for by RIM) and for me to meet my team for the first time, where it turned out that both my manager and his manager had left RIM and my team had been broken up and reassigned (more later on this, probably the next entry). That was obviously pretty intense, but the most useful result from my perspective was a Canadian bank account to which money to keep us alive could be sent, if we could figure out the SWIFT/IBAN to Canadian bank routing (I ended up sending multiple €100 wire transfers each using different methods, and those that got through were the "right" ones). We also sorted out our immigration and work visas during this trip, which I'd definitely recommend to anyone to sort out as soon as possible. Once back in Ireland, we did a whistle stop tour of Britain to say goodbye to Megan's friends, and went to Sweden to say goodbye to Johanna who was the only one out of everyone to get two of our days. This wasn't quite as expensive as my tour of my friends, but it was certainly as tiring. One thing which was interesting was that I took only my Android 4.1 phone with me and left my laptop at home, and other than battery life it performed surprisingly admirably as a general purpose computing device. Put another way: it let me manage the never ending email stream organising our move to Canada, and the Maps navigation saved our skins a number of times like when the taxi driver got lost in Manchester.

Back in Ireland for the last time at the end of September and with just days to go, the removers paid for by RIM came and took away all our stuff. By this stage I was quite fraught, as mistakes made here would deeply negatively impact the first two months of our time in Canada due to missing vital items (the stuff moved by removers takes about two months to get delivered, so you must separately send important stuff like clothes and cloud nodes by courier to get it there in a week). Hence racking your brain to make sure important things you'd urgently need in Canada weren't being removed. Sleep during this time came more from exhaustion than mind and body resting - not a lot of fun. I also had unexpected problems with moving a chunk of money - I had sent over our Canada living money via three separate parallel means to spread the risk of problems, and indeed the day before departing one of those means had got stuck so about half our money was in limbo. Better than all of it, sure, but also without that half I couldn't pay our deposits on accommodation, so not helpful. I remember being very tense indeed on our last day in Ireland, finding myself regularly snapping at people. Not how you ideally want to say goodbye to friends and family.

So, the day of emigration came, and in a weird way for me at least that was the day I could finally relax because from now on, nothing more could be done. Where you were at is where you are at, and what comes will come. After some difficulty locating our limo (paid for by RIM), we made it to our temporary flat in city centre Waterloo for the first two months (also paid for by RIM) in which I am currently typing this diary entry. So, basically, it had all worked out.

Having just arrived in Waterloo, I - despite being exhausted physically, mentally and spiritually - was in a strangely celebratory mood, so I wanted to go out on the town which we did despite it being extremely noisy and packed because the university term had just begun, so everywhere in the city centre was packed with drunk students (Waterloo has two of Canada's largest universities in it) and loud music. Unfortunately, I in my hazy, strange state misjudged a kerb and put a 90 degree kink in my left ankle which put an abrupt end to the festivities, and indeed even now one month later it's still painful. At the time, despite having plenty of cash, there was no obvious source of medical treatment - we weren't registered with the Canadian health system yet, and we were advised that doctors in Canada aren't allowed to privately accept cash for treatment so our only option was the hospital and spending the entire day waiting in A&E when probably it wasn't broken, and just needed time and ice packs and rest to heal. So much, in the end, of bringing plenty of spare cash for emergencies! Unfortunately, the following week involved a great deal of walking - to register for a monthly bus pass for which we had to go to Kitchener; to pick up my new BlackBerry work phone running BB10; to pick up a SIM for said phone which required twenty minutes of walking; to register for my security card and so on. My ankle, braced and compressed to hold in the swelling, and me dosed up on lots of codeine and anti-inflammatories for the pain, didn't exactly get the rest you're supposed to give a badly sprained joint. But, we make do with the hand we're dealt.

So that brings us to the past three weeks which mostly have consisted of us going on a mostly vegetarian diet to begin to shed the weight we put on during our goodbye tour, Megan spending the day at home alone or out registering for things like doctors while I'm at work, me weaning myself off cigarettes, off pain killers, off alcohol and finding and getting started at a gym again which was just this past week. On the 17th I took my OU M208 Pure Maths exam in London, Ontario for which I took the train between Kitchener and London - and the train system in Canada, which while better than in the US, is a pale shadow of even the worst and most run down train system anywhere in Europe with one of the roughest and bumpiest rides I've certainly ever endured, and in coaches which while they are clean, maintained and comfortable, clearly date from the early 1980s in terms of decor. Interestingly, trains in Canada are run by a single crown corporation (state owned company) called VIA from Quebec, and unsurprisingly the train system the Quebec side of Toronto is passably European. Unfortunately, I'm west of Toronto, and decades of underinvestment and political footballing show through - though it's far worse once you go west out of Ontario where 1940s era tracks just cannot remotely compete with air travel.

And this weekend is the first since we arrived that I wasn't passed out recovering from the week made worse by that inevitable fluey sickness you get after an extended emotional strain - indeed today I woke up at a reasonable hour feeling reasonably good. I also don't have to be going out to buy electricals (as the voltage is different here, we had to leave all our electricals behind in Ireland which has required repurchasing lots of simple things like printers) or get things we urgently needed like our own telephone line and local number (run over VoIP), or a semi-decent coffee in the morning as Canadians, like Americans, don't really do decent espresso based coffee, even in dedicated coffee shops, so I dropped $500 on a second hand bean grinder and a second hand espresso machine off eBay and now our coffee isn't sour, bland muck. It's actually amazing what you can do with average beans using good equipment - our morning coffee using some locally purchased beans is actually pretty good, though it won't be anywhere near the same league as the Jamaican Blue Mountain I dropped $55 a bag on last week. No, this is my first proper, non-busy weekend. Hence the diary entry!

Now, what's coming next to this diary is a reasonably long entry about my first month working for RIM. For the first time ever for a diary entry here, I'm going to have to get approval from RIM for that diary entry because it's going to talk about a raft of internal RIM stuff and I'm not sure how much of it they're going to be happy being made public. Still, I want to document and write down what I'm thinking at the end of this first month, because I think it'll be valuable to me personally later on but also because it'll be valuable to RIM and moreover, particularly valuable to those watching RIM as there's an awful lot of ignorance out there, and I'll be blunt in saying that RIM have done a poor job in communicating themselves recently and that's going to become a problem for the Q1 2013 BB10 launch. In particular, I would like to talk about what BB10 is and especially what it isn't, what RIM has done internally to itself this past year or two, and if I'm allowed I'd like to talk a bit about BB11 too as that's mainly what I was hired for. In short, I'd like to give my unvarnished impressions of RIM acquired during the past month and people can take them or leave them as they see fit.

Ok so, time for the rest of my weekend - oh how so valuable they are when you work non-stop all week! Until next time, be happy!

Wednesday 8th August 2012: 5.40pm. I'm glad to say that every item on my summer todo list from the last entry has been completed or nearly completed - unfortunately, it is at the expense of being so very tired and not thinking properly. This morning I forgot my passport for the first time since 1997 for my flight to Belgium to say goodbye to Natasja - and this being post 9/11, they wouldn't let me fly, so that was €190 down the tube and much disappointment caused for all. The tiredness resulted from being up till 2.30am replying to essential email organising the relocation to Canada, my next trip to see people around London, booking flights for same despite various people continuing to be vague about their availability, and setting a burn in test running on my shiny new 3Tb hard drive which is going to act as a separately transited ZFS pool mirror for all our important data during the move to Canada (not helped by the Sandforce SSD in my main desktop deciding to die for the third time yesterday, once again hosing my Windows install. This morning I ordered a 256Gb Samsung 830 SSD at some expense, and vowed to never, ever again use any Sandforce based SSD).

Why didn't I start all that earlier? Well, Sarah was visiting to say goodbye for the preceding four days, and with Megan working yesterday I took them to Kinsale for the day. In short, I couldn't have started anything earlier even though I kept dipping out of activities with Megan and Sarah to grab an hour or two sitting in the car with my laptop over a 3G link to attend to essential communiqués etc. (and hard drive burn in tests take several days). All in all, outta time in almost every endeavour, and that results in Niall putting up mental resistance to adding on more stress by doing subconscious blocks like forgetting passports. Unhelpful. The truth is though that the extra two days will be exceptionally useful - I now have enough days to get in my final OU Pure Maths coursework on time, and organise the removers who of course want contacting when I'm busy and won't respond when I contact them e.g. they rang when I was grabbing some very much needed sleep this afternoon. Still, I regret deeply missing Belgium. It's always fun visiting there.

You may have noticed that I upgraded the CSS (for those non-ancient pages on nedprod which use CSS) to use some newer features. There are more rounded corners than before, particularly on the navigation pane separator; hovering over links produces a fire effect I learned from developing the Deeper Economics website; and I used some CSS3 selectors to apply box shadowing to any standalone i.e. centred images which I think works really well. I ended up doing this as part of modularising the RSS feed floating pane so it can appear on the homepage of my software libraries with a running commit feed. Good stuff.

Other changes still to come include a proper SSL certificate for nedprod, and with that I can turn on the SPDY fast HTTP extension for nedprod to improve still further page load times. I rented a verified personal identity SSL certificate from StartSSL for a fairly reasonable US$60 for two years. Basically what this does is to say that someone called Niall Douglas residing in a given locality in a given country has provided a minimal amount of proof that they do have that name and do reside at the address they supplied. You can then attach this "proof" as a digital signature to your email, your website and so on under the theory that it makes it somewhat harder for another to impersonate you. Now, I'm not bothered about anyone impersonating me, rather I bothered with renting this because recent Windows throw up a warning if you try to install unsigned programs, and this includes the v1.50 alpha 1 release of BEurtle.. Future releases of BEurtle will now be properly signed rather than self-signed, and therefore not raise a warning. Obviously I also get the advantages of signed email, my email program not complaining every time I fetch mail etc. as well.

Last bit of news: I passed that damn PGCert in Educational and Social Research with the Institute of Education in London - in fact, I think I'll get a merit if I've calculated my weighted average correctly. Glad to be away from there - it was an eye opener. And very glad that some £3,200 of my hard earned money was not wasted.

Ok, so here's kinda why I'm writing a virtual diary entry now rather than later - here's me thinking out loud about how I'm going to configure my ZFS storage pool on the basis that this may help others. What I've got is my Proxmox cloud node running a copy of Ubuntu Server 12.04 LTS in a KVM virtual machine. Into that I've installed the Linux kernel port of ZFS v28 by the Lawrence Livermore laboratories (who are part of the US government, and they have huge data needs which is why they ported ZFS to Linux so I definitely trust the port) which is literally as easy as just adding a PPA to Ubuntu. I have three main storage hard drives which originate from an external USB hard drive solution which was originally called an "Icy Box":

1. IcyBox1: A Jan 2008 1Tb 3-platter Samsung Spinpoint F1 HD103UJ drive (1,000,204,886,016 bytes, 512 byte sectors, 6.7w/19.6w idle/start). This was my first solution to a stack of DVDs several feet high which had been growing since I studied at Hull University, and I remember having much fun carrying all 25kg of them in a backpack through the Madrid metro when I left Spain for London. I remember many students at St. Andrews marvelling that so much data could fit into a single drive, and in fairness so did I at that time. I also remember being fairly appalled during the load of the DVDs onto that drive that some of the older DVDs, especially those written at Hull and some of those in Madrid, had become unreadable despite much loving care and that arduous trek through the Madrid metro among other occasions of physical data transfer. Basically, DVD±R is a bad way to store long term data.
2. IcyBox2: A May 2010 2Tb 3-platter Western Digital Caviar Green WD20EARS-00MVWB0 "load/unlock click of death" drive (2,000,398,934,016 bytes, 4096 byte sectors, 2.9w/14w idle/start). This was purchased just as the Samsung drive approached capacity, and just after the 3-platter Green design came onto the market.
3. In test now: A June 2012 3Tb 3-platter Western Digital Red WD30EFRX-68AX9N0 drive (3,000,592,982,016 bytes, 4096 byte sectors, 3.84w/13.71w idle/start). Unlike the earlier drives, this I bought about six months too early in order to secure our data for Canada. Once again, this is one of the very first 3-platter 3Tb drive designs on the market, and the first I've owned to contain fancy enterprise style vibration gyroscopes.

As you can tell, I really don't like more than three platters for long term storage . I have about 1.7Tb of data in total, and currently the 1Tb drive holds what I had back when I had just 1Tb of data. Apart from that, I don't have any redundancy though the 2Tb drive is only ever plugged in when it's time to back up data - thankfully, as a result, it has a low load cycle count and won't die from the infamous "load/unload click of death" bug in those Caviar Green drives. Equally, I do need to flash a new firmware for that drive as it has the original "I can't do SMART properly" firmware, but I dare not until I have a full backup. All in all, it isn't a good long term data storage solution, but it could be worse.

So, how should I configure these drives as a fully redundant ZFS storage pool? This is slightly tricky. The naive solution is a simple 1:1 mirror, so you configure the 1Tb and 2Tb drives together to mirror the 3Tb drive, making sure you account for the fact that the 3Tb drive is slightly short (3,000,592,982,016 vs. 3,000,603,820,032 bytes). However, the 1Tb drive has 512 byte sectors, and apparently ZFS won't mirror across dissimilar sector sizes, though you can override and force a 4Kb sector size during pool creation. Another problem is that I am unsure if ZFS lets you concatenate two physical units into a vdev, then form a second vdev from that vdev and another physical unit because apparently you can't nest vdevs. However, you could stripe 3x 1Tb i.e. partition the 2Tb drive into two, and stripe all three 1Tb slices into a 3Tb vdev. The only problem with this is it would bottleneck on the 2Tb drive, because it must be constantly read and written at seek locations about half a drive apart i.e. slow, and the 128Kb strip used by ZFS isn't divided by three evenly so you'd get dreadful alignment penalties. Assuming a 5% chance of individual disk failure, risk factor of data loss is 10% (RAID 0) x 5% = 0.5%, so this configuration reduces the chance of data loss by 30x.

What if you want to expand the pool later? Well, ZFS won't let you change vdevs after configuration, so the only expansion route is to add another mirror pair as a separate vdev i.e. probably 2x 3Tb drives, or replace each drive with a larger one and resolder. This is because redundancy is per vdev, so if you lose a vdev you lose the pool. As I only accumulate about 500Gb a year, having to add in chunks of 3Tb at a cost of €400 a pop seems way overkill, never mind I dislike losing the older drives. Adding expansion increases the chance of data loss to 0.75%, but slightly improves the chance of data loss over any one of the drives failing to 33x.

What about RAID-Z? For this you need a minimum of two storage units with one failure tolerated. As this blog suggests, you can partition up the three drives as follows:

• 2x 2Tb partitions on 2Tb and 3Tb drives
• 2x 1Tb partitions on 1Tb and 3Tb drives

As with the 1:1 mirroring solution, here you also get 3Tb of available space. Assuming a 5% chance of individual disk failure, risk factor of data loss is 5% x 5% = 0.25%, or half the previous solution. Because ZFS can either stripe, mirror or RAID-Z but not concatenate, you'd see the bottleneck move onto the 3Tb drive which would stripe over about 66% of the 3Tb drive's total area. It isn't anything like as bad as the earlier mirroring solution for bottlenecking though, as the 4Tb vdev would get filled in preference to the 2Tb vdev.

What about expansion? Well, all you can do is add another vdev same as with mirroring, because vdevs are immutable with ZFS. This sucks.

What about RAID-Z2? This requires at least three storage units with two failures tolerated. However, interestingly, you could deliberately and intentionally create a degraded pool i.e. one which was effectively RAID-Z but could be "improved" to RAID-Z2 by adding a device, or you could have two parity devices with a "missing" data device with a degraded read/write latency. This is, as far as I can see, the sole and only way of creating an expandable vdev in ZFS though at the cost initially of sacrificing 66% of your storage capacity i.e. you'd get 2Tb of available space now.

In short, colour me not impressed. As much as ZFS is cool and everything, it isn't really suited to three device configurations because it isn't intended as such - it's intended for a dozen or so physical devices where a 10-15% parity overhead is just right because it reduces a 60% chance of data loss (assuming 5% individual drive failure) to (assuming no failures during rebuild) just 0.25% for RAID-Z or mirroring and to just 0.0125% with RAID-Z2 (if across all drives). That's a huge win, and that's why RAID-whatever and ZFS makes sense with lots of drives. With just three drives, the parity overhead is large, the lack of ability to reconfigure is inconvenient/expensive, and in short ZFS isn't the right tool for this job.

What we really need is the ability to do "block pointer rewrite" as online vdev reconfiguration is known in the ZFS jargon. Not a lot of chance of seeing that outside Oracle's proprietary ZFS enhancements sadly which effectively means we won't see it at all in the foreseeable future seeing as ZFS v28 is already two years out of date. I had very high hopes for BTRFS, indeed my secure off-site replicated backup solution is based on BTRFS with two copies of everything stored and remotely replicated by DRBD, and BTRFS's design is much more flexible for the low drive count user than ZFS. However, as Chris Mason (lead BTRFS developer) left Oracle for greener pastures in June, any of my hopes there are gone, especially as it's not in any commercial company's interest to push either BTRFS or indeed ZFS for low drive count use cases as in the end, where's the (serious) money in solving home user long term data storage issues? In short, ZFS v28 is as good as it's going to get for the foreseeable future, especially as Oracle are highly unlikely to release any of the improvements since v28 to the public. In reality, with the loss of BTRFS momentum, ZFS is quite literally the only game in town. I guess I'm just going to have to take those 2x 3Tb expansions on the chin!

So, basically I think I've decided to do RAID-Z for the existing 1Tb + 2Tb + 3Tb configuration - it's twice as safe as mirroring without so much of the read/write bottleneck and load placed on one drive. Further expansion though would be in 2x 3Tb mirrors, because mirrors don't lose your data if you break a pool unlike RAID-Z. At 500Gb/year, even that is two and a half years away, assuming that employment at RIM doesn't slow that rate of data acquisition down - which it very likely will. [Note added three months later: I didn't go with the RAID-Z in the end, as mirroring is inherently more fault tolerant because if a unit fails, you simply replace it and resilver whereas with RAID-Z it's a full rebalance which hammers the drives. Instead I "glued" the 1Tb and 2Tb drives together using Linux LVM to make them a fake 3Tb drive, then supplied the two 3Tb drives to KVM for FreeNAS to use as two ZFS storage units. FreeNAS has no idea it's in a virtual machine working with virtual hard drives, and nor does it matter. It all "just works", even if write speeds are in the 100Mbit range, read speeds reach about 400Mbit which is good enough].

Let's just hope that something much better comes along in two years' time. For now, ZFS as the only game in town will have to do. Be happy y'all!

Sunday 24th June 2012: 5.19pm. I got back from my PGCert exams about ten days ago, rested for a day, then launched into my Pure Maths coursework which I posted off on Thursday. I then spent a day writing up my summer todo list before we emigrate to Canada which is as follows:

1. Sign up to swimming pool and substantially improve fitness before emigration.
2. Solve nedprod diary archival problem once and for all, and finally shrink that enormous front page!
3. Enable long overdue mobile view for nedprod.
4. Enable long overdue print view for nedprod.
5. Pay 2011 corporation tax for my company and catch up on its accounts.
6. Submit 2011 Irish and UK tax returns (yes, I must pay tax in both countries!)
7. Close off outstanding bills to my company, hopefully getting those owing money to pay up, and begin making the company dormant.
8. Write up reference implementations of stuff being submitted to POSIX and ISO and submit them.
9. Do annual summer release of my libraries to propagate past year's bug fixes and fix bitrot.
10. Finish BEXML and BEurtle.
11. Finish reading The C++ Standard Library: A Tutorial and Reference (2nd Edition).

As you can tell by the strikethroughs, I've made a good start. I haven't actually swum - not once - in nearly twenty years, so the fitness regaining is going to be embarassing for the first two weeks or so as I flail around haplessly, but I signed myself and Megan up for two months' membership, so at least I'll provide her with no doubt much side splitting amusement .

As you may have noticed by this page being much shorter, I have finally solved the diary post archival problem once and for all and while I was at it I solved the lack of permalinks to posts too. Up until now, since 1998, I archived by copying and pasting each post from the front page into a page per month in the archive folder, then manually hand linking them into a sequence and adding them into the index page. This was so time consuming and boring I always kept putting it off, leading to as much as 250Kb of text on the front page which is plain silly. The new system uses a bit of PHP to autogenerate a page per post and another bit to autogenerate the index, and to archive I now simply have to copy and paste into a single archive file and leave the PHP figure it all out. Outstanding!

The last two completed items are that I finally got round to implementing CSS media views for nedprod, specifically for mobile/handhelds and for print. This required manually find and inserting a magic header insert into every HTML file on nedprod using varying regular expressions, and it took me the entire of Saturday to complete as something like six different versions of FrontPage/Expression Web have touched nedprod over the years, and they were all different. The new magic header insert lets me specify common <head> contents for all pages, so I was finally able to mark all pages as being authored by me to Google and to supply @media handheld and print stylesheets for all nedprod HTML.

Now, I have to admit it is embarassing how it has taken this long to add this to nedprod - particularly the print CSS which is so basic: hide navigational and promotional elements, hide clutter, print URLs after links. The new mobile view is more interesting though: I have been checking that nedprod renders okay on mobile devices since the days of Windows Mobile but I was happy if it looked identical to the desktop view. Over time, as I have browsed nedprod from my own mobile because it's handy as a memory aid, I've begun to prefer if the page were vertically flowed for small screens. With the work done during the last few days, any time the page becomes narrower than 720px it will autoenable the vertical flow - in fact, you can try it now with your desktop browser and see for yourself. Believe it or not, this automatic switch doesn't use Javascript - it uses the new W3C CSS3 media queries extension which any recent browser will (mostly, and not always bug free) support.

Basically, this media queries extension lets you set conditions on CSS blocks, so "if device screen is less than X apply this CSS" and so on. In fact, here is the CSS I wrote for nedprod:

@media handheld, only screen and (max-width: 720px),
	only screen and (max-device-width: 720px) {

table#autolanguagetranslation {
	display: none;
}
table#autolanguagetranslation + p {
	display: none;
}
table.bodytext > tbody > tr > td {
	display: block;
}
table.navbar {
	position:relative;
	margin-left:auto;
	margin-right:auto;
	width: auto;
}
table.navbar + p, table.navbar + p + p {
	display: none;
}
td.navbardivider {
	width:100% !important;
}
* { max-width: 720px; }
}
@media only screen and (max-width: 640px),
	only screen and (max-device-width: 640px) {
* { max-width: 640px; }
}
@media only screen and (max-width: 480px),
	only screen and (max-device-width: 480px) {
* { max-width: 480px; }
}
@media only screen and (max-width: 320px),
	only screen and (max-device-width: 320px) {
* { max-width: 320px; }
}

Note how I additionally test for screen widths less than 640px (iPhone 4), 480px (most older smartphones) and 320px (feature phones) and override the earlier max-width setting on * i.e. all elements. This max-width setting is basically a clamp on the maximum width of any element to the screen width, thus ensuring that any single element on the page will be shrunk to fit onto the screen without panning (if that element can be sufficiently shrunk of course). If the element can't be shrunk e.g. a <pre> section, then panning is available for that element alone. A lot of "how to do mobile stylesheet" guides recommend shrinking all images by 25%, but that buggers small images. This max-width approach constrains any image which is too big to fit, but leaves smaller images alone.

So, perhaps worth waiting for CSS3 media queries after all! The only final thing is to declare your HTML as mobile aware by adding this to your <head>:

<meta name="viewport" content="width=device-width,
	initial-scale=1.0, maximum-scale=2.0, user-scalable=yes" />

And voilà your HTML is mobile aware. I note that all is not perfect on mobile web browsers though. Desktop Chrome and Opera both render the vertical flow perfectly, albeit both forgetting to adjust for the fact there is a vertical scroll bar there and hence chopping off a bit of contents on the edges. My Samsung Galaxy Nexus (Android 4.0.4) seems to start off thinking it is a 480px wide device, not 720px, and renders the text accordingly (probably for compatibility with older phones) but decides about half way through the rendering to suddenly shift to a 640px page width, not 720px (perhaps for compatibility with the iPhone 4 this time?). Weird, I know. But it gets weirder: when it shifts to 640px, it doesn't bother redoing layout for inline positioned items such as almost all the text, so only block items are relaid out to have the correct width and be correctly positioned. Therefore almost all the text is offset to the left with a large empty space on the right which looks inconsistent. Megan's Samsung Galaxy S (Android 2.3.6) is both better and worse because it only observes the first screen setting it sees which is 720px and this it does render perfectly. However, inexplicably it disables your ability to zoom out and thus forces you to pan everywhere using its little 480px screen on a 720px canvas which is very irritating. Buggy browsers: what can one do? I have hope for the future though: Opera Mobile gets it right at default zooms of 100%, 150%, and 200%. Interestingly, the default default zoom in a new install is 225% at which, of course, it completely borks the page .

Anyway, it's now past 8pm so I'd better get moving as I have the final copy of the index of Economists and the Powerful to submit. So, have a great, being happy summer!

Monday 28th May 2012: 7.13pm. You've probably noticed that I've wired in my Google Plus feed to an iframe on the right - I finally got round to configuring the very useful IFTTT to auto-replicate my Google Plus posts to Facebook and Twitter, so I figured why not have them appear here too? Yeah, I guess I'm about a decade behind everyone else there in getting my content to replicate around, but well to be honest I didn't have a need to spam people with my inanity, so I didn't bother. So why the sudden change of heart? Well, the big news is that I'll be relocating to Canada end of this summer to join Research in Motion's Native SDK team as a Senior Software Engineer in their Waterloo, Ontario HQ. They executed a nearly flawless recruitment process over 20+ hours of telephone and video interviewing, and if their business execution is anything like as excellent as their recruitment was then they're due for a huge bounce back if they can make it past running out of cash as they are forecast to do by the end of this year - when no doubt investors will start calling for them to be broken up rather than take on oodles of debt. I figured that as a result of the move and the new job I'll no doubt have a whole load more inanity to spout with no time to do longer updates to this virtual diary, plus people might be temporarily interested in shorter posts to Google Plus for a short time at least, so I went ahead and wired all my data feeds together so you all can keep up if you really, really want to.

On other news, my shiny new cloud infrastructure is up and running beautifully, including secure off-site automated data replication and automated download queuing and management all of which takes care of itself each night after we've gone to bed, and shuts itself down before we wake up while sending me emails daily with its progress so I know it's working okay. I'm still waiting for hard drive prices to drop considerably (see next paragraph) before implementing my RAID6 auto-bitrot healing solution for our very long lived data (and, indeed, I'm also waiting for RAID6 support to enter BTRFS mainline), but in fairness the six virtual machines spread across two hardware nodes which now operate both the ned Productions Limited infrastructure and everything in the house including the ADSL connection all work swimmingly. And I've cut our household baseline power consumption by another 50W despite the much improved, demand-on multi-terabyte shared data solution, so it's been a massive win all round!

So, how do you decide when to buy new hard drive and/or flash storage? A few years ago I did some primary research for Freeing Growth for a section on information storage trends and it's proved useful to keep that data up to date, so here's magnetic and hard drive storage capacities per inflation adjusted dollar from 1980 to April 2012:

I used the Generalised Logistic Function (also known as Richards' Curve) as the model with least-squares monthly average fitting which isn't a bad model for this sort of thing. I didn't go nuts on the regression, so don't expect Summer 2018 to be when flash will definitely catch up with magnetic. In fact, Freeing Growth (written in 2008) originally predicted 2013 as the crossover point, but what's happened is that magnetic has had a sudden extra growth spurt thanks to perpendicular recording while flash has slowed down its rate of improvement, so the catch up has run right out much later. In fact, if higher density flash is as unreliable as they think (Grupp et. al, 2012, The Bleak Future of NAND Flash Memory), the faded out logistic curve I regressed for flash storage is much more likely i.e. there will never be parity between magnetic and flash. Food for thought! If you want the raw data including its sources, I keep an Excel spreadsheet holding the full works here.

My PGCert exams start in just over a week, so my next two weeks are basically gone on finally ridding myself of the Institute of Education and the University of London. After that comes my OU Pure Maths coursework due just the week after, then I'll no doubt have to do my annual visit to Northern Ireland with my sister who is getting married next week. So, basically I'm very occupied with no freedom until the end of June, however after that I have a very exciting last summer in Europe planned. Firstly, I'm going to get my BEurtle issue tracking GUI out the door - it has the beginnings of a Redmine backend working, so I just need to add a Github backend, wire it all together and voilá, that's another major productivity improvement in my life achieved! Secondly, I'm going to make myself read all 1,100+ pages of the newly updated The C++ Standard Library: A Tutorial and Reference (2nd Edition) from cover to cover - thanks to my ISO standards work, I'm fairly familiar with the language changes in C++11 but I can't admit the same for the C++11 STL where I feel myself woefully underinformed about the new facilities (even though no compiler supports more than a subset of the C++11 STL at present). Thirdly, I'm going to finish writing up and submitting my POSIX standard changes and my C language standard changes before I have to resign my SC22 convenorship. And fourthly, I'm going to go visit as many people around Europe as is practical to say goodbye. Sadly, another installment of my Freeing Growth book series looks unlikely - not enough time, unless my Canadian sponsored work visa gets delayed significantly, as we'll have to start packing up and selling off our stuff end of August.

Well, I think that's about it for the time being. Last three months were so boring there is nothing to report from them - it was nothing but study, coursework, job interviews, applying to jobs and migrating to the new cloud infrastructure. Quite literally that was the past three months. Amazing how time can pass when none of it is free! Until next time, be happy!

Monday 19th March 2012: 5.00pm. Wow, a full six months between diary entries! Unfortunately it's been a combination of both being incredibly busy and not having a massive amount of anything interesting to say which has been the cause of my tardiness. Much of September went on getting the World Economics Association launched, in particular upon writing lengthy email replies, and getting all editions of the Freeing Growth manifesto into print. In October I was appointed as ISO SC22 mirror committee convenor for Ireland by the NSAI's ICTSCC committee which is the mirror of ISO JTC1. In November I was contracted by the WEA to write a virus scanning plugin for the PKP Open Journal Systems framework they use for their journal management (and which meant I had to touch PHP again, which is always unpleasant), and my copy editing of that book manuscript on Economics I mentioned last entry turned into a co-authorship of a book with Handelsblatt correspondent Norbert Häring called Economists and the Powerful: Convenient Theories, Distorted Facts, Ample Rewards. That book manuscript then sucked up pretty much all my free time until it was delivered to the publisher mid-February. In between came an avalanche of IT contract work, so much so that for the first time in a long time I had to actually turn paying work down which always sucks royally.

Since mid-February, I have finally begun to get around to a major systems upgrade of our computer systems. Before the end of its tax and accounting year in January, as normal my company had ordered its annual hardware parts which this time included a bottom end Sandy Bridge server which was amazingly cheap for what it is, despite having an 80 PLUS Gold power supply (>=90% efficiency) and therefore sips just 28W which isn't much more than my old Intel Atom 220 internet gateway despite being umpteen times more powerful. My plan is to solve a long-running headache in my server deployment: right now, there are three hand configured servers on the public internet which use a manual rsync for backup onto what is effectively Megan's television as it switches itself on when used, and off after a timeout. As time has gone along, that television has ended up storing not just backups, but all the shared stuff between my computers, all the GIT repos, all my company stuff etc. Yet it runs pretty much without backup - every now and then I dump copies of things onto an external 2Tb USB hard drive, but overall this setup - while somewhat secure and flexible - is extremely manual, and very time consuming. I also have the current problem that only my main workstation has a copy of VirtualBox on it for temporary OS deployments for testing, and that gets hand configured each time too. If I want to do a quick server config change, right now I have to chance my arm, take down my services or do a lengthy rsync to VirtualBox, distill a set of upgrade steps and repeat those on the live server as my ADSL outbound is too slow to upload images. All in all, none of this is ideal. For example, some months ago I had a major outage of all my websites and email due to a late night misspelling in /etc/network/interfaces which required a technican intervention as the server was no longer able to boot. That's expensive, never mind inconvenient. The fact that email, web and everything else is all the same server is particularly unhelpful.

What I really want is a cloud platform with virtualised OS instances, so to test you simply clone a running instance and employ a whole instance per service so each service stands alone and independent from the others. I also want to be able to deploy temporary OS installs much more quickly, so instead of always having to run test suites on my workstation using a fake localised config in VirtualBox, I can instead run them against a real server which could at any moment be deployed onto the real internet. A cloud platform lets you run a proper automated backup solution which auto-syncs parts of itself with the other servers as an off-site backup of the really important stuff, plus runs anti-bitrotting sweeps on the really long-lived data. That means that the public servers automatically are backed up locally, and the local servers are automatically backed up remotely. Furthermore, instead of almost all my big and important data living on an external USB hard drive which needs to be plugged in for me to use it (I don't leave it permanently plugged in, lest it get deleted or damaged), I could have all that data on demand secured with RAID redundancy and a snapshotting filing system like ZFS or BTRFS so no one can delete anything whether accidentally or otherwise. That alone would be extremely useful.

Anyway, I've got that cloud infrastructure working and here are the instructions on how to do it, but I haven't finished data replication yet. The slow pace is once again due to lack of free time, but also because the cloud stuff isn't a priority until Ubuntu 12.04 LTS has gone gold as most of my deployments are Ubuntu Server LTS, so with the imminent release of 12.04 there seems little reason to rush. In the meantime, I have been investing a lot of effort into a once and for all distributed bug tracking solution which lets you coalesce multiple sources of issue tracking into your GIT repo via both a RESTful HTTP API as well as a programmatic API. This subproject is called BEXML, and it's an extension of my BEurtle issue tracking GUI though it's also a fully standalone library. Getting this working - and I'm not far from finished - would be a major boon to not just myself personally who has to handle issues arriving via email, public bug trackers as well as privately when I spot a problem and need to remember to fix it some day, but also to several major open source software projects and those who run and coordinate those projects as they face exactly the same management problem as me.

Adding to the demands on my time are taking two university distance courses at the same time: the PGCert, which please God will be over this summer, as myself and the University of London/The Institute of Education have definitely parted ways after their never ending dreadful customer service (they simply don't care about the student experience [they go through the motions, but it's all hot air promises and nothing ever changes], and every staff member always says "it's not MY fault/responsibility". So if no one will ever take ownership of any problem not strictly within their personal remit, no wonder there's such appalling customer service!). I'm also taking that OU Pure Maths course which is surprisingly fun with all sorts of abstract puzzle solving, and as much as Open University courses are pure spoon feeding and regurgitation I am certainly not complaining in Pure Maths where spoon feeding is just fine by me! What irks me in courses such as Education is that I don't see why my well argued, supported and referenced opinion isn't as worthwhile as most of exactly the same within the Educational literature. I'm paying for the course, so my academic arguments ought to have equal initial standing with other non-peer reviewed arguments (they may well, and indeed probably will falter after analysis, but the point is that argument is argument). What happens instead is a NIH response, so if it's Not Invented Here then it's obviously no good, especially if the so called "educational expert" has never heard of anything outside their field and are too bloody lazy or ignorant to bother doing a god damn Google search before deciding that if they haven't seen it before, it must be lies. This is not what one would expect from the Institute of Education, one of the premier educational research institutions in the the world, but there you go. I, as the paying customer, am not paying £64 per ECTS credit (about €75 per ECTS, expensive by European standards) to be told my arguments are worthless relative to the course material - especially when the course material is constantly banging on about how student's sociocultural understandings are to be treated as valid-in-themselves and not to be dismissed out of hand just they they're doing with my arguments. I dislike hypocracy at the best of times, but I especially dislike being discrimated against for pointing their hypocracy out when I'm the one paying their wages. So definitely, the sooner I can get away from those robbing bastards the better, because what they advertised for their course has little to do with its practice and if UK consumer law applied to universities I'd submit a product mislabelling complaint (it doesn't, so you have no redress except via the quality regulator). Caveat emptor I guess. I just wish it hadn't cost me two thousand pounds to find out I've been conned.

So that's basically what I've been up to during the past six months. I had to can the Oxydérkeia project sadly - without a set of students to test it upon, and now it won't be part of my Masters in Research thesis with the IoE, rationally speaking it had to get chopped. It's a real shame though - that technology would have been extremely useful in a multitude of tasks, in everything from evaluating prospective new employees down to time and motion studies of computer using employees. However, the Institute of Education won't play ball and seem mired in their own navel gazing and ivory tower as their precious government funding gets cut (they have very little experience gaining research funding from industry), and I can't see recruitment agencies paying for its development given how unbelievably technologically backward recruitment is (it's because, of course, technology will eventually make much of their utility obsolete and they know it). Better then that I focus on other more productive-to-me uses of my time - hence my focus on solving that issue tracking problem as even if nobody else uses it, it would be bloody handy for me personally.

What as to the near future? In the immediate future, finishing what I started above is the plan. In the slightly longer term, chances are good that me and Megan are leaving Ireland for economically greener pastures. As much as I'm busy and doing things like turning down paying work much to my chagrin, Megan won't see a salaried job that has anything to do with teaching any time in the next five years. If we're going to have kids, we need a stable income. Ireland can't provide that for both of us, nor will she anytime soon and neither can most of Europe in its present state. So since the start of the US H1B visa season (March) I've started applying for jobs in the US and to a much lesser extent, Canada. So far interest has been very good - to date more than eight out of ten applications I've made have resulted in telephone interviews, and it makes last year when I was applying within the UK and Europe look dreadful in comparison. Surely one of these ought to result in a visa sponsorship, and off to (probably) Silicon Valley it'll be for us. If I get a window of at least a month this summer before we emigrate, I'd like to start a second Freeing Growth mini-book this time on the natures of growth, so asking things like What is growth? What forms does it take and how can these be measured using today's tools? Very 19th century, I know, but amazingly I can't find such open ended questioning since Jevons in the 19th century believe it or not. We're well overdue an update given oh, the rise of computer analysis, the development of new statistical mathematics and such. It's about knowing what we don't know.

Almost certainly it won't be another six months till the next entry! I'm playing Mass Effect 3 in the rare occasion I have the time, and I understand from the internet that the endings are shockingly bad with over 60,000 votes in a poll on the Bioware forums and over 40,000 votes in a Facebook campaign page calling for the endings to be fixed to something like what had been repeatedly promised by Bioware since the start of the trilogy. From my own perspective, so far into the game ME3 looks rushed: there are as many graphical, camera and gameplay glitches as there were in ME1, they stupidly made a core team member into a paid first day DLC because they clearly had nothing else to hand, they haven't bothered with most of the interactive person-to-person conversations they had in ME2, and I'm struggling to understand how they came up with this storyline given the events in ME2 as there appears to be a missing bit in between the two. I know that many of the lead storywriters quit Bioware about a year ago, and there has been a general talent exodus since Bioware were bought out by EA, never mind some dreadful ME book releases since that talent exodus one of which had to be recalled it was so bad. ME3 looks just as you would expect if the core talent left or stopped trying half way through development. The endings, I understand, appear to have been bolted onto the end of an unfinished story arc, so basically you're nearly there storywise and suddenly you get some long cutscene, get given three choices each of which chooses pretty much the same ending and supposedly that's the trilogy into which one has invested a hundred hours of your free time over four years done?

So expect, once I'm finished, to express my bitter disappointment here. Mass Effect 2 is the only game I've bothered to play more than once through since the 1990s, and I can count on one hand those games which I have ever played through more than once. The fact that all those decisions I took would have absolutely zero effect on the final outcome is a travesty, and even a company well known to hate customers such as EA must surely realise that these sorts of asshole move will impact profitability (which is the sole thing that EA management understand, let alone perceive). It's real unfortunate - what a lost opportunity so close to the finish line! Hardly the first time in human history to happen due to dreadful management though. So, till that next entry, be happy!