Wednesday 20th March 2019: 8.35pm. So I was complaining last day about how my new Threadripper workstation didn't seem much faster. Today I compiled the entire work codebase. What took over two hours on the Ivy Bridge workstation now takes fifteen minutes! Which is pretty impressive, really.
Welcome to ned Productions (non-commercial personal website, for commercial company see ned Productions Limited). Please choose an item you are interested in on the left hand side, or continue down for Niall’s virtual diary.
Niall’s virtual diary:
Started all the way back in 1998 when there was no word “blog” yet, hence “virtual diary”.
Original content has undergone multiple conversions Microsoft FrontPage => Microsoft Expression Web, legacy HTML tag soup => XHTML, XHTML => Markdown, and with a ‘various codepages’ => UTF-8 conversion for good measure. Some content, especially the older stuff, may not have entirely survived intact.
- A biography of me is here if you want to get a quick overview of who I am
- An archive of prior virtual diary entries are available here
- For a deep, meaningful moment, watch this dialogue (needs a video player), or for something which plays with your perception, check out this picture. Try moving your eyes around - are those circles rotating???
You can find the posts here replicated onto Diaspora, if you prefer to subscribe there instead.
This is the very first post using my snazzy new hand-written Markdown new post editor, which you can test for yourself at https://www.nedprod.com/editor.html. I’ve been working on this at nighttimes for a number of weeks now. It live renders the Markdown which you type into HTML. It has a fixed viewport based layout, so each editing pane always consumes half of whatever screen size your browser has, which means it works as perfectly on mobile as it does on desktop. It automatically scrolls each pane so you can see, live, exactly the rendered Markdown for what you are editing. It can insert your geolocation and any link that you want to share. It retains your in-work post in your browser, letting you leave and return to finish it later without the possibility of losing anything. The only major missing functionality is uploading images with a post, and that’s my next task.
The third big thing I’ve learned is just how much easier web programming has become. The tooling is vastly better than it was. Web browsers work consistently, their debuggers are all excellent, you can inspect HTTP exchanges with ease, even newer PHP doesn’t suck anything like as much as it used to. Security is also vastly better than it was, more than half of all the time it took to write this editor went not on the editor, but on figuring out how to run Hugo, because running arbitrary programs on your server is super non-trivial to make work nowadays. PHP simply won’t run external executables any more, which is a very good thing. Docker is very good at preventing you escaping it. I tried many routes before I sucked down the reality that I was going to have to learn Docker, which turned out to be nothing like as bad as expected. Old dog, new tricks in the end, and I can definitely see the big advantages in containerising absolutely everything into its own almost entirely read-only container.
The next minor thing is to get new posts replicating automatically onto Diaspora and other social media. Then comes image upload. Then, I think I’m probably done here for now, barring bug fixes. I could add arbitrary post editing, but to be honest, given I have SSH access and can just go delete bad posts or whatever, I’m not sure I care enough to implement a web UI for any of that.
Someone that I still find amazing is how the world has turned full circle. I have a de-Googlified phone based on https://microg.org/. I have never given over my email to big multinationals, but Mailcow comes with full calendar and todo management, plus a web UI and it takes care of SSL certs using LetsEncrypt for you. So that let me migrate away from Google Calendar, transferring over all the stuff intact, and fully working, and my phone auto-integrates with Mailcow calendar and todos, as well as email.
Now I’ve completely left multinational social media as well. Everything I now do runs 100% on my infrastructure running 100% open source. Nobody mines me for any of my information anymore, except from what can be gleaned from web scraping bots, and tracking cell tower locations, and those are rather hard to prevent. I’m basically back to 1998 when I first started this website, in a way, except the technology is vastly, vastly, superior to what we had back then.
I mean, this stuff auto-curlys my apostrophes’, like Microsoft FrontPage used to until you disabled it. Ain’t that something!
The previous website was a direct evolution from when it was begun in 1998. Microsoft FrontPage was upgraded to Microsoft Expression Web, various upgrades for new content such as CSS, XHTML, Unicode and microformatting occurred over the years. A fair chunk of custom PHP scripting implemented pseudo-dynamic website features, such as scanning all the XHTML content into an Atom RSS feed, and then rendering that RSS feed dynamically as pages.
That solution served me well, as at that time there weren’t static website generators sufficiently capable that I could move on. However my recent work with Hugo to implement Outcome’s documentation persuaded me that Hugo probably – just about – had enough flexibility to do a high fidelity conversion of nedprod.com.
The new tooling, Hugo, opens many new doors. Firstly, I can now write posts in Markdown instead of HTML – this post is the very first of what will be many written in Markdown. Secondly, Hugo autogenerates a RSS feed for me, and at some point soon I’m going to hook that into auto-posting to Diaspora and Pleroma, now that Google Plus has gone. Thirdly, I should be able to plug an Android app which speaks ActivityPub (probably AndStatus) into being able to post content here. Or I may just make a private web form with Markdown editor which can post new entries here from any web browser, and make that mobile friendly e.g. with picture upload and GPS coordinates etc.
So far, so good. And just in time for the Google Plus shutdown. I made it!
Tuesday 19th March 2019: 4.36pm.
You may remember that I pulled the trigger on a new workstation to replace my six year old Ivy Bridge one:
- Sixteen core AMD Threadripper 2950x @ 3.5Ghz, turbo 4.5Ghz.
- 32Gb of DDR4-3200 RAM @
- 512Gb Samsung 970 Pro NVMe SSD @ 3.1Gb/sec
I spent no less than thirteen - yes, thirteen - hours yesterday during the public holiday pulling out the old system, and putting in the new. Went home after midnight with, finally, a machine booting into Windows, after a very long and frustrating day which included having to cut through sheet metal in the case to make space for my video card, and slicing my wrist on sharp metal. Yay.
Spent most of today installing stuff and trying to get Maystreet's code building again, as after all today is a work day. Just got the Windows build working there, currently working on the Linux build.
So, first impressions of the new platform is how little different it feels to the preceding one. Which is the first time that's ever happened to me in an upgrade! Yes it feels slightly snappier. But it's a "fresh Windows install" kind of slightly snappier, not a "I just dropped over a grand on a new workstation" snappier. Much of the cause is that the preceding Ivy Bridge system also ran at 3.9Ghz, and also had a top end Samsung SSD, albeit SATA instead of NVMe connected. But despite all these oodles of CPU cores and 4x the memory bandwidth, it makes very little difference indeed to perceived latency, for the simple reason that the old system responded in pretty much the same lag as this new one does.
However where you notice the difference big style is compiling Maystreet's code. That used to take a number of hours to compile it all, and a good twenty minutes for a complete rebuild of the subproject I'm working on. This new system does considerably better, it's under five minutes now. Which will be a big productivity bump in the months to come.
Something interesting is that even the ninja build tool rarely keeps more than eight CPU cores busy for long. Occasionally it hits sixteen cores, but it doesn't last long. The build here just isn't parallelisable enough to do better.
In terms of other observations of the new workstation, apart from how awful it was to get it booting at all, since I figured that out it's been boringly predictable. The last time I had an AMD system was over a decade ago, and the chipset drivers just weren't quite as stable nor polished as the Intel ones. So far - albeit it's been just a day - I've seen zero issues. It's been very smooth running, all the hardware has been compatible, everything has benchmarked at speeds you'd expect first time (note my RAM bandwidth is a good bit more than what online suggested, I am running this system in its true NUMA configuration instead of the hardware trying to hide that not all the RAM is equally close to all of the CPU cores). In short, once it actually booted, it's been very plain sailing indeed.
Now just need to find the time to put the old parts onto eBay!
Wednesday 6th March 2019: 9.01pm. Just pulled the trigger on the guts of a new workstation to replace my six year old Ivy Bridge workstation:
- Four core Intel i7-3770K @ 3.9Ghz ("overclocked" to turbo always on).
- 16Gb of DDR3-1600 RAM @ 21Gb/sec.
... with this instead:
- Sixteen core AMD Threadripper 2950x @ 3.5Ghz, turbo 4.5Ghz.
- 32Gb of DDR4-3200 RAM @ 64Gb/sec.
So four times the CPU cores and three times the memory bandwidth. Should be a nice upgrade, I might even notice the difference. I can see that so many CPU cores will be memory starved under load fairly often, but the AMD chip is half the cost of the equivalent from Intel. Or, rather, for the same money you get twice the CPU cores, even if they are around 10-15% slower than the equivalent Intel chip at full load (single threaded things are rather better, only 5% slower or so).
It'll be going into the same case, PSU, graphics card et al, with the old parts going on to eBay to recoup whatever I get. I'll be sad to see my vastly overkill Noctua NH-D14 HSF go, it is so overkill a cooling solution that even under full load, that quad core CPU didn't spin up the fans. Case ventilation was sufficient. The new HSF is also a Noctua, but with just one radiator block and fan. That said, it actually weighs more than the old one, despite being much smaller. We shall see how it fares.
Been making good progress on other todo task like items other than wifi routers :). I've been slowly writing https://scrapy.org/ plugins to scrape http://www.nedprod.com's content into a https://gohugo.io/ generated website. Scrapy is a great tool, highly recommended. Once I've converted all that Microsoft FrontPage 1998-era HTML, I can then finish my migration away from Google+ and all megacorp social media who resell my information onto 100% wholly owned and maintained by me social networking based on ActivityPub. I've persuaded the maintainer of https://andstatus.org/ to finish his ActivityPub support, so then I'll be able to post photos, status updates, and all the stuff I normally do from my phone to my Hugo generated static website, which in turn will plug into the federated social network. I don't know if I'll get this entirely done before Google turn Google+ off in April, but let's see how it goes.
On other todo task like items, I have fully migrated my business off its expensive business bank account with a major bank, and I have closed that account. I'm now 50/50 split between TransferWise and Fire.com on the business just in case of Brexit. I've also started migrating my personal bank account to the German https://n26.com/ bank to also get away from the new unavoidable monthly maintenance fees the Irish major banks have just imposed. It's a lot of hassle moving over all the direct debits etc, but so far so good, the German IBAN hasn't been a problem yet.
N26 is okay so far. They're a bank. Not much to see really, though I do like the instant appearance of transactions. Like, you withdraw cash from the ATM, you literally can see the payment go through its stages on the app in realtime. Same for buying stuff with the card. The realtime aspect proves that any bank could do the same, but they can't of course due to all those legacy systems they have.
Their app is pretty buggy, bits of it don't work at all, and it burns CPU if it's open (so always force close it when you leave it). Getting it to work with my de-googlified phone based on MicroG was a royal pain in the ass, it took three hours of fiddling, I eventually got it to work. Badly made app though, whereas TransferWise have a very good phone app that works very well and is the best of the banking type phone apps I've seen so far.
In fact, TransferWise are increasingly becoming head and shoulders above everyone else. The most features, the least hassle, the lowest fees. Shame they are not bank guaranteed though, you just can't trust leaving any significant money with them. And after that recent scandal at that German ePayments company where they were faking the accounts and now people's deposits will get a haircut, that bank guarantee is more important than ever.
Monday 18th February 2019: 10.02pm. Picked up an old Netgear R6220 2x2 11ac router with nasty cheap Mediatek chipset off eBay for €30 including postage. OpenWRT went onto it surprisingly easy, and now it's a second Wifi AP in my house solving the wifi blackspot in my bedroom, and more importantly, the toilet.
OpenWRT has come so very far in the last few years. Easy to install, easy to configure with a lovely GUI, performance doesn't suck like it used to either. And so very, very, very much better that the proprietary firmware they ship with.
I am pleased to announce that earlier today, I joined Maystreet Inc, a Fintech next generation capital markets platform delivering low-latency connectivity & high-precision data to banks, brokers, etc. This will be a long term gig for me, I expect to be with them for some years building out some very cutting edge technologies which deliver significant competitive advantage. I am extremely glad to not have to return to working in Dublin onsite, a year of that was more than enough.
I interviewed with a lot of other companies since November, indeed a bit of a bidding war ended up happening for me. I'd like to thank those firms for their interest in hiring me long term, and I wish them the very best in their future goals. Most of them were very impressive, and so long as this tech bubble doesn't deflate too harshly, I expect they will all succeed.
Last night I completely replaced the proprietary firmware on the house wifi router (the excellent value 3x3 11ac Archer C7 that just about everybody techie uses for their home wifi) with https://openwrt.org/. Why? The proprietary firmware was actually not awful, but my phone kept losing signal when I'm on the toilet, and it began to bug me that I can't do things like increase the transmit power, or use all of the 5Ghz frequencies available.
Anyway the switch was as easy as feeding its web interface an OpenWRT firmware file (mine is the v2 hardware, they made it not as easy on the v5 hardware). Very, very easy. And OpenWRT is completely different to the last time I used it years ago! It's now got a super easy GUI, a very recent Linux kernel, an enormous collection of installable packages, and of course more knobs and twiddles than you can imagine for configuration. It's a proper full fat Linux distro now, and while the GUI isn't quite as excellent as pfSense's, it's getting close.
Where I am typing this now used to get two bars of signal, and occasional wifi dropouts requiring me to hang the laptop off the edge of the bed to regain a connection. Now that I've quintupled the transmit power and moved my 5Ghz range outside that of my neighbours, I'm now on four bars of signal. Much better. And decent reception in the toilet now as well, so this has been a big win for very little cost or effort.
I did however lose a bit on peak transfer from LAN to WLAN. The Qualcomm chipsets implement hardware data routing for LAN => LAN, LAN => WAN, and LAN => WLAN of which the OpenWRT firmware only uses the LAN => LAN. The proprietary firmware could push Samba => Windows from a Gigabit LAN to the wifi at 77Mb/sec. OpenWRT copies the data in software, and the relatively slow CPU gets pegged at max, dropping Samba => Windows to 60-65Mb/sec. So about a 20% peak transfer performance loss.
This is unfortunate, however it's rare I bulk transfer data over the wifi. And 60-65Mb/sec is plenty fast when the broadband can't exceed 8Mb/sec anyway. So I can live with it in exchange for much improved wifi range.
After three hours carved out to "get it done already", finally got my cheap USB gnuk OpenPGP dongles purchased last August (!) to implement ed25519 on-device crypto for GnuPG and Putty SSH on Windows. That means that (a) I cycled all my encryption keys, which was becoming pressing since I last did so in 2016 (b) even I have no idea what my new encryption keys are, nor can I nor anybody else ever know, because all my encryption is now done 100% on the USB dongle (c) even a nation state actor could now take over my desktop PC, and they still couldn't get into any of my other computers from that PC.
Well, only sort of to the latter point. My USB dongles are really cheap, and are missing a physical button to authorise an operation. So, as long as they are plugged in, and a keylogger has grabbed the PIN code, they can be made to do nefarious work silently in the background. Yes I probably should have sprung for the more expensive models with a button. But, equally, I only ever need to plug these in when initiating a SSH session, which isn't that often in truth, a few times a month at best, and you can just pull out the USB key immediately after you've used it. My git repo commits and authentication still use an on-computer ed25519 key, mainly because if others grab that, it's not the end of the world, I keep backups of all my github repos anyway.
Am I being excessively paranoid? Probably. But it's little cost for a big gain.
ACCU 2019 schedule just went up. My final C++ conference talk for the foreseeable future is linked, after this there will be no more attending conferences for me. It is on Elsewhere Memory, and will be the most speculative conference talk I have ever given, musing on the possible future of all computing everywhere. I'll be taking the ideas presented here to the C and C++ standards committee later this year.
As preparation for Google Plus coming to an end soon, I improved the Python scripting which mirrors this stream of posts onto http://www.nedprod.com/ to also extract post locations and images and videos. Indeed, http://www.nedprod.com/ is now 100% served from my own servers, and no longer uses Google for anything at all. So Google Plus can now vanish tomorrow, and all my post data - seven years of it now - is safe.
I have no idea what I'm going to replace Google Plus with in a few months time. None of the alternatives I have tried I have found compelling. What I want is a very simple single person ActivityPub client-to-server service which uses the filesystem for storage, perhaps using a https://gohugo.io/ filesystem layout so your post store can double as a statically generated Hugo website. I very specifically do not want databases, Docker instances, or complexity i.e. maintenance. I also would like a server-to-server ActivityPub service which can serve the RSS feed from the Hugo website to the federated social network, but that's much less important right now than the former given Google Plus goes away in April.
Given the lack of any alternative, I suspect that I'm going to have to implement the former myself, which sucks, but it is what it is. At least ActivityPub is well documented (https://www.w3.org/TR/activitypub/), and probably others will find my solution useful to them as well. I'll probably write it in Python, it's well suited for parsing JSON and HTTP, and while we do need scalability in the post database, we definitely do not in the service seeing as it'll only ever be you personally using it. So a simple CGI script is probably fine - though I'll implement it as FastCGI anyway, as nginx can't do CGI.
Anyway, that's next week. Let's see if I'm back to employment next week first before launching into new software projects!
Just finished What Remains of Edith Finch, which was on my bucket list for inter-employment R&R gaming. It's basically a slightly interactive series of short stories, the kind of thing Megan would really love. And yes, even with my thick skin of cynicism, I have to admit I enjoyed the three hours or so of "reading" the stories. It did even tug a bit at the emotional heart strings, though I found the ending a bit natch. Definitely worth the tenner I paid for it on sale. I'm also currently playing through the original Walking Dead Tellgame game, yes it has actually sat in the inbox this long despite being listed on https://en.wikipedia.org/wiki/List_of_video_games_considered_the_best. And well ... it's a bit like with Half Life 2. Sure, it's a great game. But it's a bit set-piecey, and you can see the set pieces from miles away, and to be honest there's nothing particularly special in this game, as there wasn't in HL2, that I could see as why people laud it as they do. Nevertheless, I can see myself finishing it, which is rare for me with most computer games as boredom sets in early for me usually. I also picked up a raft of other games on Steam Winter Sale. Spent about thirty euro for six games, all of which are apparently somewhat like Mass Effect 2, which I rate as the best overall game in the 2010s bar none. We'll see how much time I get to play those.
Back in Ireland from our two weeks away in California. Feeling quite rough from jet lag, it'll take a few days to recover properly. Total cost was almost exactly six grand in euro, which was slightly disappointing given my budget was five grand. I underestimated the cost I think of food, restaurants and takeaways were Irish prices in California rather than Indiana prices. I was routinely surprised at how expensive groceries were outside of Walmart. We also dropped three hundred dollars or so on booze, specifically very high end very rare in Europe bourbons and ryes, half of which came home with us (and will almost certainly get their own post, I really lucked out in my choice).
Was this a better break than last year's in Disneyworld Florida? No, Disneyworld was much more of a holiday, this had a significant work and stress element to it, partially because wilderness and nature don't suit small children well and the long drives between places encourage them napping instead of going to sleep at nights, whereas Disneyworld for all its faults is much easier on the adults because it knocks them out and travel and food is taken care of for you. It also cost ten grand in euro of course.
Our holiday in 2019 will be at Halloween rather than Christmas. I expect much less driving long distances, with a corresponding benefit to children sleeping early and well. I also expect more fun stuff which costs a lot of money, so two weeks might be too much for me to afford. We spent a full grand in the last four days because we were in San Francisco doing daily fun stuff! But equally as much as I'd personally like to do nothing whatsoever - which is my definition of time off - bored children would make that an unrelaxing hell. So you either spend lots of money to entertain them and get some peace, or you don't get a holiday. Which sucks, but that's having children for you!
Wednesday 2nd January 2019: 3.36am.
Sunday 30th December 2018: 4.32am.
Friday 28th December 2018: 1.18am.
Friday 28th December 2018: 1.17am.