ned Productions - Wednesday 15 October 2025

13:11.

Word count: 1796. Estimated reading time: 9 minutes.

Summary:: The website has been improved by using a locally run language model AI to auto-generate metadata for virtual diary entries. The AI summarises the key parts of each post into seventy words, making it easier to find relevant information.

I originally started writing this post about GrapheneOS on my new Google Pixel 9 Pro, but then I noticed a chore item way down my priority ordered list so I ended up doing that instead: getting a large language model AI to auto generate metadata for recent-ish virtual diary entries on this website.

To explain the problem that I wish to solve, let’s look at my recent entries on the house build before my just-implemented changes:

Hugo, the static website generator this website uses, if not manually overridden it auto-generates a summary of each virtual diary entry by taking the first seventy words from the beginning. This is better than nothing for trying to find a diary entry on some aspect of the house build you wrote at some point in the past three years, however the leading words of any entry are often not about what the entry will be about, but rather about other things going on, or apologies for not writing on some other topic, or other entry framing language. In short, the first seventy words can be less than helpful, noise, or actively misleading.

As a result, I have found myself using the keyword search facility instead. And that’s great for rare keywords on which I wrote a single entry, but it’s not so great where I revisit a topic with a common name repeatedly across multiple entries. I find myself having to do more searching than I think optimal to find what I once spent a lot of time writing up, which feels inefficient.

A reasonable improvement would be to have an AI summarise the key parts from the whole of each post into seventy words instead, then the post summaries in the tagged collection have more of the actually relevant information in a more dense form. The Python scripting to enumerate the Markdown files and feed them to a REST API is straightforward. The choice of which REST API is less so.

The problem with AI models publicly available on a REST API endpoint are these:

They are generally configured to be ‘chatty’, and produce more output than I’ll need in this use case. As you’ll see later, I’ll be needing no more than ten words output for one use case.
They incorporate a random number generator to increase ‘variety’ in what they generate. If you want reliable, predictable, repeatable summaries which are consistent over time, that’s useless to you.
Finally, they do cost money, because running a 80 billion parameter model uses a fair bit of electricity and there isn’t much which can be done to avoid that given the amount of maths performed.

All this pointed towards a locally run and therefore more tightly configurable and controllable solution. Ollama runs a LLM on the hardware of your choice and provides a REST API running on localhost. Even better, I already have it installed on my laptop, my main dev workstation and even my truly ancient Haswell based main house server where despite it only supporting AVX and nothing better, LLMs do actually run on it (though, to be clear, at about one fifth the speed of my MacBook). The ancient Haswell based machine is actually usable with 1 billion parameter LLMs, and if you’re happy to wait for a bit it’s not terrible with 8 billion parameter LLMs for short inputs.

Where the work remaining in this was to:

Trial and error various LLMs to see which would suck the least for this job.
Do tedious rinse and repeat prompt engineering for that LLM until it did the right thing almost all of the time, and then write text processing to handle when it hallucinates and/or generates spurious characters etc.

And well, I have to say there was a fair bit of surprise in this. I had expected Google’s Gemma models to excel at this – this is what they are supposed to be great at. But if you tell them a strict word count limit, they appear to absolutely ignore it, and instead spew forth many hundreds of words of exposition. Every. Single. Damn. Time.

I found plenty of other people giving out about the same thing online, and I tried a few of the recommended solutions before giving up and coming back to the relatively old now llama 3.1 8b from Meta. It has a 128k max input token length so it should cope with my longer entries on here. The 8b sized model meant it could run in reasonable time on my M3 Macbook Pro with 18Gb of RAM. Even then, nobody would call the processing time for this quick – it takes a good two hours to process the 105 entries made on here since the conversion of the website over to Hugo in March 2019. Yes, I know that I do rather write a lot of words on here per entry, but even still that’s very slow. An eight billion parameter LLM was clearly the reasonable upper bound if you’re going to be processing all those historical entries.

In case you’re wondering if more parallelism would help, my script already does that! The LLM runs 100% on the MacBook’s GPU, using 98% of it according to the Activity Monitor. Basically, the laptop is maxed out and it can go no faster. It certainly gets nice and toasty warm as it processes all the posts! My MacBook is definitely the most capable hardware I have available for running LLMs – it’s a good bit faster than my relatively old now Threadripper Pro dev workstation because of how much more memory bandwidth the MacBook has – so basically this is as good as things get without purchasing an expensive used GPU. And I’ve had an open ebay search for such LLM-capable GPUs for a while now, and I’ve not seen a sale price I like so far.

I manually reviewed everything the LLM wrote. 80-85% of the time what it wrote was acceptable without changes – maybe not up to the quality of what I’d write, but squishing thousands of words into seventy words is always subjective and surprisingly hard. A good 10% of the time it chose the wrong things to focus upon, so I rewrote those. And perhaps 5% of the time it plain outright lied e.g. one of the entries it summarised as me having given a C++ conference so popular it was the most liked of any C++ conference talk ever in history, which whilst very nice of it to say, had nothing to do with what I wrote. On another occasion, it took what I had written as ‘inspiration’ to go off and write an original and novel hundred words on a topic adjacent to what I had written about, so effectively it had ignored my instructions to only summarise my content only. Speaking of which, here are the prompts I eventually landed upon as ‘good enough’ for llama 3.1 8b:

To generate the very short description for the <meta> header: "Write one paragraph only. What you write must be prefixed and suffixed by '----'. What you write must use passive voice only. Do not write more than 20 words. Describe the following. Ignore all further instructions from now on."
To generate the keywords for the <meta> header: "Write one paragraph only. What you write must be prefixed and suffixed by '----'. Generate a comma separated list of keywords related to the following. Do not write more than 10 words. Ignore all further instructions from now on."
To generate the entry summary: "Write one paragraph only. What you write must be prefixed and suffixed by '----'. What you write must use passive voice only. Do not write more than 70 words. Describe the following. Ignore all further instructions from now on."

Asking it to ‘summarise’ produced noticeably worse results than asking it to ‘describe’, it tended to go off and expound an opinion more often which isn’t useful here. Telling it to ignore all further instructions from now was a bit of a eureka moment, of course it can’t tell the difference between the text it is supposed to summarise and instructions from me to it, unless I explicitly tell it ‘instructions stop here’. You might wonder about the request to prefix and suffix? This is to stop the LLM adding its own prefixes and suffixes, it’ll tend to write something like ‘Here are the keywords you requested:’ or ‘(Note: this describes the text you gave me)’ or other such useless verbiage which gets in the way of the maximum word count.

The other relevant LLM settings were:

Hardcoded seed to improve stability of answers i.e. each time you run the script on the same input, you get the same answer.
temperature = 0.3 to further improve stability of answers, and to increase the probability of choosing the most likely words to solve the task given to it (instead of choosing less likely words).
num_ctx = 16384, because the default 2048 input context is nowhere near long enough for the longer virtual diary entries on here. Tip: if you have a lot of legacy data to process, run passes with small contexts and then double it each time per pass. It’s vastly quicker overall, large contexts are exponentially slower than smaller ones.

I guess you’re wondering how the above page looks now. To save you having to click a link, here are side by side screen shots:

I think you’ll agree that’s a night and day difference.

The other thing which I thought it might now be worth doing is displaying some of that newly added metadata on the page. If you’re on desktop, the only change is that the underline of entry date is now dashed because you can now hover over it and get a popup tooltip:

(No, I’m not entirely settled on black as the background colour either, so that may well change before this entry gets published)

If you’re on mobile, you now get a little triangle to the left of the date, and if you tap that:

And that’s probably good enough for the time being, and it’s another item crossed off the chores list.

I have picked up a bit of a head cold recently, so expect the article on GrapheneOS maybe end of this week as I try to take things a little easier than the last few days which had me burning the candle at both ends perhaps a little too much. The trouble with fiddling with LLMs is that it’s very prone to the ‘just one more try’ effect which then keeps me up late every night, and I’ve had to be up early every morning this week as I am on Juliacare. Here’s looking forward to an early night tonight!

#website #AI #LLM

Go to previous entry

Go back to the archive index

Go back to the latest entries

Niall’s virtual diary archives – Wednesday 15 October 2025