Frog Twaddle

I will not give up my power. I read this on the small web today and it helps. Maybe someone else will find this useful.

I Am Part of a Whole

As I was cleaning up some of my older files today and moving them off device, I wondered if I could use AI to build a manifest that I could keep on device in case I ever needed to find one of the folder files again. As someone who generates a lot of files, it's important I have good drive hygiene and I'm always looking to up my game.

Getting a list of the files into a document is easy. You can simply list a directory and redirect it into a file. On Mac this can be done with something like,

ls -lh >> myFiles.txt

It's very basic but it does create a simple file list that can be easily searched. Sometimes, with me anyway, the file name is not enough for me to recall what information a document contains. This is where I thought that bringing AI into the mix may help up my game.

After some minor fiddling, I came up with the following,

#!/bin/zsh

# Define the output file
output_file="output.md"

# Clear the output file if it exists
> "$output_file"

# Find all Markdown files recursively and process them one by one
find . -type f -name "*.md" -print0 | while IFS= read -r -d '' file; do

  # Run the temp command with the content of the current file
  result=$(cat "$file" | ollama run llama3 "Briefly summarize the text in this file. The summary should be no more than 3 sentences in length. Suppress any response that is not part of the summary. If the file is empty just respond with the phrase 'Empty File'")
  
  # Append the formatted output to the output file
  echo "---\n" >> "$output_file"
  echo "**${file}**" >> "$output_file"
  echo "\n" >> "$output_file"
  echo "$result" >> "$output_file"
done

In this case, the AI I am using is Ollama with the llama3.1:8b model. In my experimenting, this model provided decent summaries with the correct prompt. As you can see in the code block above, I give the model the following prompt.

“Briefly summarize the text in this file. The summary should be no more than 3 sentences in length. Suppress any response that is not part of the summary. If the file is empty just respond with the phrase 'Empty File'”

I've found that the llama3 model can be a bit chatty so I give it multiple guardrails to keep it on task. One: be brief. Two: limit the response to 3 sentences. Three: don't provide anything other than a summary. Four: if the file is empty, just say “Empty File” and move on. I will say that even with those guardrails, Ollama will still occasionally give some commentary, but it's knocked down enough that the output is useful and only requires some minor cleanup. In a future revision, I'll probably refine the prompt as well as provide some hinting so it has a format to follow.

As I see it now, this approach could be easily adapted to solve multiple problems and could be adapted to more sophisticated programming languages like Python to do even more. For now, though, I've scratched my itch and it's time to archive some files.

I was recently asked this question by my wife's dad. I decided it might be fund to do a little back of the napkin style math. Below are my calculations.

To solve this problem, we need figure out what a “standard” printed page consumes in storage. According to Microsoft, an average typed page on US Letter-sized paper (8.5 x 11 inches) typically contains about 2,500 to 3,000 characters, including spaces. This estimate can vary based on factors such as font size, typeface, and spacing. For example, using 12 pt Times New Roman with standard margins, you might expect around 2,500 characters per page. For our estimate, we’ll use 3,000 characters to be conservative and account for smaller font sizes.

Next, we need to determine how much storage each individual character requires. MacOS (my platform of choice) and other modern operating systems support a wide variety of encodings, but the primary encoding on MacOS is UTF-8. A regular text character is 8 bits (1 byte), but to support other characters like emoji, Kanji, etc., we need twice that. Therefore, UTF-8 uses 16 bits (2 bytes) per character for these cases.

Now we can calculate the total storage capacity. Five gigabytes is equivalent to 5 billion bytes or 40 billion bits. Since an average page contains 3,000 characters (or 48,000 bits), 5 gigabytes of storage can hold over 800 million (833,333,333) US Letter pages of text.

For fun, according to the blog Reedsy, the average number of characters in an English-language novel can vary significantly based on factors such as genre and writing style. However, a rough estimate can be made using the average word count of a novel, which typically ranges from 70,000 to 100,000 words. Assuming an average of 5 to 6 characters per word (including spaces and punctuation), we can estimate the total character count as follows:

For 100,000 words: 100,000 words × 5.5 characters/word ≈ 550,000 characters or ~9 million bits. This means our 5 gigabytes of storage could hold roughly 4,444,444 novels.

Is the above useful? No. Accurate? Meh. Fun to consider? I thought so!

Kris and I cleared Alberta today and tipped into Saskatchewan which, in my opinion, is the most fun Canadian Provence to say out loud. Getting to our final spot for the night though required that we cover some serious ground. As of this evening, we're more than 2,000 km into our trip and we saw some pretty neat things along the way. If you were to look at a map of the North American continent, you'd see that we're the equivalent of only about ⅔ across Montana. This is somewhat surprising as the landscape is more like what you'd see in South Dakota except much more green. The wind is similar to what you'd expect in the prairie as well and we had make way against a headwind that I'm sure didn't help our gas milage. Thankfully, the 4Runner is more than up for the task and covered the undulating hills with little difficulty.

One of the highlights of the day was an accidental discovery of the Battelford Tail Ruts which just happened to be near a gas station and a field where we could let Denali stretch his legs. According to the plaque,

From 1882 until 1891 Swift Current was the closest railway shipping point for the populous Battleford district. Trade, especially the buffalo bone trade, created a heavy volume of ox-cart traffic between the two centres while Battleford was under siege during the Riel Rebellion, Swift Current was strategically important to the entire country.

And so, you can see the actual ruts from the Ox Carts more than 100 years ago today!

The other nice thing about this site... a geocache. This was my first Canadian Geocache which earned me the Canada AND Saskatchewan souvenirs. Sweet!

We also passed the time listening to a documentary-style podcast called Wilder which talks about the Little House on the Prairie books, Laura Ingall's Wilder, her relationship with her daughter, and the legacy of those books and their lives. It was quite appropriate given the day's scenery.

Finally, we ended our day at a cozy camp site called Bin There. Kris and I were both happy to not have to set up the trailer for an evening and instead get hot showers, wash laundry and setting into a real bed for the evening. In fact, I am writing this as a run our laundry through the wash! Below is a quick shot of our digs for the evening.

And with that... I'll call it an evening! Stay turned for the next installment!

As I write this, Kris, Denali, and I are settled in for our third night of camping. Of course, my intent was to blog every day but a full day’s drive followed by setting up camp in the rain (all three nights so far), and wrangling dog have made an evening of blogging slightly ambitious. I’m happy to report, however, that our trip across the Trans Canada Highway has so far been fantastic. We’re more than 1,000 km into our trip and we’re passed the Canadian Rockies. As someone who spent a year of his life in Colorado, the Canadian Rocky Mountains have a grandeur and magnitude all their own. As we wound our way through Roger’s Pass we were treated with endless views of rivers, steep cliffs, and signs warning of mountain goats and avalanches. Alas, I could not take pictures of these things as I was driving, but I did grab some shots from the places we stopped. This marker has a little history of the pass’s namesake.

As it happens, it’s also Canada Day!!

I’ll share more details from the trip in the next couple days. For now, these are two last shots from the start of our trip in WA at Oostema Farm.

This first shot is Denali and I walking out for our morning walk before we crossed the border and embarked (see what I did there) on a long drive.

This second shot is from the end of that day’s drive. Life on the road can be hard.

One final thought. Please be forgiving of this post. I typed it out on my iPhone. : )