Frog Twaddle

As I was cleaning up some of my older files today and moving them off device, I wondered if I could use AI to build a manifest that I could keep on device in case I ever needed to find one of the folder files again. As someone who generates a lot of files, it's important I have good drive hygiene and I'm always looking to up my game.

Getting a list of the files into a document is easy. You can simply list a directory and redirect it into a file. On Mac this can be done with something like,

ls -lh >> myFiles.txt

It's very basic but it does create a simple file list that can be easily searched. Sometimes, with me anyway, the file name is not enough for me to recall what information a document contains. This is where I thought that bringing AI into the mix may help up my game.

After some minor fiddling, I came up with the following,

#!/bin/zsh

# Define the output file
output_file="output.md"

# Clear the output file if it exists
> "$output_file"

# Find all Markdown files recursively and process them one by one
find . -type f -name "*.md" -print0 | while IFS= read -r -d '' file; do

  # Run the temp command with the content of the current file
  result=$(cat "$file" | ollama run llama3 "Briefly summarize the text in this file. The summary should be no more than 3 sentences in length. Suppress any response that is not part of the summary. If the file is empty just respond with the phrase 'Empty File'")
  
  # Append the formatted output to the output file
  echo "---\n" >> "$output_file"
  echo "**${file}**" >> "$output_file"
  echo "\n" >> "$output_file"
  echo "$result" >> "$output_file"
done

In this case, the AI I am using is Ollama with the llama3.1:8b model. In my experimenting, this model provided decent summaries with the correct prompt. As you can see in the code block above, I give the model the following prompt.

“Briefly summarize the text in this file. The summary should be no more than 3 sentences in length. Suppress any response that is not part of the summary. If the file is empty just respond with the phrase 'Empty File'”

I've found that the llama3 model can be a bit chatty so I give it multiple guardrails to keep it on task. One: be brief. Two: limit the response to 3 sentences. Three: don't provide anything other than a summary. Four: if the file is empty, just say “Empty File” and move on. I will say that even with those guardrails, Ollama will still occasionally give some commentary, but it's knocked down enough that the output is useful and only requires some minor cleanup. In a future revision, I'll probably refine the prompt as well as provide some hinting so it has a format to follow.

As I see it now, this approach could be easily adapted to solve multiple problems and could be adapted to more sophisticated programming languages like Python to do even more. For now, though, I've scratched my itch and it's time to archive some files.