My experience using LLMs

TL;DR

This article isn't going to be about how to optimise your AI usage. I'm no expert and the only person who can figure out how to use an LLM effectively in your own workflow is you. In this post, I talk about how I don't really like using LLMs for code generation as I end up spending lots of time iterating on the generated code that is spat out by an LLM by either manually editing the code or telling the LLM what to change and iterating on that. Instead, I think its more valuable getting an LLM to build tools and automations to help automate tasks and that by doing this, instead of relying on lots of prompting or skills for the AI agent you're using, your time would be better well spent getting the LLM to configure static analyzers and tooling in the domain you're working in to help reduce friction.

Introduction

I've seen lots of talk on how AI is going to be priced out eventually as it is unaffordable to maintain given the current prices that subscriptions are costing and eventually, these services will need to increase the price of their subscription models to stay afloat.

The thing that I don't understand is that lots of the content surrounding AI is "check out this new model benchmark" or "look at the cool new thing I got X LLM model to build" or "check out this post on how to token max" from the crowd that endorses AI usage. From the other side, I've seen lots of posts from developers feeling like their skills are deteriorating overtime by using AI.

Outside of the Primeagens content, I haven't seen any content creators talk about their negative experiences using AI or how they are trying to "get the magic back". I mention the Primeagen specifically as some of his videos recently have been very candid about how using AI has made him feel less joy when programming as the aspect of programming is part of the joy that he experiences when solving problems and handing that off to an LLM takes away some of that joy (there are other reasons but I recommend checking out his channel and watching his content as his views are very insightful). To solve this, he hasn't been using LLMs to just generate production code for him, he's been getting LLMs to generate tooling to help him accomplish his goals like building new agent harnesses to interact with LLMs the way he wants to fit his workflow.

From my perspective, if you think a new harness will help improve your development workflow, try building one and see how it fairs. The thing I want to focus on isn't production code generation or making LLM harnesses. I think what we should be focusing on is building tooling to help support us and reduce friction of our daily workflows.

Context

The other day, I was chatting to a colleague about how I am feeling less joy because I am reviewing code all the time and iterating on code written by something else. It was only until my colleague told me "You can just use LLMs for the things you don't enjoy doing" and it really resonated with me. I don't need to use an LLM to generate code and iterate on it just because its available, I can just hand off to an LLM to do the things I don't enjoy doing. Thats when I remembered how the Primeagen uses LLMs and I thought, I don't like having to tell an LLM to structure something in a particular way/to follow naming conventions in a project and other things that are mildly frustrating. I know that these things can be fixed by creating a skill that the LLM can use for context but there had to be a better way.

Then I remembered, static analysis tools exist. One example I can think of is, in our team, we use Agrona collections to avoid boxing around primitives. A common problem would be that an LLM may generate some code without using Agrona collections even though its a project convention. To resolve this, instead of creating a custom skill to always use an Agrona collection where possible, we got the LLM to create a custom errorprone rule to enforce this in our application. Now, using boxed collections requires suppressions and the LLM can pick this up automatically as its built into our build system without us having to prompt it to generate the code following our projects conventions.

This is very Java specific but, theres probably a way to automate the things you're getting an LLM to do without using skills and just by using tooling that can be plugged into your build system or as a git hook. Before, these types of endeavors would be seen as too much effort to dedicate time to but with LLMs, you can have them working on it in the background while you're working on meaningful tasks. There are other things we have got an LLM to do like writing git hooks to automatically run different static analyzers in other projects that follow different style conventions (ansible, terraform etc). Skills tied into your AI agent could achieve the same purpose but, when we are eventually priced out of using LLMs, all these skills will be redundant while, these tools will still be available (given they are still maintained in the future).

Conclusion

If you enjoy using LLMs to generate code for you, thats fine. If you dont, thats fine too. I'm not here to police how you use it or what you use it for. I think its worth looking at your current workflow and wondering, can some of this be done outside your LLM to reduce the friction in your workflow.

Try asking this in your next agent session and see if it gives back anything useful:

Using our conversation history, what tooling can I tie into our build system or through a git hook to automate the things I keep correcting you on

Trying out Qtile

QTile

As of writing this, I'm currently going through the "Obey the testing goat" book from oreily to refresh my knowledge on python as the last time I touched it was back in university. As i'm touching python, I thought I might as well try ricing qtile

Booting up qtile

The first time I booted into qtile, I was greated with the same screen as my greater (SDDM) but with a bar at the bottom of the screen. Not entirely sure why that happened but I could at least spawn a terminal to take a look at the initial config. The config was very bare bones, in contrast to awesome and i3 where you are given a lot of keybinds out of the box, the qtile config just has the essential keybinds to help you start using the window manager. I dont particularly mind this as I have keybinds from other configs that I can use to base my current keybinds off of but for a new user, they will probably need to look at documentation or other examples to know whats possible.

Setting a background

Normally, I use feh to set the background on all of my monitors as it works fine for i3, leftwm and spectrwm (that articles coming soon) but for some reason, qtile wasnt able to render the background at all and would default to giving me a blank background entirely. This was especially weird when I tried to move windows and would get a "ghosting" effect with the old windows position still appearing on screen. This behaviour would also happen when I closed windows where the screen would still show the previous window that was open but opening a new window in full screen would then override the image rendered as the background. To fix this, I set my background in the qtile config for each screen to load my wallpaper using:

Screen(
    wallpaper="path/to/wallpaper.png",
    wallpaper_mode="fill",
    top=bar.Bar(...widget_list_here...),
)

Now my backgrounds at least look ok.

Configuring screens

Besides stealing the bar from CozyTile (I highly recommend checking it out, the themes look amazing), the rest of my config is pretty much stock. I split out my config into separate files but I havent used classes to handle my config even though its written in python. Thats more personal preference as I prefer the classes java has and I never really liked how OOP was implemented in python but moving your config to an OOP style shouldnt be too difficult, I recommend checking out qtile-examples for inspiration for your own qtile config if you want to try customizing qtile. The only other thing I stole was some multi monitor set up code as by default, qtile doesnt have an example of setting up multiple monitors with bars and backgrounds in their example config so, I used g-wizzy's config to dynamically grab the number of monitors I have usign xdisplay.

Pain

The one problem I have with qtile and lots of window managers are the way tags are handled. I like how awesome has tags for each screen so you could have 18 tags with 9 tags on each screen. Although I dont like sharing tags between all screens, I like how i3 handles that where when you select the tag that is in focus on a different monitor, it takes you to that monitor instead of moving the tag to the current monitor you are on. Qtile on the other hand, works exactly like leftwm where you bring the tag to you instead of you moving to the monitor the tag is on. I find this very frustrating as my screens arent always allocated the same tags every time I boot into qtile so when I select a tag that I think is on a one monitor, it'll take me to the other monitor. This would probably be fine if you always open the same windows on the same tags but I am not as organized with my placement of windows on tags.