TL;DR
This article isn't going to be about how to optimise your AI usage. I'm no expert and the only person who can figure out how to use an LLM effectively in your own workflow is you. In this post, I talk about how I don't really like using LLMs for code generation as I end up spending lots of time iterating on the generated code that is spat out by an LLM by either manually editing the code or telling the LLM what to change and iterating on that. Instead, I think its more valuable getting an LLM to build tools and automations to help automate tasks and that by doing this, instead of relying on lots of prompting or skills for the AI agent you're using, your time would be better well spent getting the LLM to configure static analyzers and tooling in the domain you're working in to help reduce friction.
Introduction
I've seen lots of talk on how AI is going to be priced out eventually as it is unaffordable to maintain given the current prices that subscriptions are costing and eventually, these services will need to increase the price of their subscription models to stay afloat.
The thing that I don't understand is that lots of the content surrounding AI is "check out this new model benchmark" or "look at the cool new thing I got X LLM model to build" or "check out this post on how to token max" from the crowd that endorses AI usage. From the other side, I've seen lots of posts from developers feeling like their skills are deteriorating overtime by using AI.
Outside of the Primeagens content, I haven't seen any content creators talk about their negative experiences using AI or how they are trying to "get the magic back". I mention the Primeagen specifically as some of his videos recently have been very candid about how using AI has made him feel less joy when programming as the aspect of programming is part of the joy that he experiences when solving problems and handing that off to an LLM takes away some of that joy (there are other reasons but I recommend checking out his channel and watching his content as his views are very insightful). To solve this, he hasn't been using LLMs to just generate production code for him, he's been getting LLMs to generate tooling to help him accomplish his goals like building new agent harnesses to interact with LLMs the way he wants to fit his workflow.
From my perspective, if you think a new harness will help improve your development workflow, try building one and see how it fairs. The thing I want to focus on isn't production code generation or making LLM harnesses. I think what we should be focusing on is building tooling to help support us and reduce friction of our daily workflows.
Context
The other day, I was chatting to a colleague about how I am feeling less joy because I am reviewing code all the time and iterating on code written by something else. It was only until my colleague told me "You can just use LLMs for the things you don't enjoy doing" and it really resonated with me. I don't need to use an LLM to generate code and iterate on it just because its available, I can just hand off to an LLM to do the things I don't enjoy doing. Thats when I remembered how the Primeagen uses LLMs and I thought, I don't like having to tell an LLM to structure something in a particular way/to follow naming conventions in a project and other things that are mildly frustrating. I know that these things can be fixed by creating a skill that the LLM can use for context but there had to be a better way.
Then I remembered, static analysis tools exist. One example I can think of is, in our team, we use Agrona collections to avoid boxing around primitives. A common problem would be that an LLM may generate some code without using Agrona collections even though its a project convention. To resolve this, instead of creating a custom skill to always use an Agrona collection where possible, we got the LLM to create a custom errorprone rule to enforce this in our application. Now, using boxed collections requires suppressions and the LLM can pick this up automatically as its built into our build system without us having to prompt it to generate the code following our projects conventions.
This is very Java specific but, theres probably a way to automate the things you're getting an LLM to do without using skills and just by using tooling that can be plugged into your build system or as a git hook. Before, these types of endeavors would be seen as too much effort to dedicate time to but with LLMs, you can have them working on it in the background while you're working on meaningful tasks. There are other things we have got an LLM to do like writing git hooks to automatically run different static analyzers in other projects that follow different style conventions (ansible, terraform etc). Skills tied into your AI agent could achieve the same purpose but, when we are eventually priced out of using LLMs, all these skills will be redundant while, these tools will still be available (given they are still maintained in the future).
Conclusion
If you enjoy using LLMs to generate code for you, thats fine. If you dont, thats fine too. I'm not here to police how you use it or what you use it for. I think its worth looking at your current workflow and wondering, can some of this be done outside your LLM to reduce the friction in your workflow.
Try asking this in your next agent session and see if it gives back anything useful:
Using our conversation history, what tooling can I tie into our build system or through a git hook to automate the things I keep correcting you on