On AI coding and its discontents
Home » Blog » On AI Coding and Its Discontents
Back in January, I received a note from a senior software engineer in Silicon Valley. He described himself as an AI skeptic who became converted after trying Claude Code for the first time. “Overnight, it changed the way I do my job,” he wrote. “It’s really, really good.”
As he explained, he no longer used a standard development environment. Instead, he “exclusively uses Claude Code” to get the job done, interacting with the tool in a terminal window and allowing it to program on his behalf.
“If I had to guess,” he concluded, “I’d say a task that would have taken me a week now takes me 2 days.”
This past winter, when I surveyed more than 300 software developers to learn how AI was transforming their jobs, the majority told a similar tale of shifting from writing their own code to instructing AI agents. The speed with which this new tool became ubiquitous in this industry was stunning.
This story matters for the rest of us because AI coding tools have emerged as the prime example of the power of AI—the first step of many more soon to come on this technology’s disruptive march through our work and our lives.
But what if the reality here is more complicated?
Last week, I received a new message from that same senior engineer who wanted to share an alarming addendum to his tale…
“I’m writing to give you an update on my current thinking about the state of AI in software engineering,” he began, “because my attitude has shifted quite a bit.”
He told me that features he generated using Claude Code ended up crashing their product on two different occasions. His boss told him that if it happened one more time, he’d be fired. “I’ve never had quality issues like this before in my career.”
The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems. As a result, you should carefully review your agent’s output, but this is difficult. As the engineer told me, it’s “famously hard” to understand code you didn’t write yourself, so this extra step becomes “easy to just blow it off (especially when we are all trying to ‘10x’ our velocity).” Soon, systems start to break.
“The coding harnesses are useful and make life as a developer easier,” he summarized, “but they also encourage laziness.”
In response to these issues, this disillusioned engineer has returned to largely programming by hand. Here’s how he explained his current philosophy:
“Writing your own code, slowly but surely, and using LLMs for narrow or particularly annoying tasks (say like writing tests or throw-away scripts), is the best way to produce the highest quality code, since it’s the only way to properly understand it.”
Here’s the thing: he’s not alone.
I increasingly hear similar rumbles from many other people in the software industry (see, for example, this podcast episode from May). Tools like Claude Code can feel like magic, but the strategy of outsourcing all code production to AI isn’t currently sustainable.
In addition to reliability issues, it often engenders a mind-numbing workflow and an environment where junior developers will never acquire the expertise to become senior developers capable of designing complex systems.
Meanwhile, as the frontier labs reduce their subsidies on underlying computing costs, the old habit of burning through as many tokens as possible in search of workable results is proving prohibitively expensive.
From the outside, software development seemed like the poster child for AI’s potential. On the inside, it’s a mess.
This doesn’t mean that coders will abandon AI; its facility with programming languages is too valuable to ignore. But I think there’s a lot more work to be done trying to figure out how to integrate AI into this industry in a way that actually works.
This is a key point.
This last year has been exhausting. The PR departments of the frontier labs have done an excellent job convincing us that AI developments are occurring at an astounding, world-changing rate. But if you zoom out, it becomes clear that almost every “breakthrough” since last summer has concerned the narrow domains of computer code and math, which are defined by highly structured languages and come accompanied by massive amounts of specialized training data.
And yet, even in this best-case-scenario setting for AI, we’re still struggling to figure out how to actually use these tools in a way that makes sense in the long run.
This doesn’t mean that AI doesn’t work or is useless. But it does emphasize an important truth: AI is not a magic “infinity machine” that can solve all our problems, and ultimately deliver us a sense of meaning in a cold, confusing world. It’s a normal technology, and perhaps it’s time we start talking about it that way.
13 thoughts on “On AI Coding and Its Discontents”
-
I can only second that sentiment.
I was one of the developers who wrote to you about how much I love building things with Claude Code.
That was the honeymoon phase.
I was mostly building my own project (green field) and I used it for my angular frontend as well…
It’s pretty good at generating the first features fast. And it sounds reasonable when you brainstorm ideas with it.
It’s also pretty good with angular frontends when components are not too complex.
But I have since used it for production code of our very very complex reinsurance contract software.
It fucked up every time.
Our codebase is just too complicated and messy, it seems.
The reasoning for changes sounded very reasonable and the code does look good as well.
But I have had to go back and redo most tickets I used it for.
And looking at the actual changes after taking the time to deeply understand the logic I was changing made me embarrassed.
One example: The changes cc and I made for the biggest ticket, worked for this one case and broke all other cases.
I am now much more narrow and careful with how I use these tools.
The only AI tool I love and still use often is curser’s tab completion. Which is still fun.
But now I only use “agentic coding” for scaffolding of DTOs and things like that.
Low risk, annoying changes. Like in your example as well.
At this point I have a hard time seeing the business case for these tools.
Thank your for your work, Cal 🙂
And for being out there and speaking about this reasonably.
All the best 🙂 Reply
-
I was not part of your survey, but I am a professional software developer who has been doing this work for over four decades, and am full-time employed today. So, you could say that I’ve had a front seat to the evolution of coding tools over the years. LLM coding agents are fundamentally different.
Prior to them coming on the scene, all coding tools were deterministic encodings of the hard work of other programmers — programmers with shared human understanding and intention. While they were tools which could save me time, not even the best ones could replace the need for me to do the hard work of understanding of how the code I was writing was designed and implemented. And therein lies the crucial difference. LLMs produces code by a process of statistical extrapolation from the vast corpus of coding examples–both well written, and poorly written–delivered up with all of the confidence of an experienced human collogue. But that is a facade. There is no shared human understanding or intention. The LLM is incapable shared intention and understanding, because it is incapable of embodied intention or understanding.
What the programmer you quoted in your post is experiencing is the result of the cognitive offloading to technology, not just of semantic knowledge, but of understanding. The results may look the same on the surface, but the difference hits home when things go wrong. Reply
-
I’ve been going through a similar experience. I confirm all of the above, and want to add that the more I rely on AI to do my job, it also becomes increasingly boring, as I find myself being mostly a tool operator without much agency or mental involvement over the tasks I’m performing.
I often noticed that, although the AI output works most of the time, the quality is lacking (even with flagship models such as Fable 5). I got used to simply vibe coding features, which work just fine, but once I started to actually review how it is implemented, I sometimes found myself feeling ashamed as it does things in a works-but-poor way I’d never do myself.
The feeling of becoming dumber is real! It’s like I can’t start implementations on my own anymore, and I feel almost afraid to write code manually because I feel less confident in myself.
It’s all relative, though. It’s good at doing code reviews, and often fantastic at doing things that humans would never commit on doing, and now we can “because AI will get it right”, like one time when a coworker wanted to introduce a whole library because of one complex component it does well, and I pushed back, and used AI to rebuild the lib’s method manually, which it did in ~1000 lines (much smaller risk/footprint than the lib would have), with a better end result UX-wise. Reply
-
Thanks for the update! I’m a copywriter, and my wife is a graphic designer. Do you have a take on the effects of so-called AI tools on graphic design? Reply
-
+1 @Lem Bach; Would love to hear your opinion on this on the show. Keep up the great work Reply
-
I also emailed you about using Claude all the time in January and I use it more now. Reply
-
I think we need to prioritize domain specific automation with clear bounds on functionality. Automated trains on the DC Metro are great for local travel, but it’s obviously that I can’t use a train as a chatbot for dinner recipes.
I write 100% of my code by hand without any form of LLM AI assistance. It’s surprising to see how quickly LLM AI assistance has become popular.
None of my work has been produced with LLM assistance with two exceptions. One is demonstrating how I think LLMs don’t make sense as a technology. The second is trying to understand what they can do. Reply
-
There is also the brutal and underspoken effect on the ability of people to handle the tasks they used to handle before outsourcing this to the chatbot.
I was one of the very early adopters just after they got measurably good, and ended up in a chatbot driven workflow for a good six months. I completely lost the ability to do my job, largely thanks to ADHD. I ended up having to reteach myself how to plan and work again.
As it is, there were some things that they did well. Namely, I now plan with self talk and i actually structure my restarts with essentially a prompt for myself. So it’s not all negative.
However, I don’t use AI because it is actively dangerous to my ability to stay employed. It cannot produce code for me. It is severely damaging the ability of people to think. Reply
-
I really don’t see this at all in my work. Maybe because I’m largely working on greenfield projects. I’ve built all sorts of things with Claude Code since the start of the year and they’re all running fine and in a few cases bringing in real revenue. But I don’t stop at getting LLMs to do the implementation, I also have them do code review and testing. Use Codex to review Claude’s work, that kind of thing.
And if bugs appear in production, as also happens with human-written code, I get the LLM to fix those too.
I don’t really get people’s complaints of boredom, I’m having a lot of fun doing more at a greater scale than I could have before. I do product work, I have back and forths with the AI over architectural decisions, I’m busier than I’ve ever been. Reply
-
[ The following reply was sent to me by someone having trouble posting it, so I am posting it on their behalf ]
I have also been working on greenfield projects.
LLMs allow us to quickly figure out what is worth building for the customer.
Greenfield gives you space to create and structure projects to make it is easy for LLMs to write code that works. This code has been running in production for months without anymore issues than human written code.
The most important step in the whole process is to making sure you have spoken to your stakeholders, customers, designers, etc and design a high level solution. Once you have the solution you can use the LLMs to explore tradeoffs and look into any unknowns that arise.
In order to maintain enough knowledge of the system/project, there is a balance that needs to be struck. I have to know, when I do not know how something works (system, concept, method, etc) . Those are the moments to build deeper understanding via search and discussion with the LLM so that I know enough to achieve the outcome I want.
Another outcome I may want is to develop a sound mental model for how every piece of the code works. In order to do that I would write every line.
The key is to know when you need to transition between knowing every single detail and knowing enough to get the LLM to do the correct thing (e.g. having unit tests for the LLM to check against) Reply
-
I recently transitioned back from being an engineering manager to an individual contributor. Since I’m pretty rusty at coding, I figured I would fully embrace the Agentic Software Development Lifecycle (SDLC) our leadership is pushing and try not writing code by hand anymore – my job would be to orchestrate the agents and I would spend my time doing the high level engineering and architectural work! I spent some time fixing a nasty bug that was in part of the project that was new to me. Using Claude Fable I had results immediately – but I spent the next several days carefully reviewing the code before submitting it for my colleagues for review. I did end up asking for a few adjustments, and my code breezed through review. That was quite satisfying, because the code reviews on this part of the project are quite stringent.
The next week in my 1:1 with my manager, I tried to walk him through how I fixed the bug – and I could barely explain it! Highly unusual for me – I don’t suffer from anxiety or anything like that – I simply didn’t understand “my” work!
I’m going back to coding by hand. Reviewing LLM written code, no matter how carefully, just doesn’t result in me developing the mental model of the code that I need to have to be an effective engineer. I doubt I’m unique in this regard. Reply
-
Junior Engineer here (somehow got hired despite the cooling junior hiring market), my opinion on AI has also matched what this engineer experienced. In the early days, I was a quick adopter and able to put together quick proof of concepts that gave me a good reputation on the team as someone who can build.
But I’ve been reflecting on “So Good They Can’t Ignore You”, thinking about developing rare and valuable skills and career capital to land the job I want, and I realized that I haven’t been doing my mental curl-ups while programming–reaching for AI too quickly and not spending the time being uncomfortable doing deliberate practice. I’ve been struggling to land a higher-level position at other companies despite my experience.
I’ve seen this called “deskilling”, whether through programming, writing, or other deep-work skills (in my case, “never-skilling”). I think this topic would be interesting for Newport to talk about on his pod. Reply
-
Si la IA es el futuro ¿Que se recomienda a los jóvenes estudiar para esto? Reply