LLMs Suck at Code

(Est. Reading Time: 2 min)     

So, here I am, tinkering with ChatGPT and other LLMs for over a year now, learning to leverage it for success for software engineering. It needs to be called out. LLMs suck at code.

If a chat gets too long, they start to introduce artifacts and oddities into the code. For example, multiple unconditional return statements in a single function, adding a new one every time it "helps". It does have its good moments, though, as it helps discover bugs, sometimes. Ask it how to do something you've never done, and it will give you a basic understanding to work on. But it does not have the same logical mentality to actively participate in writing code.

It gets worse. OpenAi researchers found that, beyond face value simple code, LLMs (even the unreleased frontier models) suck at code. Once the code starts to grow beyond a few classes, or a little complexity between different functions, it cannot hold the context in a way to provide meaningful output. Basically, if you break the code-writing process down into very tiny and simple fundamental functions, and have an LLM assist one at a time, you might have more productivity. But once you try to solve a complex bug or code change spanning multiple functions, classes, or files, the LLM breaks down quickly.

Even LLMs that specialize in code fail to do better than junior level understanding. This isn't a training issue, either. It's science, math, and  distributed system limits. You have top FAANG and Big AI companies investing billions, each, to double-down on AI. They are building massive power-hungry AI manufacturing centers in places like Texas, and it looks like they are attempting to solve the AI shortcomings by scale. However, just because you add more "processors" and "memory", it doesn't mean the math will scale to utilize it properly. That is just not how any of this even really works.

But hey, business leaders, if you want to wholeheartedly fire your software engineering staff and replace them with AI, or build your whole business upon the success of what AI can do today, be my guest and find out the hard way. Thin gilded clouds obscure your vision, and sometimes businesses just need to be allowed to fail.

As for us software engineers, we have found the most productive way to use LLMs: using them to speed up research and problem-solving. You know, just as long as the LLM doesn't write the code.