It also hits its 'maximum limit allowed at this time' quite often due to it's inability to understand its own environment and just modify an existing software artifact rather than regenerating the entire thing. I know it can do it, it knows it can do it because it will tell you why it failed but it just wont do it for some reason. Of course this comes after you manage to convince it to quit removing sections of code so its context window is already polluted from past failures.

I spent a solid three days trying to get it to successfully do a somewhat simple thing (something that would've taken me an afternoon if I dedicated myself to the task) just to see if it could manage to do it and it failed horribly, multiple times, even after it was able to say why it was failing, no matter how many tries I gave it. And it's not like it is some impossible task either, it was able to successfully (might I even say impressively) accomplish the exact same task with no problems at a smaller scale.

It's also pretty horrible at debugging, even if you tell it exactly the problem, but that's just me testing the limits of what it can do.

So, like TFA states, it is pretty amazing at smaller tasks but utterly and completely fails at anything larger than a couple hundred lines of code.

The thing is it starts to fail when you are just prompting it without any example code.

And to fully prompt it to be exactly what you wanted to write is basically writing the code yourself.

So for novel code I have had far less success.

But any kind of translation or matching an API shape, it works pretty much easily.

Also Deepseek R1 can do this for much much cheaper and at virtually the same level of quality.