Thanks for writing this up! I have been in software development for longer than most people here have been around, and you are absolutely right, over the last several decades the majority of the work shifted from writing new code to figuring out how to best connect the pieces of the puzzle that is a hodgepodge of APIs, idiosyncratic implementations, poorly documented messages and undocumented gotchas. There is usually a million ways to accomplish the same task, too, all different in terms of performance, scalability, costs and applicability, and knowing which to pick is what differentiates a senior SWE from a beginner.
That said, I expect the whole paradigm to shift in the short order. Even now you can tell ChatGPT something like
and it will generate a working python code, and then execute it!
You can also ask it to generate unit tests for the PreorderToPostOrder() function, and it will do a passable job. You can further ask it to add a unit test for a specific test case, and it will do that, too. You can even go deeper and ask it to figure out what test cases might be missing, and then add them. You can request a performance estimate in the big O notation. It will often make mistakes and hallucinate answers, but it is very close to being at the level of a junior SWE at this point, and in many regards much better. Also, much cheaper. It can automate a lot of mundane work for you, and find answers to questions that would be not easy or time-consuming to look up online. It also excels at documenting your design and code, something human programmers suck at and hate doing. It can also look for common pitfalls and test them.
What I am getting at is, while LLMs is unlikely to replace a senior SWE… at least in 2023, they will eat away at the bottom end, interns and junior programmers. The team leads can instruct the LLM the same way they instruct their team, and increasingly more usable and reliable applications will be popping up. At some point soon, the whole idea of a “high-level programming language” will go the way of “Assembler”. You will talk to LLM in English, it will do the rest. One can call it “prompt hacking”, but it is really a new high-level language that is much more human-friendly.
In your example:
The work was about 80% reading the API’s documentation, 18% configuring my API keys and downloading the example project and things like that, and 2% writing code to hook everything up.
most of the work is actually mundane and automatable, such as extracting information from the API docs, following the standard protocol for API key configuration and creating the glue between APIs. The real work is the remaining 1% that is instructing your LLMule to do the heavy lifting.
Using a code-generating AI instead of a programming language would simply mean that your job is figuring out how to use natural language to specify software instead of a programming language, and that wouldn’t be an improvement.
Well, my contention is that it would be a vast improvement.
Basically, the state of the field I expect to see is that the repositories would not consist of C/python/Java code, but of LLM instructions. Moreover, the LLMs can read these instructions and optimize them, too! Not yet, probably not this year, but soon enough.
Software is hard. Computers are difficult, finicky, alien things. Programming languages are our most promising source of power over them. I imagine a world where, instead of hiring programmers, managers simply tell AIs what they want in plain English, then pat themselves on the back for saving so much on payroll; now the manager is the programmer, and he’s writing code in English: and I laugh to myself heartily.
Well, yes to the first two, for sure. No to the rest. AI bots, not programming languages “are our most promising source of power over them”. And where you “laugh hysterically”, I nod and feel that this time cannot come soon enough. Having started my career handcrafting Assembler and FORTRAN, I would be most gratified to see these monsters and their descendants go the way of the dinosaurs that they are. I might be wrong, and maybe there are some severe obstacles there, but I would give more than even odds that the jobs currently performed by interns, junior and intermediate SWE, QA department, and a big chunk of IT support will fade away in the next 5 years or so.
At some point soon, the whole idea of a “high-level programming language” will go the way of “Assembler”. You will talk to LLM in English, it will do the rest. One can call it “prompt hacking”, but it is really a new high-level language that is much more human-friendly.
A person who can debug a device driver or a distributed system is a person who can be trusted in a Hobbesian nightmare of breathtaking scope; a systems programmer has seen the terrors of the world and understood the intrinsic horror of existence.
— James Mickens
ETA: H/t Eliezer for the link, which he cited in a related context.
Hah, a great link! To be fair, we will not rid ourselves of monsters, we will replace them with different monsters, which may or may not eventually finish us off.
Thanks for writing this up! I have been in software development for longer than most people here have been around, and you are absolutely right, over the last several decades the majority of the work shifted from writing new code to figuring out how to best connect the pieces of the puzzle that is a hodgepodge of APIs, idiosyncratic implementations, poorly documented messages and undocumented gotchas. There is usually a million ways to accomplish the same task, too, all different in terms of performance, scalability, costs and applicability, and knowing which to pick is what differentiates a senior SWE from a beginner.
That said, I expect the whole paradigm to shift in the short order. Even now you can tell ChatGPT something like
and it will generate a working python code, and then execute it!
You can also ask it to generate unit tests for the PreorderToPostOrder() function, and it will do a passable job. You can further ask it to add a unit test for a specific test case, and it will do that, too. You can even go deeper and ask it to figure out what test cases might be missing, and then add them. You can request a performance estimate in the big O notation. It will often make mistakes and hallucinate answers, but it is very close to being at the level of a junior SWE at this point, and in many regards much better. Also, much cheaper. It can automate a lot of mundane work for you, and find answers to questions that would be not easy or time-consuming to look up online. It also excels at documenting your design and code, something human programmers suck at and hate doing. It can also look for common pitfalls and test them.
What I am getting at is, while LLMs is unlikely to replace a senior SWE… at least in 2023, they will eat away at the bottom end, interns and junior programmers. The team leads can instruct the LLM the same way they instruct their team, and increasingly more usable and reliable applications will be popping up. At some point soon, the whole idea of a “high-level programming language” will go the way of “Assembler”. You will talk to LLM in English, it will do the rest. One can call it “prompt hacking”, but it is really a new high-level language that is much more human-friendly.
In your example:
most of the work is actually mundane and automatable, such as extracting information from the API docs, following the standard protocol for API key configuration and creating the glue between APIs. The real work is the remaining 1% that is instructing your LLMule to do the heavy lifting.
Well, my contention is that it would be a vast improvement.
Basically, the state of the field I expect to see is that the repositories would not consist of C/python/Java code, but of LLM instructions. Moreover, the LLMs can read these instructions and optimize them, too! Not yet, probably not this year, but soon enough.
Well, yes to the first two, for sure. No to the rest. AI bots, not programming languages “are our most promising source of power over them”. And where you “laugh hysterically”, I nod and feel that this time cannot come soon enough. Having started my career handcrafting Assembler and FORTRAN, I would be most gratified to see these monsters and their descendants go the way of the dinosaurs that they are. I might be wrong, and maybe there are some severe obstacles there, but I would give more than even odds that the jobs currently performed by interns, junior and intermediate SWE, QA department, and a big chunk of IT support will fade away in the next 5 years or so.
When will it replace the Night Watch?
ETA: H/t Eliezer for the link, which he cited in a related context.
Hah, a great link! To be fair, we will not rid ourselves of monsters, we will replace them with different monsters, which may or may not eventually finish us off.