Previously, I said that ProgrammerBot could eventually generate a functionally correct program with enough training against human testers. I lied.
ProgrammerBot can generate an overfitted model that responds correctly to the QA Engineer’s tests, but not much else. ProgrammerBot’s code quickly fails in a production environment.
A human software developer operates something like this:
- Come up with some examples of the problem you’re trying to solve.
- Make a set of rules that work for all the examples.
- Simplify the rules.
- Repeat Steps 1-3 until sufficiently good.
- Hide all evidence of Steps 1-4 so it looks like the solution was immaculately conceived from your brilliant skull.
ProgrammerBot can handle Step 2. Often Step 3. ProgrammerBot can’t perform Step 4 because it has no idea what problem you’re actually trying to solve.
The human engineer often doesn’t understand the problem either, not until Step 5. Developing the solution is the process of understanding the problem.
When non-programmers ask why there doesn’t exist a program that self-authors software, what they really mean is “Why isn’t there a tool that turns my human problems into an app?”
We already have fantastic tools for translating human problems into code; they’re called compilers. And unless you program in assembly, you’re writing in a human language.
The reason there doesn’t exist a program that creates software to solve your problems is because you’re incapable of expressing your stupid problems. Understanding the problem is 99% of the solution.
Why not do what the machine learning folks do and have some validation examples to check whether the problem is overfit?
What if it then overfits for those validation examples?
Either give it time-out or a beating, depending on your parenting philosophy.