As generative AI models get better, AI tools become a greater part of our everyday lives. Humans can begin to automate parts of their knowledge work not before possible. Employers have greater incentive to push employees to use AI for productivity gains. Several companies have done layoffs with AI efficiencies as the primary motivator. Yet, this efficiency does not always match the gain experienced by the knowledge worker. This is because gain experienced from AI depends on the following factors:
- Did the user prompt correctly, with enough details to be able to achieve the task?
- Is the prompt by the user in the training data of the underlying model?
Users can always improve their prompts to get better output. But, if the response the user needs is not in the training data, it is not reasonable to expect a response which achieves the objective. Since users do not have observability on which sources were used to come to an answer, the only way to know whether their task is achievable is by trying it out. In this way, AI use has many of the properties of a slot machine. It is a black box that can do everything imaginable or nothing at all. Exactly how and when to use AI becomes the responsibility of the user. Without observability as to what is and is not useful with AI, we ought to experiment, see where it is useful for ourselves, and find the work we can potentially automate away.
With coding (an area particularly impacted by AI use), I see the following four levels of automation, each with its own pros and cons:
- Human mode (Do everything yourself)
- Best for learning. Do things yourself. Full ownership. AI can be used to explain concepts and assist the human in understanding. Best for work that requires deeper understanding.
- Very slow. Not great for busywork where automation can be of benefit. Less time to work on hobby projects, since you invest more time and energy into a single project.
- Agentic/context engineering
- Prompting a model to generate code, then checking the code. Human controls the inputs (context) given to the model. Human has control over which tools the agent uses. Code review tools to test the code alongside humans. Extensive code review and QA by humans.
- It is difficult to review code written by others as opposed to writing every line yourself, where each line and its purpose is better understood. Faster to implement coding practices such as test driven development, since the creation of code is offloaded.
- Vibe coding
- Prompting an AI model without necessarily looking at the code on each step. Doing QA at the end to test the output for correctness.
- Control over the final output through prompting. A lack of understanding as to what is happening under the hood. At the mercy of the AI model to finish work. Able to manage several projects at once while waiting for AI output. Tools like Conductor become useful.
- Hands off the wheel (claw mode)
- Openclaw, completely automated, runs in the background, no human intervention
- Good for initial prototypes and busywork that is easy to automate. Hard to maintain. No control over ownership. Even more at the mercy of the AI model. Expensive.
If we are a startup founder and want to find product market fit, we should vibe code to see whether a there is market interest before investing additional time into any one idea. If we are students we should learn things deeply. To do so, we should learn by doing the task ourselves rather than offloading this work to an AI model. If we are doing a task for the first time, we should first do it ourselves, so that we can gain an understanding that allows us to give the correct context to the model and understand its output. If we are working in a production environment, we should have great care to review the code, whether generated or not. Applying to jobs, taxes, outreach, and other busywork would benefit us if automated away. Based on our goals, we should have the courage to use our own reason on how and when to use AI.