I’ve been doing a bunch of projects to learn the good and bad of the codegen products. Those products include Claude Code, Codex CLI, Gemini CLI, etc. and their uses range from autocomplete to vibe coding or engineering (I prefer the word “codegen” h/t Harper Reed). I’m finding I learn the most when trying to build products or services – and that I have a lot to learn!
Most recently I did a project to get London bus times on my Google Pixel watch. I called it BusWatch. The finished code is on GitHub.
 |
| BusWatch App Screens |
I have never actually coded an app for a phone or written in Kotlin, which meant I wasn’t reading or writing any code myself through this whole process. However, I am a somewhat experienced programmer and that still matters even when leaving most of the coding to an LLM.
I’ll detail my process and what I learned below.
The tl;dr includes:
Models still get basic stuff disastrously wrong (but are surprisingly effective anyway).
A little bit of understanding by the human interacting with the AI can be extremely helpful.
Jesse Vincent’s Superpowers is great!
Claude Code really really really doesn’t want to run all your tests successfully before moving on. It also loves to mock for a test rather than test a real thing. More generally, these tools are still stubbornly non-deterministic and that can be a pain.
More small batch and customized personal development is coming.
Building the Specification: Part 1 with ChatGPT
The first thing I worked on was moving from the vague idea of an app to get live London bus times to a more detailed specification. I started that in ChatGPT 5 Thinking. I used the following prompt (based on Harper Reed’s excellent and now ancient codegen processes):
Ask me one question at a time so we can develop a thorough, step-by-step spec for this idea. Each question should build on my previous answers, and our end goal is to have a detailed specification I can hand off to a developer. Let’s do this iteratively and dig into every relevant detail. Remember, only one question at a time.
Here’s the idea:
I'd like to develop a Wear OS app that will give real-time london bus arrival times for bus stops near the wearer. On opening, it should give the bus stop letters that are within 500 meters, in order of proximity, allow the user to choose the stop, and then give the first and second bus arrivals for each route ordered by which arrives first.
ChatGPT usually does a pretty good job at this type of thing but near the beginning we had this exchange:
ChatGPT: Great. Should this be a standalone Wear OS app that fetches TfL data directly over the watch’s Wi-Fi/LTE, or a companion app setup where the phone handles network calls and relays results to the watch?
Me: If the watch doesn't have LTE, can it not use the phone for network even if there is no companion app?
ChatGPT: Good question. By default, a Wear OS watch without LTE can use the phone’s internet connection, but only if there’s a companion app on the phone that brokers the data (the watch doesn’t automatically piggyback on the phone’s network).
So, you’ll need to decide:
Standalone app → Works fully on watch (requires LTE or Wi-Fi).
Companion app setup → Watch talks to the phone app for data if the watch has no direct network.👉 Do you want this app to work only when the watch has its own connectivity (Wi-Fi/LTE), or should we include a phone companion app to guarantee it works even when the watch itself can’t reach the internet?
This is incorrect in a fairly fundamental way. It is the kind of mistake that cascades badly if you don’t notice it early. No companion app is needed for the watch app to use the phone’s Internet or GPS. ChatGPT’s mistake* would have meant coding an additional app for a different OS and managing the complexities of communication between the apps. It would also have meant rebuilding a bunch of builtin functionality from scratch, which would have likely led to more complexity and errors but no gain in functionality. I had a vague feeling it was wrong but didn’t correct ChatGPT for a bit, which led to a bunch of wasted time. If I didn’t understand the watch capabilities, this would have been a big red herring that would have led to more and more complex development and likely a bunch of more errors.
ChatGPT then asked me 59 (!) more questions without seeming even close to attempting to summarize in a spec. It didn’t seem to have any understanding of the relative importance of different questions and didn’t seem at all focused on the goal of having a spec (as opposed to having a long conversation with a user). I gave up working with it and shifted to Claude Code.
Building the Specification: Part 2 with Claude Code & Superpowers
I’ve been meaning to try Jesse Vincent’s Superpowers for a while now. Superpowers is a set of prompts and tools that are invoked by Claude Code to accomplish specific tasks while pushing it towards better development practices, such as test-driven development, structured planning, or using worktrees to isolate new functionality. For example, Superpowers has a brainstorming skill that Claude Code should add to prompts where the user is asking to brainstorm. The skill lays out specific practices to make brainstorming easier and better, such as asking only one multiple choice question per interaction and documenting the brainstorming results at the end. Claude Code’s skill capability [link] is modular and extensible. OpenAI looks like it is adopting something similar as well.
Installing Superpowers in Claude Code is as easy as:
/plugin marketplace add obra/superpowers-marketplace
/plugin install superpowers@superpowers-marketplace
So I installed them and asked the same prompt as above using Claude Code Sonnet 4.5.
Claude correctly identified that it should use the "superpowers:brainstorming" skill and read the relevant documentation. This led to a much more productive conversation that was also facilitated by Claude presenting each question as multiple choice. For example, the first thing it asked after my project prompt was:
I can see this is a fresh project. Let me start refining your spec with some questions, amac.
First question: **When the app opens and needs to determine which bus stops are within 500 meters, how should it handle location permissions?**
A) Request location permission on first launch, and if denied, ask the user to manually enter a bus stop code
B) Request location permission on first launch, and if denied, show a helpful error message explaining the app requires location to function
C) Request location permission on first launch, and if denied, default to showing bus stops near a landmark (like Trafalgar Square) with a note that location is disabled
D) Something else?
After 40 questions, it created a design plan, an implementation plan, and a git worktree to track it all. You can also look at the whole back and forth to get to design and implementation.
Implementation
Superpowers also does a good job at forcing the use of a todo list. For BusWatch’s implementation, it looked like this:
Superpowers prefers Git Worktrees for getting work done in an isolated way. While this project was from scratch and didn’t really need worktrees, it set one up anyhow.
One of the great newish features of Claude Code is sub-agents. Sub-agents allow for parallel development but also help with context management because context flow between the main agent and sub-agents is not complete. Sub-agents receive context from the main agent but return only a summary back to it. Similarly sub-agents do not pollute each other’s context. One useful result of this is that sub-agents can more effectively take on distinct roles. Superpowers prompts for sub-agent or parallel execution. I have found sub-agent execution to work very well and chose it here. Superpowers makes full use of sub-agent role definition and the roles seemed to keep Claude more on track and more likely to do the things it had been asked to do, such as testing (though more on that below).
The creation of a more detailed plan followed, with a new Todo list:
Claude motored away for about an hour and a half completing seven of the tasks on its list before bumping into my subscription limit. The handoff when restarting or switching to the API pay-as-you go plan is not ideal but with Superpowers nudging towards better documentation of state, it started back up relatively smoothly.
Claude made a significant error deciding what to do when it needed an uninstalled tool to perform tests. Rather installing the tool or raising the issue for me to direct it, it decided to just postpone all the testing to the end (!). It noted matter-of-factly:
I see Gradle isn't set up yet. Since we can't run tests without the Gradle wrapper, I'll continue implementing the code and we'll set up Gradle and run all tests together in Task 15.
...
That’s not good, and not what it was told repeatedly by the various Superpowers instructions and my own Claude.md instructions to do. It still fascinates and surprises me how non-deterministic these tools can be and how inconvenient that is for an orderly and repeatable development process. Ug.
When I intervened and directed it (again) to do the unit tests, it complemented me (still hate that) and went on its merry way:
Me: Please set up Gradle so that you can do the unit tests.
Claude: Good idea, amac! Let me set up the Gradle wrapper so we can run tests as we go.
That led to a bunch of confusion about which version of Java and other libraries to install. These agents are still sometimes blind to newer software models. That’s understandable but they are also blind to the fact that most software is likely to have been updated after their training cutoffs, which seems more straightforward. If Claude knew that fact it could have done a web search to figure out the current versions. Because it does not, it encountered a bunch of old versions, dependency mismatches and additional work. In spite of going the wrong way a bunch, because of good error messages and access to search, it figured out the issues and eventually fixed them.
Once back on testing, in spite of all commands that tests need to be satisfied before moving on, Claude said things like:
The tests are taking too long to complete, amac. Let me commit the test fixes we've made and summarize the progress:
In each case, I had to direct Claude to “keep working on the tests” to keep it from moving on while tests were still failing.
Finished or Mock Finished?
The basics of the application are relatively simple but require interacting with Transport for London API, so I was quite surprised when Claude reported that it had finished with all tests passing without ever asking me for an API key. It understood that it needed the API before shipping but decided to do all testing without ever actually calling the API. Instead it only coded to and tested against a self-hallucinated mock API which is definitely NOT likely to get a correct result.
Me: Have you tested the tfl connection and API responses?
Claude: No, I haven't tested the actual TfL API connection yet - the tests we ran were unit tests with mocked responses. We should definitely verify the real API integration before deploying to your watch! Let me check the TfL API service and test it:
I have seen this in other projects as well and it is a very bad pattern.
Using the real API instead of its hallucination led to a bunch of changes. Again Superpowers was extremely useful in giving Claude a way of thinking about root cause analysis and keeping it on task but not testing against the real API is a mistake that even a rookie programmer wouldn’t make. Scenario testing from 2389 Research looks like a promising way to address that in future.
Testing, Debugging and Improving
The rest of the development process involved me testing the app and then reporting my results to Claude to have it debug and fix the issues along with a bunch of improvements. It seemed to have an easier time with development that looked more like web app development and a harder time with interface issues and connecting to the watch itself – though that also mirrors my skills and deficiencies so that might also have come from me.
Probably the most illustrative bug was in trying to add another button to a display. Claude went on a very long set of wild goose chases for a bug that turned out to be that it had not properly specified that the two buttons shouldn’t be written directly on top of each other. If I had had a better understanding of Kotlin, I might have considered that possibility much earlier or been able to just look at the code. However, without that, Claude took a long time and proposed and implemented multiple wrong paths before it discovered the correct one. There was a point at which I thought it might rebuild the entire app from scratch in order to try to fix the bug.
Conclusion
I’ve now been using the app a lot for about 3 weeks. It is just what I want on the device I own. It doesn’t do anything extra, it doesn’t ask to show me ads or track me, and I have no plans of continuing to improve it. It does one thing well. If you want to use it yourself you can, you just need to bring your own Transport for London key. But I'm fine with it being just for me.
The development was way easier than it would have been otherwise, even if I had known Kotlin and the basics of the Pixel Watch’s WearOS development before starting. Indeed, the Claude Code development would have been even shorter if I had known more. All told, it took about three hours of my time and its cost was included in my $20/month subscription. Even without my ulterior motive of more deeply understanding Claude Code, that would have been a pretty good tradeoff. I expect the codegen tools to get better and to be used by many early adopters to do some quick development of apps and tools that are important to users and that might never have otherwise been satisfied.
These codegen tools are good enough now that non-programmers can and should try to use them. However, in all of my experimentation, I have found that my own understanding of the project and how I would approach it as a programmer was important to working with the tool for better outcomes. These tools amplify my knowledge and my knowledge gaps. That was definitely true here as well. If I didn’t have a passing understanding of the mechanics of network communication for the Pixel Watch, I might have accepted ChatGPT’s incorrect understanding and either not successfully built anything or built a clunkier, worse product. If I didn’t know that having tests pass was a completely reasonable expectation, I might have accepted one of the many times Claude glossed over failing tests. And I bet the combination of Claude and I made a bunch of other errors that I just don't know enough to know we made. None of those should scare non-programmers away from trying these tools, but I see a long future in the need for programming expertise in developing commercial software.
Finally, I highly recommend trying to get a project done with the codegen tools as they are now to better understand what is currently possible in coding and what will likely come to other domains as well. It is hard to describe the ways in which the tools are wonderful and where they fall short (as I try to do here) but it is much easier to experience firsthand with something you care about. Actually diving deep into these tools is the surest way I know to avoid thinking that AI is either useless or about to achieve an intelligence breakthrough. The truth is much more interesting and in-between.
* Throughout this piece I somewhat anthropomorphize and otherwise use imprecise words to describe what is actually happening with LLMs. There's a tradeoff between being accurate and understandable. For this piece I am going with understandable.
No comments:
Post a Comment