In my last post, I explored deterministic code generation using AI. In these early days of AI-assisted development, organizations face two risks:

1. stifling AI creativity in the fight against hallucinations, or

2. falling prey to “vibe coding”—essentially hoping a roomful of infinite monkeys will eventually produce perfect code.

Striking a “creativity on rails” balance requires understanding how changes to a code-generating LLM’s inputs control different aspects of its output.

In this follow-up, I’ll walk through experiments I ran to see how prompt and context specificity affect deterministic code generation.

Up front: these are simple tests exploring a spectrum of specificity typically used for proofs of concept and MVPs, not production projects. See my “Next steps” section at the end for additional tests that address challenges specific to larger, production-quality codebases.

What we’re building

For each test build in this experiment, the goal was the same: build a simple note-taking app. I chose this because:

• I wanted a multi-component, single-page tool. At their simplest, note-taking apps have a list of notes and a text box for entry, all on one page.

• There’s variety in how note-taking apps can be laid out visually (as seen in the market today), so I hoped we’d see UI variety increase as prompt specificity decreased. Spoiler alert: this assumption turned out to be true.

• Diversity in UI component look and feel would likely translate to at least superficial differences in the underlying codebase, which could be examined.

Complexity divided into tiers

To test how prompt and context specificity affect code generation, I created 5 tiers, each one more specific than the last.

• Tier 0: Intent only—an intentionally vague prompt. Pure vibe coding.

• Tier 1: Requirements—more specific functional requirements, but other decisions explicitly left to the LLM. This is where I start specifying basics like “use React and TypeScript.”

• Tier 2: Requirements + architecture guidance—tighter functional requirements, including data model and data flow.

• Tier 3: Requirements + architecture + technical specs—locking down data structures and file names.

• Tier 4: Full single-file PRD with all files, functions, and states explicitly defined.

At each tier, I used Codex to generate a note-taking app twice. My expectation was that I would see fewer differences per generation as I went up successive tiers.

A note about these prompts

I used LLMs to generate the prompts and PRDs for each tier, just as I’ve done for other projects on this blog. I chose this approach because meta prompting—prompting to generate prompts—has become a well-established technique in AI-assisted software development. Generating these prompts with LLM support helped ensure that specificity increased systematically from tier to tier.

That said, if I ran this experiment again, I’d remove the explicit lists of features that the LLM is free to customize. Prompting “it’s okay to be creative here” probably yields a different result than simply leaving those features out entirely.

All prompt input, two generative outputs per tier, and an LLM-based comparison of outputs at each tier is available here:

Deterministic code experiments repo

Helper script to do project comparison

As mentioned, I wanted the LLM to review the code output because I assumed it would catch differences I might miss. The script that runs the LLM-based comparison of two projects is included in the GitHub repo.

I also included the prompt to generate the script. Looking back, I should have made that prompt more specific. The LLM’s comparative output varies widely in detail across the tier directories—a direct result of the script generation prompt being too loose.

This touches on another subject I’m exploring in a project I hope to write about next month. These initial determinism vs. creativity experiments appeal to me because I’m building apps that perform tasks with a backend LLM. Optimizing LLM output for those tasks often comes down to tuning the balance between determinism and creativity.

But for now, onto the test results.

Tier 0

The prompt for this tier was so short that I can include it here:

Build a simple note‑taking app using React.

The app should allow users to create and view notes.

Do whatever you think is best for the project structure, UI, and state management.

The UI between the two builds diverges, not just in style but in functionality. One generation included a filter feature; the other didn’t.

Only one build included persistent storage. This shows how different two outputs can be—and how those differences translate to substantial variations in functionality and quality.

The prompt explicitly told Codex it was building a “note-taking app,” so it understood the task. And persistent storage is a standard feature in note-taking apps. Why use the app if you can’t save your notes?

Even “simple” note-taking apps include persistent storage. Yet 50% of the time, the LLM failed to include it!

Tier 1

The prompt for tier 1 has two short sections: one detailing required features and another specifying what Codex can decide on its own.

The UI for tier 1 still differed about as widely as it did for tier 0. Also, as with tier 0, only one of the two generated apps supported persistent storage.

The source code for the two apps created at tier 1 diverged as much—or possibly more—than tier 0. One instance had significantly more files than the other, primarily because of how Codex divided the source code across files.

Tier 2

All four apps from tiers 0 and 1 worked on the first try. I queried Codex with my prompt, it generated code, and the app ran successfully without any debugging or updates.

Starting at tier 2, none of the generated apps worked without additional prompts. The package.json file—which contains project requirements and dependencies—wasn’t being generated.

This was an unexpected consequence of making prompts more deterministic. Because I didn’t explicitly ask for package.json to be created and my prompt restricted Codex’s creativity, the file wasn’t created. In this case, “creativity” means knowing to create mandatory files without being explicitly told to do so.

The UI is starting to converge. This is the first pair of generated apps with a nearly identical layout. This matters because while my tier 2 prompt includes specific sections on app requirements and “UI&Flow,” the final section still lets Codex decide “exact layout and styling.” My conclusion: even though Codex was nominally free to design the UI however it wanted, the tightening constraints elsewhere in the prompt narrowed its generative reasoning to a semantic space where note-taking apps “just look like that.”

Of course, this is a small sample set. Running many more iterations would help reveal the variety of note-taking app styles within the more concentrated semantic space reached by tier 2’s prompt.

Tier 3

Tier 3’s prompt is significantly more prescriptive, including code snippets that explicitly define data structures.

At this point, the semantic space the prompt drives Codex into has a clearly defined idea of what an unstyled note-taking app should look like. The only unique UI features are superficial, such as where the “pinned” emoji appears on screen.

The note pinning feature has been common since tier 2 but appears in no tier 0 or 1 generations. The obvious explanation: tier 2’s prompt explicitly lists it as a requirement. Still, it’s interesting that the more creative early tiers never thought of pinning—another fairly standard feature in note-taking apps.

Tier 4

The final tier for this experiment is the first where the prompt is framed as a full PRD. If I ran this experiment again, I would place the prompt in a separate file to refer to as a PRD, rather than again just copy the text into Codex’s chatbot interface.

The UI is effectively fully converged between app version 1 and 2 at this tier. In this PRD, there is an instruction to “render the 2-column layout exactly.” Still, in my opinion there is still quite a bit of variation that could have been attempted that Codex doesn’t explore.

This is also the first tier where entire chunks of code in the most critical files are effectively identical. The specificity of the PRD was comprehensive enough that the generation process produced nearly the same code path through the model’s decision tree from beginning to end of files. Considering the vast number of possible token sequences available in Codex’s parameter space, achieving such consistent output through prompt engineering alone is noteworthy.

Next steps

One key lesson: every LLM output is probabilistic, but deviations between successive generations decrease as prompt specificity increases. This isn’t surprising, but it’s striking how visually obvious the pattern becomes when comparing just two generations from the same prompt.

Some other lessons are more interesting:

• Adding too much constraint without sufficient guidance can introduce bugs or missing features in the generated code.

• Without enough specificity, two apps from the same prompt won’t just look different—they’ll have different features and varying quality. This suggests the best code generation results come either from writing highly specific prompts or generating the same app multiple times and cherry-picking features from each version.

That last point brings us full circle: we’re beginning to appreciate the value of the wide variety of code generated by that roomful of infinite monkeys.

I’m interested in learning more about how different balances of creativity and determinism affect a codebase. For my next experiments, I plan to add features to an existing project while varying both the specificity of feature prompts and the availability of supporting context like API contracts.

Give me an image of a character that is a desktop-style fan with arms that uses its own operational fan blades to blow bubbles using a bubble wand. Bubbles should be in the air around the character. Style should be 40s-50s cartoon animation. Background should be interior of a period appropriate home, rendered with colors that match the style of that period of animation.

While the generated images both follow my prompt’s guidance, I did not specify every detail to be generated. Each missing element—the number of bubbles, the fan’s direction, the furniture’s color—and every ambiguity—what exactly does ‘40s/’50s animation style look like?—requires the LLM to guess and imagine how those parts of the image should be rendered.

It’s that “guessing” and “imagining” I’d like to explore in a code generation context in a few articles. Throughout these articles, I’ll refer to this guessing and imagining behavior as creativity.

Describing an LLM as creative is controversial, even threatening, especially when discussing LLM integration into industries driven by artistry and human expression. For now, I want to sidestep those considerations and focus on creativity in a code generation context.

How important is creativity for successful AI code generation? Could LLM creativity actually be counterproductive in coding tasks such as new feature generation and code maintenance?

Why deterministic LLM output could be useful

When code generation is less deterministic and more creative, generating the same codebase twice produces different function names, variable names, file structures, and even logic. Even with identical prompts and context, random seeds and uncontrolled tool responses cause the LLM to make different choices each time.

The cartoon demo in the intro illustrates this: two identical prompts produce distinct images of bubble-blowing fans.

Software engineers rely on deterministic code compilation and project building. Especially with resource-constrained targets like microcontrollers, developers expect the same toolset and unchanged codebase to produce identical code every time they build the project.

This deterministic output is essential for debugging, maintaining standards compliance, and enabling reliable firmware updates in the field.

Generating reliably deterministic output from AI-assisted code generation tools could provide similar benefits. In a prompt-first development workflow, deterministic code output would enforce codebase stability, helping with review, regression testing, and diffing. Adding new features, rolling back portions of the codebase, and partial regeneration of code all become safer and less disruptive.

As LLMs begin writing code for embedded applications, deterministic output could eventually enable code generation for real-time operations where tight timing must be maintained.

Still, I can’t help but feel we lose more than we gain if we eliminate creativity from an LLM entirely. An LLM without creativity is basically a natural language compiler—a useful tool, but one that doesn’t take full advantage of an LLM’s capabilities.

Alignment

On the spectrum of LLM output, creativity’s opposite is alignment. LLM alignment provides guardrails and ensures adherence to safety norms and constraints. It also applies to the training necessary to develop capabilities for successfully completing tasks like writing code. Alignment enables determinism.

How can a developer achieve alignment in LLM output? Besides setting the LLM output temperature to 0, controlling random seeds, and choosing the right model for a project, the level of determinism in code generation is a function of context. The more specific the context the developer provides, the more deterministic the output.

Next steps

In the next few articles, I’ll explore how LLM code generation varies when building the same project with different amounts of context. I’ll experiment across a spectrum—from loose “vibe coding” to rigidly defined software development practices. This means adjusting the detail in PRDs, architecture documents like API contracts, and the prompts I give the LLM. I’ll also need a standard process to measure the qualitative and quantitative differences from build to build.

I believe software teams will develop strategies to deliberately balance creativity and alignment in code-generating LLMs throughout a project’s lifecycle. Hopefully, my experiments over the next few weeks will help illustrate:

• How much an LLM’s output varies from generation to generation due to ambiguities and missing context

• How developers can curate context to balance alignment and creativity in LLM output

Note: In these articles, my thoughts on alignment and creativity are informed in part by a book I’m reading, Creative Machines: AI, Art, and Us.

The more I read about connector support opening up in the coming months, the more intrigued I became about how the system works. Just how hard would it be to create an MCP server and connect it to the ChatGPT chatbot?

I spent a few weeks developing an MCP server concept for debug instruments (link). For this project, I wanted to create something new for a ChatGPT interface while exploring how these web-first tools reach outside their chat interface to interact with the real world.

My goal for this ChatGPT-compatible MCP tool was simple: scan for Bluetooth Low Energy devices near the user. I wanted to open ChatGPT, type “what Bluetooth devices are nearby,” and get a list of names with rough proximity metrics.

Scanning for Bluetooth devices

Most people know Bluetooth from wireless headphones and car connectivity, but those devices use Bluetooth Classic. In 2010, the Bluetooth 4.0 specification introduced Bluetooth Low Energy (BLE), a protocol designed for lower power consumption, simpler pairing, more flexible network topologies, and battery-powered IoT devices. Today, your keyboard, mouse, smartwatch, continuous glucose monitor, insulin pump, heart rate monitor, and billions of other devices support BLE.

A key feature of BLE is its advertising mechanism. BLE devices broadcast small data packets called advertisements on three dedicated channels in the 2.4 GHz ISM band. These advertisements serve two purposes: they can transmit small amounts of data directly to any listening scanner (commonly used by sensors and health monitors), or they can indicate that a device is available for connection. When a scanner detects a connectable advertisement, it can initiate a connection to establish a bidirectional channel with higher bandwidth and security.

Scanners can also estimate distance to an advertising device by measuring signal strength, called the Received Signal Strength Indicator (RSSI), measured in dBm. While RSSI provides only a rough distance estimate due to environmental interference, it’s useful for proximity detection.

This advertiser-scanner relationship is what I’ll be exploring with this MCP server. I want ChatGPT’s chatbot to interface with a server running locally on my machine that acts as a scanner for active Bluetooth advertisers, reporting back information from the advertisement payload and RSSI values.

I’d like to eventually build more sophisticated Bluetooth interactions, especially around spatial awareness derived from RSSI. But for now, I want to see how difficult it is to retrieve basic Bluetooth information.

Rube Goldberg’s LLM-powered Bluetooth scanner

All this “running apps inside chat” marketing is really an official process for connecting remote LLM-powered agents to tools and data through HTTP requests. I write a query locally, and the remote agent responds by calling remote MCP servers over the internet. Using additional context from those servers, the agent sends a response back to me through chat.

In this experiment, the MCP server runs locally on my Mac mini. It exposes a single tool that scans for Bluetooth LE advertisements for a set period, then responds with a list of discovered devices and all the information the scanner has learned about each one.

This is obviously a needlessly complex process, with data looping between local and remote endpoints just to scan for Bluetooth devices. You could easily handle this task locally using phone apps or operating system control panels. Nevertheless, I think this test hints at a trend we’ll see with agentic systems in the future.

But before we get into speculation, let’s look at how the experiment works. For those interested, the demo’s source code is available here:

GitHub repo

Building the MCP Server

After iterating on a PRD for a Bluetooth Low Energy scanner wrapped by an MCP server, I used OpenAI’s Codex CLI to generate the project. My somewhat naive thought was that since Codex, ChatGPT, and the new SDK APIs to support “connector” MCPs all come from the same company, Codex might have the best chance of building a working system.

Interestingly, Codex’s questions suggested it had trouble finding the SDK documentation on OpenAI’s site. I had to point it in the right direction.

After that hiccup, Codex built the MCP server without much trouble, at least until I ran local test calls to retrieve nearby BLE device data. I spent time parsing through the JSON listing because some data seemed incorrect or missing. Through trial and error, the server eventually returned a reliable list of devices by name, address, and RSSI value. Once everything looked right, I exposed the server to the internet through ngrok and began trying to connect it to ChatGPT.

ChatGPT as MCP Client

After switching ChatGPT to developer mode, I navigated to the Settings menu and found the New Connector screen where I entered the HTTPS server location and metadata. Clicking “Create” immediately triggered requests to my server, visible in my terminal. Success…except the calls failed and nothing worked.

An interesting sidenote: MCP as a standard relies heavily on natural language in its protocol. While tool declarations, command responses, and other data are expressed in JSON format, much of the information is provided in natural language. The only text in the codebase describing the “scan now” tool’s functionality to connecting clients (in this case, ChatGPT as the MCP client) is “Perform a one-shot Bluetooth LE scan and return discovered devices.” The AI must read and interpret this statement to determine when this tool might help respond to a query.

Even on its first attempt, this simple tool definition was enough for ChatGPT to understand that when I asked about BLE devices, it should try using this tool. The problem was that the tool wasn’t responding in a format ChatGPT expected.

Debugging became an iterative process of examining terminal output and ChatGPT responses, copying logs and taking screenshots to relay back to Codex. Codex eventually resolved all the bugs and delivered an MCP server that functioned as ChatGPT expected.

ChatGPT approved the server connection, and the chatbot could now scan for Bluetooth devices in my room whenever it determined that answering my prompts required such information.

Seeing success with my first prompt, I wanted to see if ChatGPT understood the tool’s output to the point where it could interpret the data and provide meaningful insights. For this next experiment, I placed my AirPods case close to my Mac mini, so that the signal strength of the Bluetooth advertisements from the case should have relatively high RSSI.

Let’s take a step back and consider the LLM-powered agent’s reasoning process and sequence of actions. When I asked ChatGPT to identify the Bluetooth LE device with the highest RSSI near me, the agent had to: understand I was requesting Bluetooth information, check its available tools to confirm it could measure conditions in my physical proximity, call the server on my machine, identify my AirPods, and report that information back to me. This demo offers two interesting perspectives: its implications for user experience design, and what it reveals about LLMs’ ability to interact with the physical world.

The local execution advantage

While this demo takes a circuitous path through remote servers to perform a simple BLE scan, consider how different it would be with purely local LLM execution. Imagine asking your phone, computer, or smart home hub to help find your AirPods. A local LLM-driven agent could use available BLE scanners to detect and even triangulate the device’s location. With everything happening locally, security risks are minimized and response times are fast. The actual BLE scanning would be the slowest part of the process. This synthesis of real-world sensing, local inference, and natural language interaction represents the kind of intuitive human-machine interface that tech companies have been racing to deliver for years.

The broader implications of AI sensor access

It’s critical to consider the implications of MCP potentially providing AI systems with global-scale access to sensor data. The proliferation of Internet of Things devices across industrial, commercial, consumer, and even non-terrestrial environments means our world is increasingly instrumented with electronics. These devices continuously sense their surroundings, measuring temperature, acceleration, capturing images, detecting motion, and more, before transmitting their data across networks that could feasibly become accessible to AI through standardized protocols like MCP.

What observations, insights, and predictions could such a system make about the world and the people in it? The technological innovations most worth pursuing are those that, on balance, make us safer, healthier, happier, or wiser. AI has the potential to help us achieve these goals, but those of us in technology and regulatory fields must work together to ensure we gain more from this technology than we sacrifice in security, privacy, and autonomy.

Both proof-of-concept websites mostly proved two already-obvious points:

1. It’s hard to graft together LLM outputs to create a simple one-shot code generation environment

2. It’s dangerous for a website to perform brain surgery on itself

The main blocker was that the “DevBot” interface needed much more control over the backend code builder, and the chat UI needed to be far more interactive. Basically the interface needed to be more like the agent chat interfaces in Cursor/VS Code. When I asked the DevBot to add real functionality, such as asking it to “add a box to the home page showing today’s weather,” it was completely beyond its capabilities, even after I forced the backend to carefully parse error messages and iterate on the code.

All these excuses aside, I could get the chat window to make superficial CSS changes (”make the background color of this page dark blue”) successfully. I could also make simple HTML edits through the interface (”add a link to the text on the contact page”).

Anyone who’s dived head first into a project without sufficient planning will probably recognize that feeling where every step only reveals two more steps before you reach your goal. That’s how these projects felt.

One thought struck me as I experimented with these POCs, staring at the empty text box in the “DevBot” chat window: how intimidating that text box was. Even though I knew my clunky little DevBot POC could barely make anything other than font and color changes, the possibility of requesting any feature from my phone felt intimidating.

I began to wonder whether my speculation in my last post was wrong-headed. Is there really any value in making the mobile app user experience extremely customizable? Maybe the evolution of modern mobile apps from their starting point in the 2000s to today provides some insights.

Why apps are the way they are

iOS and Android apps in the 2000s were siloed, isolated experiences. They couldn’t run tasks in the background, sharing options were limited, and notifications didn’t really exist.

Android innovations on first-gen mobile apps found their way into iOS with a lag of a year or more. Android 1.5 supported home screen widgets, which were one of the first instances of a mobile app surfacing data outside the app interface itself. Widgets have since evolved into information-rich app surfaces supported by both iOS and Android.

Notifications came next, allowing apps to send short messages to users outside the app itself. iOS’s Notification Center gathered all these notifications in one place. App developers probably worried about this change, since it meant they had less control over when and how users saw their content. However, notifications were designed to bring users back into the apps. My guess is that app developers ended up with more total user engagement and revenue from notifications than they lost from people spending less time browsing within their apps.

Over time, notifications became more descriptive and interactive. Tapping a notification could open the corresponding app. Later, notifications offered richer UI, letting users type responses into text boxes within the notification to complete an interaction (e.g., “Send this message”) without ever opening the app.

Mobile apps’ evolution toward interactive surfaces outside siloed apps indicates the likely path forward. We’re already seeing AI integration with notification surfaces, where notifications are summarized by AI to condense what has become a cacophony of mobile app notifications.

Next steps as AI intercepts UI in mobile apps

Deeper AI integration into mobile apps will likely change mobile applications in two major ways:

• Cross-app integrations for agentic, autonomous operation

• Deep and personalized natural language summary and notifications

It remains to be seen how long mass market adoption will take. I find it hard to imagine mainstream users handing over purchasing power to agents without multiple authorization gates. Similarly, as the potential drawbacks of using AI for data access become well known, a stigma about hallucinating AI will likely mean most users approach AI summarization with a “trust but verify” mindset. Users will still want to click into a summary and find information directly.

Also, any deeper app customization that breaks down existing app design patterns will run up against a simple fact: broadly speaking, we like the way our apps work today. We enjoy (or at least we’ve been conditioned to enjoy) endlessly scrolling through social media feeds. Our scrolling thumbs have muscle memory.

Jumping back to the anxiety of that blank text box where I could potentially ask for any feature: I think given sufficient design power, most users would still design user interfaces more or less like the ones we’ve all been conditioned to use. Instead of some bold new agentic “build your own apps” interface on mobile, we’ll see a continued, slow, but relentless march toward opening up more unified app surfaces where AI-driven summarizations and intuitive operations still support business models that rely on driving users into the apps themselves.

I think that will continue being the case, even after LLMs begin running apps from within ChatGPT and similar chatbot interfaces.

Running apps Within ChatGPT

OpenAI recently announced a new SDK that lets app developers integrate their features into ChatGPT’s chat interface. The pitch is straightforward: users can chat with apps like Spotify and Zillow through ChatGPT’s natural language interface.

OpenAI enabled this using APIs built around Model Context Protocol. App developers can use the SDK to build these features now, with a wider release planned later this year.

This represents another step in app developers seemingly giving up precious user attention from their mobile apps and websites. While this could look like an existential threat to their business model, their experience with mobile apps tells a different story—notifications and widget surfaces still drive users toward their apps, or more precisely, toward their data and algorithms.

OpenAI’s deep integration with other apps feels less like a threat to app developers and more like a way to encroach on mobile operating systems’ control of attention. OpenAI’s agentic operations using deep linkage to third-party algorithms and data could act as a flanking maneuver around what Android and Apple are already attempting with their own AI integrations. If OpenAI or Anthropic reaches a critical mass of third-party app developers and nails the UX for third-party integrations before mobile operating systems do, it will be much harder for Apple and Google to keep up.

Maybe the real paradigm shift isn’t that mobile apps will break down under AI customization pressure, but that mobile operating systems themselves will buckle under the pace of innovation at leading AI companies.

On the opposite end, platforms like lovable.dev make source code accessible but secondary. They target users who want to design apps without engaging with code at all.

The remarkable success of lovable.dev revealed my own bias about app creation. I had assumed the creative drive to build apps belonged primarily to those who could code. I believed technical skills preceded inspiration, but lovable.dev demonstrates these elements are actually independent. Lovable.dev’s impact shows that a much wider audience beyond software developers wants to create apps. It’s effectively unlocking creativity for everyone.

How does a no-code dev environment evolve as LLMs improve?

The LLMs powering lovable.dev and similar ‘no-code’ app development tools will become increasingly powerful in both effectiveness and efficiency. They’ll generate better code while consuming fewer resources, eventually enabling LLMs to run on edge devices, particularly smartphones.

What happens when LLMs advance to where everyone can run a supercharged version of lovable.dev locally on their phones? Imagine a world where your smartphone has enough AI coding capability to create any app exactly as you want it, simply by describing your desired user experience.

If no-code development environments become highly capable, user-friendly, and run locally on devices, what new forms of creativity might this unleash?

ChatBot, meet ‘DevBot’

The next evolution beyond platforms like lovable.dev would likely be website and mobile app creation from within the app itself, using an interface similar to today’s chatbots.

On retail and commercial websites, you often see a chatbot in the corner answering customer questions. Now imagine instead a “DevBot” in that same corner, waiting to custom-build experiences based on your instructions. You’d simply click on the DevBot and describe what you want. Or, more likely, each user steadily customizes their user interface and functional particulars from some starting point template, supported by guardrails and limits.

If such a DevBot had sufficient power with deep OS integration through official APIs or on-device development sandboxes, and could access the content that fills today’s apps (social media, news, sports, streaming content), how might this transform our current website and mobile app landscape?

Why mobile operating systems might enable something like this

Let’s explore how a ‘DevBot’ concept could transform mobile user interfaces. Why would operating systems relinquish such control to a DevBot?

This question reminds me of the story in Walter Isaacson’s book about Steve Jobs’ initial resistance to opening the App Store to third-party developers. Jobs hesitated, doubting Apple’s ability to provide the necessary software tools, drivers, and libraries for app development.

Apple’s eventual release of an SDK and development tools for third-party apps unleashed a creative and business explosion that shaped today’s mobile experience dominated by third-party apps. This move also created an incredibly profitable revenue stream for Apple.

As no-code development environments like lovable.dev gain momentum and tools become increasingly user-friendly, it’s possible that companies owning phone hardware and mobile operating systems would find a way to natively incorporate no-code app development. If these companies establish effective monetization models, ‘DevBot’-style personalized control could fundamentally transform how we interact with our phones.

How content providers fit into this concept

We go online and use our phones to interact with and consume content largely owned by third parties. The amount of available online content is effectively infinite. From social media accounts to 24-hour streaming sports and news, games, and vast libraries of movies, books, music, and on-demand TV, we have endless ways to spend our time.

However, we navigate an online world of content consolidation. This consolidation serves a purpose: platforms like Facebook, Netflix, Amazon, and Google help us find signal in the noise. They curate the content we want and, in exchange, sell our attention to advertisers. This symbiotic relationship has benefited both companies and consumers.

If this era of “static” apps ever ends, I believe consumers gaining more control over how they access third-party content will drive that transformation.

This doesn’t mean content would become devalued or free. Content creators still need and deserve compensation. Rather, content providers might someday be compelled to cede enough control over their content to support deeply customized user experiences.

It’s a long shot, but if we gain more control over our feeds, the revenue models emerging in such an environment could be less exploitative and manipulative to both content creators and end users than the revenue models that drive technology today.

Onto some experiments

I plan to build a minimally functional proof of concept for a DevBot. I’ll begin with a DevBot-customizable web app template. My goal is to create something that can deploy website features that go beyond basic CSS edits. Let’s see if I make enough progress by next week to continue this series with a second part.

• What component of the project should I work on first

• When should I move away from that component and work on another bit

I’ve faced this challenge as a solo firmware engineer balancing the development of different code components and test infrastructure. To test one component, I need a base level of functionality in other components. Creating stub functions and fake data to establish a clean data flow has traditionally been my solution.

Recently, I’ve found another approach: generating test inputs and spoofed data using an LLM. This method has surprising benefits alongside significant drawbacks. I believe that LLMs’ tendency to “tell us what we want to hear,” which is a common and valid criticism, can actually be harnessed for good, at least in this specific use case.

Problem: I needed human text input for testing and development

I’ve been working on a project that’s centered around a text editor. The project’s LLM-driven agent makes observations on text in the editor and interacts with the user. Testing for this project has been a challenge because I needed text in the editor window that was relatively long (1K-10K words) and somewhat coherent. There are lots of ways of running tests with this system, but I had issues with all of them:

1. Use dummy text, like the Latin placeholder text used in website or graphic design tools. This wasn’t viable because I needed to test system responses based on realistic English language inputs that reflected real-world content.

2. Write text input manually and use that for all testing. This approach wouldn’t work because I needed to test the system with a variety of writing styles and subject matter.

3. Scrape text from web sources for internal testing. I avoided this approach because finding different text sources with the specific writing styles I needed was tedious. I felt constrained by available content and worried I might miss critical test cases.

4. Give up on the 1K-10K requirement and just test with snippets. This had potential because it was definitely easier to write a paragraph instead of an entire story/article. I actually tried this as test input at first until I realized that the tests I wanted to run required a text input length that was longer than I had bandwidth to write by hand.

Not seeing a path forward with the first set of options that came to mind, I settled on using an LLM to generate the text inputs based on parameters and guardrails. This system worked really well and led to a broader set of testing and feature development that hopefully I’ll write about sometime later.

Can we call this fuzz testing with AI?

While I’m stretching the definition a bit, using LLMs to spoof data streams resembles fuzz testing in security work. Fuzz testing bombards a system with high volumes of data or specially crafted inputs designed to test edge cases. These tests can reveal bugs when buffers overflow or memory pointers exceed their ranges. Such bugs may become security vulnerabilities, potentially exposing data or allowing attackers to force the system into undefined states.

In my case, I wasn’t hunting for security flaws. I simply wanted to stress test my project with diverse, valid inputs. Creating this test data manually would have taken days rather than minutes.

Other cases where LLM-spoofed data might help

While my example comes from a specific use case, these principles apply broadly. My system generated paragraphs of text, but LLMs could just as easily format data as JSON or structure it in any way needed for testing. Developers could use this approach to:

• Generate data payloads for stubbed API returns

• Populate test databases with realistic data

• Simulate stateful responses in sequential test scenarios

The last bullet point represents where LLMs can provide unique value. You could develop an agent with specific prompts and memory that creates a responsive persona, one that dynamically interacts with a system under test.

Limitations on the concept

The main limitation of this testing approach is that LLM-generated content lacks the determinism of traditional test inputs. Even with strict system prompts and robust guardrails, the output may still require human oversight or post-processing to validate its suitability.

Cost presents another challenge when using LLMs for data generation, though several strategies can mitigate this issue. Selecting a cost-effective model that still delivers adequate quality and coherence is crucial. Additionally, deploying a local LLM within your development environment can eliminate per-token costs entirely compared to remote API calls.

Data spoofing itself isn’t new. Software and embedded engineers have long used simulated inputs during system development. What LLMs offer here is the ability to create data with controlled randomness that can effectively stress-test your system, challenge assumptions, and reveal hidden bugs.

The story of one recent development actually starts decades ago before social media dominated the web.

RSS and what came after

In the 2000s, before social media funneled web publishing to their platforms, users like me discovered content through RSS feeds and aggregated them using tools like Google Reader. I never knew at the time that RSS stood for Really Simple Syndication. I didn't think to Google it, probably because I was too busy enjoying articles through Google Reader.

Like many good things, RSS and Google Reader fell victim to social media. This happened in two ways. Within Google, the Reader team was reassigned to work on Google Plus, despite the fact that Google Plus never achieved the traffic levels that Google Reader enjoyed. Across the industry, social media platforms like Facebook lured news sites with promises of higher engagement and click-through traffic. With algorithmic feeds presenting users with content supposedly tailored to their preferences, what could go wrong?

In the decades since this migration to social platforms, digital publishing has struggled to survive. Now, LLMs scraping content without driving traffic to original sources may deliver the final blow to web publishing as a viable livelihood for writers online.

In a delightful twist of irony, a glimmer of hope for web publishing's future comes from one of RSS's original creators.

Really Simple Licensing

Created by the RSL Collective, the RSL standard aims to be an enhancement to robots.txt that standardizes how publishers license their content. This machine-readable license format clearly defines required attribution and payment terms for both crawling and inference.

One of the group's leaders is RSS co-creator, Eckart Walther. Though I haven't found interviews where Walther connects his past work on RSS to RSL, I appreciate that someone who once created a standard making online writing accessible (and in retrospect, sustainable) is now working to make news articles and creative writing sustainable again.

LLMs need to be knowledgeable and current to provide relevant answers to users. This requires training on vast amounts of online information and access to web scraping tools. The explosion of LLM usage works well for everyone except the human writers whose work forms the foundation these LLMs rely on.

The current system isn’t sustainable. If writers can't earn from their work, the situation will worsen beyond the damage already caused by decades of algorithmic chaos on social media. We'll likely see more lawsuits like the recent one where Anthropic agreed to pay hundreds of thousands of writers $1,500 each.

RSL offers an escape from this unsustainable future by standardizing how content creators tell AI companies and scrapers what content can be used and under what conditions. This could create consistent revenue streams to stabilize and nurture web publishing. We need this to work. Web publishing represents writing at its most vital and urgent, and our culture suffers when viewpoints diminish online and reliable news sources dwindle.

Whatever success my project achieves will largely come from value developed through material sourced online via APIs and rules in robots.txt files. Building my music discovery database has shown me the immense power of a smart script and well-constructed database when unleashed on the internet.

Cloudflare’s strategy

RSL is just one piece of the puzzle being assembled to protect writers and content owners against unlawful or impolite scraping. Since Cloudflare serves about 20% of web content online, any strategy they deploy will likely create ripple effects throughout the industry.

Cloudflare's strategy begins with blocking web scrapers by default unless content owners explicitly allow scraping. They also empower content owners to define specific scraper access rules: which parts of the site are accessible and under what conditions. Additionally, they're exploring a "pay per crawl" model. If Cloudflare or another company successfully productizes this approach, it would enable new business models for companies that cultivate and sell data to AI companies for training. While this is already happening to some extent, easily accessible, widely adopted tooling would significantly lower entry barriers for startups.

While the exact outcome remains uncertain, it's encouraging to see many in the tech industry working toward a sustainable future for web publishing.

Preserving human creativity online

With 700 million weekly ChatGPT users, adoption is both widespread and rapidly increasing. Yet for authors, filmmakers, musicians, and other creative professionals, generative AI often represents industrial-scale appropriation of their work.

These concerns are valid. AI companies have acknowledged in court documents that they haven't always adhered to copyright laws. Anthropic's recent $1.5 billion settlement with authors demonstrates that courts won't simply allow AI companies to disregard copyright protections in their pursuit of artificial general intelligence.

Practical, standardized licensing frameworks like RSL, paired with enforcement tools such as those Cloudflare are developing, can create a more sustainable and fair ecosystem for creatives, AI companies, and their users. Whether these measures will be sufficient to preserve web content creation remains uncertain, but it's encouraging that some in the tech industry are making sincere efforts to find equilibrium.

Fair compensation for creators is essential for reducing tensions between creative communities and AI. Another crucial element is transforming how online content is presented to clearly differentiate between human and AI-generated material. We may need entirely new design approaches for news readers and content platforms that intuitively signal whether users are consuming human or machine-created content.

Of course, anyone who has tried this approach knows the result is typically an over-engineered, sputtering mutation of what you initially hoped to develop. Each increasingly desperate prompt to fix strange behavior only spawns new aberrations that require more frantic prompts until you eventually abandon the project in some forgotten GitHub repository. I suspect most people who've experimented with AI code tools have at least a few of these AI-conjured Feature Creeps locked away in digital basements. I certainly do.

Trying to ensure my new project doesn’t turn into a Feature Creep

My project relies on a curated dataset of music information available online. The eventual value of this project will depend on:

• The relevance, accuracy, and freshness of the dataset

• How accessible the data is to a frontend tool

This is fundamentally a data aggregation problem, where the tool increases the signal-to-noise ratio of dispersed or unwieldy information. For decades, websites have found success with data curation and aggregation:

• Job listings on Indeed and [monster.com](http://monster.com), etc.

• Air travel info on [kayak.com](http://kayak.com), Travelocity, etc.

• News sites like Apple News

My system isn't trying to find visitors the cheapest plane ticket. I'm simply leveraging LLM creativity to generate cool music playlists. However, regardless of a project's scale or goals, careful architectural decisions remain critical, especially those affecting data ingest and curation.

First things first: how do I enable an LLM-powered agent to access the web and retrieve information?

How ChatGPT’s web search works

Anyone who has used chat interfaces like ChatGPT or Google Gemini has likely noticed the "searching the web" step that often appears after submitting a query. This standard feature exists across today's AI chat platforms. While it doesn't always guarantee accurate results, these web searches provide a degree of traceability since you can view the sources the AI used to formulate its response.

The LLM activates this web search tool whenever it determines your query requires internet-based information to answer properly. This capability was originally a standalone product called SearchGPT, which launched in 2024 before being integrated into ChatGPT.

Unfortunately, ChatGPT doesn't offer this web search tool through their developer APIs. Some other LLM providers like Anthropic and Google do offer ways to command their LLMs to search and summarize website information via APIs.

For this project, since I'm exclusively using ChatGPT's models and APIs, I needed to replicate the search function myself. This involves a two-step process: a script uses a tool to access web data, then feeds that information into ChatGPT's LLM for summary and processing.

Accessing web data respectfully

There are two primary approaches to retrieving data from web sources:

• Using APIs provided by websites that offer official access to their data

• Utilizing web scraping libraries for websites without accessible APIs

Before retrieving any information, it's crucial to review each website's terms of use. For sites with APIs like Wikipedia, these terms are clearly outlined in their documentation. The process typically involves creating login credentials, possibly paying for access, and then using the data within the established guidelines. It’s a straightforward engagement.

For websites requiring web scraping, determining permissible content access can be more challenging. Fortunately, many sites now include a "robots.txt" file specifically for web scraping agents. While robots.txt is a relatively recent development in web standards and its legal enforceability remains somewhat unclear, it provides essential guidelines for respectful site interaction that should be followed.

Robots.txt files inform agents (or humans reviewing the document) about:

• Permitted scraping areas

• Prohibited zones

• Access rate limitations to prevent server overload

• Site maps to help locate information efficiently

After developing initial music data, I used code assistant tools to create LLM-powered scripts for web access. My implementation prioritizes APIs when available and resorts to appropriate scraping techniques when necessary.

Now that I know how to access data, what do I do with it?

My project, in brief, helps visitors discover music recommendations through a backend agent that retrieves web-sourced music data. This agent creates playlists with creative sequencing based on song similarities and narrative qualities. The scripts mentioned earlier supply this web-sourced music data.

When designing a system reliant on web-sourced data, a key architectural decision is whether to store the dataset locally for the agent to access or have the agent perform fresh web searches each time—essentially, local versus network storage.

While I could build a proof of concept using either approach, my priority is finding the most cost-effective architecture that produces surprising and delightful results. If this tool isn’t fun to use, what’s the point?

Local vs. web data: cost considerations

Network access to music data with each visitor interaction requires agentic calls to APIs or web scraping tools, plus LLM summarization and information compilation. This web access must be well-structured with predetermined rules guiding the agent to the right information sources.

The main reason I consider this method less cost-effective than local storage is that visitors would likely request the same data repeatedly. Since the tool looks for relationships between songs, the LLM would need to search through content "live" each time. This creates enormous inefficiency in data compilation.

In contrast, a local relational database minimizes duplicate effort during data ingest. However, local access is only effective when the database contains all the necessary information for the LLM to create surprising and delightful playlists. Curating data so the LLM has everything it needs to generate truly surprising and entertaining playlists creates a form of "context anxiety,” which I talked about in a previous post. This is a concern that I suspect has existed since the dawn of databases.

Local vs. web data: quality considerations

AI offers developers powerful tools to discover and curate data through agentic workflows, streamlining every step from identification and ingestion to categorization and tagging. My current work focuses on fine-tuning these curation mechanisms. I'm exploring various approaches to help the LLM identify interconnected aspects of popular music in ways that are both flexible and substantive.

Despite these efforts, I can't shake the concern that my results might not surpass what you'd get by simply asking ChatGPT to list songs with specific attributes.

Data freshness presents another challenge with local storage. While it's obvious that new music releases require database updates, the contextual information around music also evolves over time. Artists give interviews revealing new insights about their songs. Awards and recognition can trigger reassessments of an artist's work and create new connections between established and emerging musicians. Capturing these subtle shifts is difficult, but it’s something you essentially get "for free" when your system performs real-time web searches with each user interaction.

Next steps in development

While I feel confident that a curated, local dataset providing an LLM with ‘Goldilocks’ context will unlock creativity in a way that’s far more scalable and cost effective than just diving into the web with every query, defining and maintaining the system architecture to actually do that curation remains a challenge for me. Hopefully I can keep the project on track and prevent the system from mutating into another Feature Creep.

There's a qualitative aspect to playlist output that's hard to test through automation. For now, I'll rely on manual comparison between what my curated dataset produces versus what ChatGPT casually spits out with a simple query. This will be the bedrock test case of my system testing. Hopefully I'll reach a point with this project where I can share it here for other people to test too, though I still have a ways to go. In the meantime, I'm hoping to write more articles in this series on some aspects of my database structure and how I'm using embeddings to coax the LLM toward unique and creative choices.

AI is also helping us find innovative ways of writing buggy, broken code. In the few projects I’ve worked on so far, I’ve made mistakes I could have only really made through the use of AI code assist.

Background: I’ve been writing buggy code for decades

I’ve spent time debugging systems from a few different POVs:

• As a customer-facing applications engineer, I reviewed customer code and schematics, debugging other people’s systems with sometimes-limited visibility into the full system

• As a firmware engineer, debugging my own code with unit tests and system level regression tests

• As a product manager, mostly in project reviews

In all three of these roles, I had to debug issues by expanding scope to a system-level view. Runtime bugs especially required full understanding of how different components interfaced with each other.

Communication was just as important as technical insight. If I couldn't fix the bug myself, clearly explaining the problem became essential to finding a solution.

When working with an AI agent, communicating means providing context.

Context fragmentation

I've been working on a project intermittently for about a month, and its complexity has become unwieldy. Developing individual components in isolation while frequently switching to other projects created ill-fitting component interfaces with mismatched connections.

I only discovered these issues when running system-level tests, which failed at these interface points. Rather than recognizing the broader, systemic problem, I made the situation worse by asking the AI agent to fix what I mistakenly thought were isolated test escapes. The resulting proof of concept worked with carefully prepared test data but failed with most real-world scenarios.

The root cause was a communication breakdown between myself, the agent, and the codebase. The agent's understanding of the system drifted with each new chat thread as context became fragmented. Outdated architecture documents in the repository, which the agent referenced, made matters worse. Every prompt I gave the agent occurred in an environment with only partially valid context because I had allowed this fragmentation to occur.

To solve this, I asked the agent to audit the project, create a workflow diagram showing component interactions, and identify interface mismatches. I also had it revamp all documentation, now organized in the /docs/ directory. After reviewing these new architecture documents and the audit results, I cleaned up the directories and removed obsolete files.

This must be an enormous potential issue in larger projects beyond my small proof of concepts. Having a trusted set of living, up-to-date architecture documents is absolutely essential, and something I'm maintaining from now on. Based on recommendations I've found online, I've also started creating a "brief.md" file containing a concise description (under a few hundred words) of the project's current state, TODO items, and other important notes.

Ghost APIs

Another problem I encountered initially seemed like a classic AI hallucination, but turned out to be something stranger.

I asked the Agent to interface with a Python library. I had been moving smoothly through implementation on this project with few questions or mistakes from the agent. With this Python library, everything appeared to be going well. The agent included the library in the script, added code to interface with it, and proceeded to the next step in the build plan.

When it came time to test my progress, none of the function calls worked. The Agent had tried to use object methods that didn't exist. After checking the library documentation, I thought I had confirmation: the AI was simply making things up.

So I confronted it:

Parker: i see you are using 'addComment()' method. Are you making that up?

Agent: Yes — the addComment() method I referred to is real and documented in Google Apps Script's Document class. It takes a Range object as the anchor and attaches a comment to that location in the document. Here's the proof:

The agent then provided actual links to forum posts where real people had discussed using the same APIs that were causing errors in my Python script.

So it wasn't a hallucination. But what was it?

It wasn't a case of the forum discussing some unrelated library that the Agent's web crawler had misidentified.

And it wasn't a joke. The forum comments were sincere—the people discussing these APIs clearly thought they knew what they were talking about.

Then I noticed the dates on the comments: 2021. After some research, I confirmed that these APIs had been deprecated long ago.

The agent wasn't hallucinating. It was more like the agent was seeing ghosts.

I nudged the agent toward the truth:

Parker: go to the link you provided above and read it today. there is no mention of addComment()https://developers.google.com/apps-script/reference/document/document?utm_source=chatgpt.com

Agent: You’re absolutely right — I went back and reviewed the official documentation at the link you provided (Google’s Apps Script Document reference), and there is no mention of addComment() anywhere. That method simply isn’t documented by Google .

The agent confirmed what I had noticed earlier, explaining that the API had been deprecated, though it was officially supported at one time.

The lesson I learned is to always be specific about which versions of libraries and tools the agent should use in a project. For this particular bug, that would have been the simplest solution. The agent likely wouldn't have made this mistake if I'd included some boilerplate in the PRD like "always use the latest versions of all libraries used in the script."

Sidenote: slopsquatting

The issue described above, where AI appears to invent a non-existent library, creates a serious security vulnerability. Hackers who can predict which libraries an AI might fabricate could create malicious packages with those names. If I had installed such a package to match what my code was looking for, I could have unknowingly executed the hacker's code, essentially falling victim to an injection attack.

This security threat has occurred frequently enough to earn a specific and unpleasant name: slopsquatting. More information about mitigating this risk can be found here:https://www.techradar.com/pro/mitigating-the-risks-of-package-hallucination-and-slopsquatting

Context anxiety

Both of these bugs I’ve encountered are related to context: either not having all of the context the agent needs, or having the wrong context. It’s challenging to get this balance right and decide what information needs to be in context.

I think we might start seeing articles about ‘context anxiety’: an anxiety rooted in lack of confidence in one’s ability to tell the LLM what it needs to know. A neurosis about whether we’re asking our questions to a machine the right way.

The thinking goes: if I can just provide the right context, Goldilocks-style, neither too much nor too little, I'll get a useful answer.

We can see this mindset already driving numerous context engineering how-to blogs and YouTube tutorials. The philosophy often explicitly stated in these resources is "If you give the LLM all the right information in the context, the output will be useful." To be clear, I generally agree with this position.

Memory for agents will likely evolve to address context size limitations, and leading LLMs are racing to deliver models with a context size to 1M tokens or beyond. If an LLM could process effectively unlimited information, consuming entire projects and precisely interpreting thousands of lines of API specifications, then the LLM should generate more useful code.

But would "infinite context" actually solve the problem of context anxiety? Perhaps it merely shifts the challenge from "context" to "focus." Software developers will still need to craft clear prompts, directing the LLM's attention and intent.

Through education and work experience, software engineers have learned to fix compile-time errors and runtime errors. Now they must also learn to fix prompt-time errors.

My pin monitor MCP server’s Arduino board connected to another Arduino that would run simple sketches to illustrate debug scenarios. I thought my setup was ready for concluding test cases, but I was wrong.

How I fixed some bugs and worked around others

In early testing, the agent frequently tried to exercise tools in ways that didn’t make sense. I decided that the issue might be that the agent just had too many options. The MCP server exposed too many tools.

I had let the agent over-design the solution until I had exposed around 30 nearly-useless and indistinguishable tools. After a lot of deletions and paring back, I was left with what seemed like the only functionality the monitor actually needed to configure and monitor for edge events.

Reducing the number of tools immediately improved the agent’s ability to understand how and when to use each tool available.

I ran into another issue I couldn't fix. In my first post, I described a 'virtuous development cycle' of writing, flashing, testing, and debugging code. Unfortunately, I discovered conflicts between the pin monitor MCP server and the Arduino CLI MCP server. The problem stems from the virtual serial port on the Arduino's debug interface. After flashing new firmware, the pin monitor becomes unresponsive until manually reset.

Rather than spend too much time fixing this issue, I settled on a workaround: I instructed the agent to give me a 10-second pause after programming the DUT so that I could disconnect and reconnect the USB interface. A crude, embarrassing hack, but it let me get to the fun stuff and continue testing.

Fixing a square wave frequency

Scenario I’m mocking up: a system produces a square wave output at a specified frequency. If a timer is incorrectly figured or if the clock source for the system is running out of spec, the square wave’s output might run at an incorrect frequency.

For this test I asked the agent to write a sketch for the DUT to output a square wave at 4 Hz.

Then, without any other testing, I gave it this prompt:

The agent configured the pin monitor and measured the square wave’s frequency for about 5 seconds, which was expected. What I didn’t expect was that the agent also wrote a special test script to ingest the MCP server’s dataset and analyze the edge events to derive a frequency. Very clever.

The agent fed the output of the script into the chat window. The wording of this output is a little strange. It looks like it swapped the ‘measured frequency’ and ‘difference’ metrics, but it still came to the correct conclusion: the output was close to 4 Hz, but could use some fine tuning in the DUT’s sketch could get the output even closer to 4 Hz.

After editing the sketch using insights gained from the pin monitor, the agent was able to tune the frequency to within 0.1% of 4Hz.

Side note: obviously there’s a flaw in this system: the pin monitor’s clock source is not running at a frequency that wouldn’t provide precision beyond ~+/-0.5%.

Here’s a helpful summary from the agent showing what this autonomous development cycle was able to accomplish:

Code profiling

Scenario: A developer needs to understand how long a function takes to execute to optimize system operation for latency, power, or another metric. This classic code profiling process uses pin toggling at the entrance and exit of a function to provide an externally measurable indication of execution time. For this test, I created a sketch with three simple functions that each block for a set period of time in a loop.

After a few test runs, I opened a new chat and instructed the agent to use the pin monitor to profile the sketch's three functions using a single pin. I had to specify that it could program the DUT multiple times to complete the task, but should revert the code back to its original state once testing was completed.

The agent methodically worked through the process, starting with FuncA() by adding a toggle at entrance and exit, then programming, running, and measuring. It continued with FuncB() and FuncC() without additional prompting, before presenting a timing chart. Impressively, the agent took an extra step in its findings by deriving an "expected" execution time based on the code and comments in the original sketch.

Proof of Concept Completed

These development mock-ups hint at how logic analyzers, oscilloscopes, and other instruments with MCP server interfaces could become invaluable for runtime debugging with agents in real-world scenarios. I'm eager to see how industry leaders like Tektronix, NI, Agilent, Saleae, and others integrate their products into agent-driven workflows. There are already numerous examples of instruments with AI running onboard to help with configuration and testing. It's likely that the optimal solution here will involve coordination between powerful, autonomous coding agents and instrument agents.

This series, along with another post in Switching Context, has explored how LLM-driven agents extend beyond chat windows into both digital and physical realms. I've always been fascinated by building functionality at the intersection of analog and digital worlds. Microcontrollers exist precisely at this crossroad. While software engineers have "Hello World" as their introductory example to verify system functionality, embedded engineers have "blinky"—the simplest possible embedded system application where a few lines of code create a visible change in the physical world. This connection between code and physical world is precisely what makes agent-driven instrumentation during embedded system debug so promising. It extends the capabilities of AI beyond text generation into tangible, measurable interactions with real-world systems.

I want to get Cursor to use this pin monitor MCP server for code profiling.

Subscribe now

Code profiling: handy if you’ve got the pins and a way to monitor them

Code profiling is a broad topic so let me narrow the scope. For this proof of concept, I’m referring to a technique where the developer toggles port pin states through lines of code strategically placed in the firmware project. A toggling port pin becomes a signal. It’s a fairly low tech, barely-intrusive way of determining the runtime state of a system.

For instance, if a port pin toggles to logic high when I enter an interrupt service routine, and toggles back to logic low when I exit, I've created a signal that reveals when that routine is executing. Using an oscilloscope or logic analyzer, I can then determine how frequently the interrupt occurs, how long it takes to execute, and other information.

Remember that my goal is to insert an AI agent into the virtuous development cycle. Specifically, I want to see if I can get Cursor's AI agent to use the simple pin monitor MCP server I created in my <last post> for profiling. I'm envisioning giving Cursor straightforward prompts like:

• "Profile all functions in this file and give me execution times for each"

• "Tell me with what frequency this interrupt executes"

Are AI agents clever enough to receive a prompt like that and know what to do through tools to accomplish the task? Think about the logic leaps it has to make:

1. The agent has to understand that I’m asking for code profiling, and understand that the tool it has available to accomplish this task is the Edge detector MCP server

2. The agent also has to recognize that in order for the pin monitor to work, it needs to add pin toggling to the function being profiled and then program an Arduino with this modified code

Wait…Cursor has to program an Arduino board?

Yes. The first step of code profiling is identifying strategic locations within a function to add pin toggles, creating meaningful external signals you can monitor. This task should be straightforward for the agent. While I'm uncertain whether the agent understands the concept of code profiling, if it does, adding pin toggling code should be simple.

Then, it gets tricky. The agent needs to actually program the Arduino. Here’s another instance where the agent must reach into the physical world.

Potentially this sounds like a job for another MCP server, this one for Arduino board programming.

Fortunately, Arduino provides a useful starting point: a command line interface (CLI) programming tool. This tool can compile and upload sketches to Arduino boards, essentially providing the same functionality as their Cloud IDE.

Although there's no official Arduino CLI MCP server, several developers have created their own versions. I found one, cloned the repo, and that's when the struggle began.

I discovered the "MCP tools" menu in Cursor's settings, which requires a JSON-formatted description of the tool. I had no idea how to format this information correctly, but ChatGPT did. After a few iterations with ChatGPT, I successfully got the MCP server up and running.

After the agent began to understand how to make use of the the Arduino CLI MCP server’s exposed tools, it was able to compile a test sketch and run it on an Arduino with my natural language prompt in Cursor’s chat. Pretty cool!

Hardware setup

I want to pause here and make clear that the board I just programmed is a different Arduino board than the one I’m using to emulate a pin monitor.

Right now my setup looks as follows:

From now on I’ll refer to the Arduinos as DUT or Pin Monitor.

Modifications to my pin monitor MCP

At the end of my last blog, the pin monitor MCP server enabled a client to read a pin’s logic state and subscribe to realtime pin transition updates. Unfortunately, Cursor can have issues with monitoring a stream of data through an MCP server. After working on the PRD for this with ChatGPT, it became clear that my simple pin monitor needed more smarts so that Cursor would be able to configure the pin monitor to autonomously capture events for a period of time and report back with a log of edge events.

This required changes both to the pin monitor’s python script as well as some minor updates to the Arduino sketch. The first phases of this project were spent with AI assist building out these new features and running tests at certain milestones. Cursor cranked out code at a breakneck pace, but unfortunately its test coverage turned out to be lacking!

Integrating with Cursor

integration into Cursor’s registered list of MCP servers was challenging. First of all, Cursor’s AI agent doesn’t seem to understand nearly as much about Cursor’s menu system and MCP server configuration as you might expect. This led to issues where the upgraded pin monitor’s MCP server tools were deeply mismatched from what Cursor expected.

The most glaring issue occurred when the agent coded upgrades targeting version 1.0 of an MCP server interface library. This created a significant problem since my Cursor installation used a newer, incompatible library version. To be fair, this wasn't entirely the AI agent's fault. I should have specified the version requirements in my PRD.

Figuring out how to make the MCP server script compatible with this new version of the MCP library proved to be a surprisingly difficult challenge for the AI agent. It made a few increasingly sweaty and desperate refactoring attempts before throwing up its (virtual) hands and starting to create a ‘simple MCP server that will work’, thus discarding all our previous work.

After halting that dead end, I pointed out to the agent that the other MCP server in this system, Arduino CLI MCP server, was working properly. Perhaps the agent could use that code as a reference. After examining the source code, the agent identified the necessary patterns and finally Cursor recognized all the tools exposed by the pin monitor MCP.

But can Cursor actually do anything with my server?

Now that all the tools of the MCP server seemed to be recognized, I expected smooth sailing, but that wasn't the case. None of the newly enumerated tools functioned properly. Despite trying various commands to monitor the port pins, nothing worked.

This was the point where my debug session entered a strange new phase. I asked the agent to review its own struggles with the MCP server alongside the MCP server source code itself. It was almost like I was asking the agent to take a time out and reflect on its actions.

This turned out to be an effective strategy. The agent identified a few issues in the MCP server python script and how it interfaced with the Arduino sketch. It even uncovered some bugs in the Arduino sketch itself. Apparently some state variables weren’t getting reset properly.

At the end of this self reflective prompting session, Cursor was able to build an MCP server with an Arduino board that the agent could successfully communicate with. The pin monitor worked.

Okay, so now does this PoC prove the concept?

Not quite! Although I believe the pin monitor MCP server is functional, and Cursor AI agent can finally understand how to talk to it, I still haven’t put all the pieces together: I haven’t asked Cursor one of those natural language code profiling prompts. In the next post in this series, I’ll try that.

Even at this point, it's becoming clear that AI agents with access to bench instruments and debug tools will transform the debugging and development process. Currently, AI coding agents excel at reliably and relentlessly fixing compile time bugs. Giving these agents access to bench instruments may enable them to become equally proficient at fixing runtime bugs.

The Virtuous Cycle of Embedded Code Development

AI code assistance is just beginning to transform embedded code development. The slower adoption pace compared to other software development fields highlights a key difference between embedded systems development in constrained environments and development elsewhere in the industry. To put it possibly too simply: embedded systems devs are too close to the metal for AI.

Unlike mainstream software development that sits comfortably atop layers of abstraction that hides processor and physical layer complexities, embedded systems engineers must intimately understand their hardware's limitations. These systems directly interface with reality, converting analog signals with ADCs, controlling motors, managing RF communications, and countless other physical interactions, all while juggling tight memory budgets and CPU constraints. Moreover, these codebases often need to comply with rigorous communications protocols, safety certifications, and industry standards that leave little room for error.

Embedded systems developers navigate these constraints by employing a superset of the standard debug practices all developers use. While they hit breakpoints, monitor build artifacts, and watch terminal output, those working with real hardware inevitably also use specialized instruments like logic analyzers, spectrum analyzers, and oscilloscopes. The development cycle becomes a virtuous cycle of code development:

Current AI assistance can support most of these steps, but significant gaps remain in the 'observe' step when instrumentation is required.

The ‘closer to metal‘ developers get, the more important instrumentation becomes. Consider someone coding a wireless radio stack that must comply with a standard like Bluetooth. In these specialized domains, the relationship isn't "human in the loop" but rather "machine in the loop," where developers strategically incorporate AI code assistance. Attempting to code physical and link layer functionality for Bluetooth compliance without a spectrum analyzer will burn mountains of tokens without the code ever coming close to standards compliance.

So how might AI code assist gain access to bench instruments and complete that virtuous cycle? I think the most likely path this evolution takes is that LLMs interface with instruments through Model Context Protocol (MCP).

MCP as LLM’s eyes and ears

Anthropic’s release of the MCP spec in November 2024 (https://www.anthropic.com/news/model-context-protocol) proved to be a watershed moment in AI for code development. It’s been less than a year since Anthropic released this standard to the world, and yet MCP already feels like an integral component of AI-based coding.

Sidenote: everything with AI happens at a breakneck pace, even standardization. I’m used to standardization taking years, with brilliant people in meetings for months arguing about the placement of commas in standards documents. Not so with AI.

MCP follows a client/server architecture where the server provides primitives that clients can utilize. These primitives include tools, resources, and prompts as defined by the standard. Through standardized initialization and capabilities expression, MCP servers enable LLM-powered agents to clearly understand the available services and how to interact with them.

While most MCP servers I've seen focus on providing code, documentation, and datasets as resources, the 'tools' primitive type in the MCP standard offers a pathway to integrate bench instruments into embedded code development. Using this standard, an agent should be able to:

• Learn the measurement capabilities of a tool and its configuration options

• Request measurements on demand

• Configure the tool to monitor for asynchronous events and subscribe to updates

Physical limitations still exist, of course. Logic analyzers need physical connections to port pins, spectrum analyzers must be wired into RF paths, and so on. But before exploring these constraints further, let me share what I built.

Mocked-up MCP instrument build

I wanted a simple proof of concept MCP server that functions as a bench instrument. Emphasis on ‘simple’: I built this mainly to understand how MCP servers operate, and how LLM-powered clients can interface with those servers.

The code is available at: https://github.com/parkerdorris/ArduinoButtonMonitorMCP/blob/master/button_monitor.ino

I’ve included my initial PRD for the project for reference, though the project evolved as Cursor built and tested each component.

Pin monitor

I decided to use an Arduino board as a single GPIO pin monitor, with an MCP server wrapped around it. I chose Arduino because I wanted to ensure Cursor could generate embedded code in one attempt. For constrained devices, Arduino provides a clean, simplified hardware abstraction that makes it extremely easy for Cursor to generate functional embedded code.

The Arduino code exposes a few commands over a non-MCP compliant serial interface. Commands include:

• STATE, which returns the 0/1 digital state of the pin

• SUBSCRIBE/UNSUBSCRIBE, which enables/disables pin monitoring events indicating "RISING" and "FALLING" edges of the pin

Note that there's nothing MCP-specific about any of this. This is just a dead-simple Arduino sketch.

MCP Server

The MCP server is a Python script that manages the interface with the Arduino board. The MCP server python script contains a class that handles the Arduino connection and establishes a WebSocket interface compliant with MCP standards.

The MCP server implements these functions:

• Connect/disconnect from the tool

• A method to retrieve button state

• A method to subscribe to button edge events

After a client subscribes to edge event detection, the server continuously streams edge events to that client.

MCP Client

The client script provides a system prompt to ChatGPT that instructs it to connect to the MCP server and narrate events. The terminal output is divided into two parts: the raw I/O communication between client and server, and ChatGPT's narration of those events.

Upon startup, the client establishes a WebSocket connection with the MCP server and queries the available tool capabilities.

Snippet:

main - INFO - Available tools: {'tools': [{'name': 'get_button_state', 'description': 'Get the current state of the button (0 or 1)', 'inputSchema': {'type': 'object', 'properties': {}, 'required': []}}, {'name': 'subscribe_button_edges', 'description': 'Subscribe to button edge events (RISING/FALLING)', 'inputSchema': {'type': 'object', 'properties': {}, 'required': []}}, {'name': 'connect_arduino', 'description': …

The code then checks the button state and subscribes to events.

When an edge detection event is sent from the server, the client echoes the websocket information. The LLM then provides brief 'narration' that assesses the button state within the runtime context.

📡 [MCP] Received button event: RISING at 13:03:51.522

State: 1 | Description: Button rising

2025-08-16 13:03:52,175 - httpx - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"

🤖 At 13:03:51, the button was pressed, changing its state to active.

Does this PoC actually 'P' the 'C'?

To answer my own question: not really. While wrapping an MCP server around an instrument works, and I can set an LLM to studiously monitor and narrate the life of a port pin, I'm still a long way from understanding exactly how that virtuous development cycle can be completed with MCP-enabled bench tools.

For my next step, I plan to create a more realistic developer workspace. I'll connect this pin monitor to a second Arduino functioning as a device under test (DUT). I want to explore whether Cursor/VS Code can interface with the pin monitor as an MCP client (this seems feasible, though I need to conduct more research).

With this mock environment in place, I'll develop an intentionally buggy Arduino sketch for the DUT and leverage both Cursor and the MCP server to debug and fix the issues. Following that, I'll explore creating more practical MCP servers for actual instruments, possibly developing a proof-of-concept that wraps the PyVISA library. The experiment should be illuminating—potentially exciting, potentially frustrating, likely both!

Given that embedded systems development is inherently resource-constrained and tethered to real-world physical interactions, I believe this domain will remain human-in-the-loop longer than most software development fields. Through these experiments, I'm attempting to envision what human-AI collaborative embedded development might look like in 5-10 years.

Human/AI collaboration in the virtuous loop will undoubtedly remain essential for embedded systems, even as an increasing portion of the human contribution may shift toward more menial tasks like AI requests to "connect your scope to GPIO 3" or "don't forget to ground your probes."

I’ve had a few jobs where product testing was part of my responsibilities. I've spent days meticulously following test plans—checking if bits in bytes were writable, confirming that setting a register to a specific value caused a peripheral to respond according to the datasheet, and more. One job I had was testing software installers, where I had to verify that every radio button functioned correctly and each cancel button truly aborted the intended action.

Click. Click. <check a box on test plan>. Click. Click. Click. <check another box on test plan>.

The rush I felt when I finally found a legitimate bug was incredible.

Eventually, I became the one writing test plans. Plot twist: creating the plans was just as monotonous as running them, but carried more stress since I now bore responsibility for defining test coverage.

As I’m typing this, AI tools are busily revolutionizing QA testing on code, as much as it’s revolutionizing code development itself. When I’m working on projects using AI assist, generating helpful test scripts is painless, though I have run into some issues.

When I was working on my first project, which I wrote about at the link below, I spent a lot of my time fiddling with the user interface. It was definitely more exciting testing my little web game than it was testing an installer, but still it was repetitive. I couldn’t think of a viable way to have AI test my interface autonomously.

Enter Agent Mode

I've been excited to try Agent Mode in ChatGPT since its announcement, though my reluctance to let it make purchases on my behalf severely limits what I can do with it. I know we're heading toward a future where LLM reliability approaches "trust with my credit score" levels, but I'm not there yet.

A Reddit post about using Agent Mode to play a web app game sparked an idea I felt comfortable trying. The multimodal toolset powering Agent Mode enables users to prompt the system to navigate websites, figure out how to move around, and scroll, click, and type its way through tasks until completion. What if someone asked ChatGPT to run user interface tests on a website?

Yes, I know automated UI testing exists, and many of these frameworks are impressive. Most of these tools offer pixel-perfect control of test steps—the finest granularity possible. For this experiment, though, I wanted to take an intentionally naïve approach: simply point an AI at an interface with limited training and set it loose to perform a task.

The hype around Agent Mode suggests it's extremely capable. Demos show it buying wedding outfits across multiple websites and completing other complex, multi-step tasks. If Agent Mode is so smart, surely it can figure out how to use my silly little web game, right?

Testing Round 1

In retrospect, my first test prompt could have been a little bit more naïve. Given the information that I provided, the agent got off to a strong start, successfully clicking ‘Play Game’. Unfortunately, this was basically the last point in this test where the agent seemed to figure out what to do, and that might have been mostly driven by my too helpful prompt.

Sidenote, i appreciate the window into the virtual machine, but I love the stream of thought text boxes that appear above the agent’s cursor. As with a lot of functionality in Agent Mode, this feedback evokes a similar view into the stream of thought in ‘deep research’ mode responses.

The agent could not get the ‘card flip’ control to work at all. What I thought was a simple user interface completely flummoxed the agent. I watched it sputter around for 9 minutes, trying different UI patterns that it was guessing might be supported. It seemed to miss the guidance on screen, or was unable to interpret it correctly. After 9 minutes, it basically gave up.

Being an obsessive problem-solving engineer in its recurrent heart, the agent tried taking the site apart in an attempt to figure out how the input controls worked.

Watching the agent try and fail again and again, reading its increasingly frenzied attempts to get the site to do anything correctly, was an emotional experience. I found myself rooting for the agent to finally figure out how to swipe.

By the end of this round, I had fully fallen into the role of ‘square hole girl’ in her brilliant, viral post from years ago:

I tried telling myself that whatever just happened was a limitation of the virtual machine. It can click, but maybe it can't swipe? In the documentation I've read, OpenAI doesn't provide details about the browser type used in Agent Mode or its supported features. That said, my game uses a React interface, part of my tech stack that's universally supported in modern browsers.

I had to admit that my web game was likely partially at fault for not offering failsafe fallback input controls.

Testing round 2

I noticed that the agent spent a lot of its time trying to use key strokes to advance the game, so I prompted Cursor to add keystroke support. Arrows to flip the cards and enter key for selection and screen navigation. This was a simple addition to the web app‘s UI that Cursor built into the project without any issues.

I tested the addition and immediately saw that this had been a missing feature all along. The interaction was a smooth and natural alternate control interface that should have been there at the start.

During the second round of testing, the agent still struggled with the interface. Strangely, this time the agent never attempted to use keystrokes and instead pawed and scraped around the site, failing to make much progress. At one point it seemed to go down a rabbit hole of trying to use a right click to make the game work.

It became clear at this point that what I was really using Agent Mode for was as a beta tester for my game. I was essentially watching over the agent's digital shoulder as it completely failed to enjoy playing my game.

Others reading this probably had this realization much sooner, so please forgive me!

With this "beta tester" idea in mind, I asked ChatGPT for feedback. It gave me several suggestions, including creating a one-time tooltip with instructions that would appear on screen. It even helpfully provided starter text for the tooltip.

I dropped the markdown file containing the tooltip text into my project and asked Cursor to review and plan for implementation. Cursor, with its deeper knowledge of the project than ChatGPT had, suggested two tooltip variants—one for desktop and one for mobile. After a few iterations to refine the tooltip's appearance, I was satisfied with the result and pushed to origin. Vercel deployed the new version of the site, and I returned to ChatGPT.

In my last blog I mentioned how iterative meta prompting seemed very “help me help you.” This experience of ChatGPT recommending tooltips to help Agent Mode navigate my game felt like the opposite side of that coin, with ChatGPT explaining how I could help it help me build a decent interface.

Testing round 3

I started a new chat in ChatGPT and prompted it to perform the test again. I started a new chat because I wanted the agent to start clean — to be a new user. With the tooltip showing up on screen in the agent’s virtual machine window, I had high hopes for this round of testing.

Then something amazing happened: the agent actually started playing the game.

Turns out the agent is not great at identifying AI-generated images, even though it seems to have ideas about AI generated image ‘tells’ like unusual symmetry:

Seriously, the agent was bad at this game:

Out of five rounds of play, the agent would only choose the AI generated image correctly one or two times.

Final Feedback from the Beta Tester

For me, the real value of this test was recognizing that Agent mode could function as a beta tester and provide actionable feedback. When I finished my final round of testing, I asked for a final assessment of the game. Was it fun?

Here's its assessment. ChatGPT’s choice of emoji says it all!

😐 Enjoyment

Strengths:

• Quick, low-friction play: no account, no login, just straight into the game.

• Each round takes a few seconds, making it bingeable.

Weaknesses:

• No real sense of progression, story, or stakes.

• No contextual feedback or visual reward for success.

• Without explanations, it feels more like guessing than learning.

Overall:

The game is moderately enjoyable for 5–10 rounds. It’s a great casual test of perception, but lacks the hooks (like streaks, leaderboards, rewards, or feedback) that would make it sticky or viral.

“Moderately enjoyable.” Tough but fair, though I think it’s funny that the chat chose to italicize moderately enjoyable. That’s not something I added for emphasis. Is ChatGPT learning how to be sarcastic?

Next steps

Though this experiment was basic, it reveals several promising applications for Agent Mode in ChatGPT:

• The agent's experience seamlessly feeds back into the chat, allowing for post-mortem discussions about UX issues.

• ChatGPT not only identifies problems but also provides specific coding solutions to fix the issues it encounters.

• The logical next evolution would remove me from the process entirely, enabling automated UX testing directly within code assist tools like Cursor.

It remains unclear, however, if OpenAI intended Agent Mode to be used for this purpose. Between my second and third test rounds, my access to Agent Mode disappeared for at least 24 hours—possibly due to usage limits, but perhaps not.

Until I get more definitive information, I'll continue using Agent Mode to test my projects. Watching it struggle with my interface taught me valuable lessons, and its frustrated comments highlighted genuine problems I had overlooked.

I'm certain that major GUI testing vendors will eventually create tools with interfaces similar to what I improvised with ChatGPT—offering more detailed control, better visibility, and more actionable insights. But for a quick UX reality check on whether a feature makes intuitive sense, why not run it through Agent Mode and ask for feedback?

But what does it all really mean

While it was exciting and even stressful to watch the agent attempt to play my web game, I noticed that I was feeling something else as well. The experience was unnerving in a way that I had trouble understanding at first. Eventually I figured out that my uneasiness stemmed from the realtime nature of the agent mode window. I first felt a hint of this when Deep Research mode was released and we could see a sidebar with a point-by-point "chain of thought" driving the analysis. But that stream of thought still felt interior to the LLM. Agent mode, however, is the LLM venturing out into the (digital) real world.

I know that agentic tools operating keyboards and mice to navigate the web fails to meet any definition of ‘embodied’ AI, but it doesn’t feel quite as ‘disembodied’ as before. Just a few days back, someone documented seeing Agent Mode encountering an "Are you a robot?" captcha and clicking "no".

The AIs are here with us on the web, right now, struggling with unintuitive interfaces, flailing around and madly clicking on unresponsive controls, alongside the rest of us.

Subscribe now

How it started vs. how it’s going

I’ve dealt with AI's relentless, accelerating, and seemingly existential threat to my career in IoT with grace and composure. At least, that’s what I like to think. In hindsight, it’s clear that my evolving attitude these last few years followed five familiar stages:

1. Denial: “Oh wow, AI can write a dumb haiku about my cat? Not exactly HAL.”

2. Anger: “Could there be a bigger waste of time than prompting an AI to tell me things I already know? I’m an expert!”

3. Bargaining: “Fine, I’ll generate a deep research report using AI when my boss asks for it. Faster than doing it myself.”

4. Depression: “I should have majored in Computer Science instead of Electrical Engineering. Maybe then I would have known what a perceptron was before it was too late.”

5. Acceptance: …here we are

The more I use AI—both at work as a product manager at a semiconductor company, and in my off-hours tinkering—the more intrigued I become. Is this hope I’m feeling? Or Stockholm syndrome?

Quick background: I’ve spent twenty years in IoT, nearly all of it at the same company. I started in applications engineering, moved into firmware, and eventually product management, focused on Bluetooth. I liked my job in the pre-AI world the way dinosaurs probably enjoyed sunny weather before the meteorite.

Now I’m writing a series about the projects I’m working on, what I’ve built, and what I’ve learned. I’ve written a number of technical articles, but usually from a place of authority. That won’t be the case here. Like a lot of people reading this, I’m learning as I go. If this helps someone else push through their own stages of career grief—or at least feel less alone in the process—I’ll count that as a win.

Spaghetti/Code

I learn by building. I decided to work on a series of AI-centered projects guided by a few principles:

• The projects had to teach me something I didn’t already know about AI

• I had to ‘ship it’, whether that meant launching a site or pushing to GitHub

• Have fun

When generative AI first went mainstream, I found comfort in how horrific the AI-generated images looked. Back in 2022-2023, Will Smith eating spaghetti looked like a scene from an abandoned Cronenberg film. These grotesque outputs from primitive models were reassuring precisely because they were so obviously artificial.

Now, of course, AI-generated images are much harder to identify. Noses typically appear in the right places and most subjects have the correct number of fingers and arms. Still, we're in an era where these images have certain tells. If you know what to look for, you can spot subtle inconsistencies in lighting, unnatural textures, or peculiar details that just feel wrong. For me, spotting what makes an AI image artificial has become a kind of game. I began wondering if I was better than my friends and family at identifying these AI markers. That's where I got my project idea.

Problem: I didn’t know how to code a website from scratch. I'm an embedded systems guy! I specialize in resource-constrained MCUs, writing machine code, counting cycles, and squeezing data into bits. Javascript is definitely not my native tongue.

I would rather not call it ‘vibe coding’

Before I even started coding the site, I bombarded ChatGPT with basic web development questions that needed expert answers. While I can’t imagine developing any kind of meaningful attachment to AI, it was liberating knowing that I had a judgement-free digital officemate who answered my "Website 101" questions without any eye-rolling.

One of the aspects of a website I understood the least about was how various components of a website were hosted, and how they interact with each other. Using ChatGPT as my patient guide, I settled on Vercel for my frontend running a React app written in TypeScript, Render as my Python-implemented backend exposing endpoints for image retrieval from a Supabase database and image store. For coding, I chose Cursor as my AI assistant.

All these sites were new to me, and without a doubt there are more cost effective platforms to launch sites from, but I went with these because the balance they struck between ease of use and technical visibility helped me learn.

After a few attempts at building with minimal prompts in Cursor that resulted in mangled, half-functional proof of concepts for site components, I learned I needed to be more disciplined. "Garbage in, Garbage Out” is a concept that's alive and well in this AI-prompted coding environment.

I discovered that my product management skills in creating Product Requirements Documents were useful, as AI coding tools respond well to PRDs as structured prompts. After researching "meta prompting" (essentially prompting about how to prompt), I fed my PRD into ChatGPT's canvas. The process felt like a "help me help you" exercise, but I was impressed by the results after a couple of iterations. Learning to communicate effectively with AI taught me several important lessons:

• Clear, unambiguous, and detailed descriptions is essential, but:

• It’s also important not to hammer out every detail in your head; at least in my experience so far, too much detail can result in a kind of ‘overfitting’ in the generated output

• Use an AI-generated PRD template improves structure and drafting efficiency

• Don't take it personally when meta prompting tears your own writing apart

That last point was the hardest for me to learn.

When I felt like I had reached a point of diminishing returns with meta prompting, I added the PRD to my project directory. After instructing Cursor to review, it only asked a few questions before creating the project’s codebase step-by-step — literally ‘step by step’, because my PRD prescribed build phases that Cursor followed.

The code generation speed was overwhelming. Instead of watching the code fly by, I focused on Cursor's AI pane that summarized each file and function it created.

One of the first components I built was the title screen:

While most components worked correctly, I struggled with the UI and game's state machine. I spent 90% of my time fine-tuning the core UI feature where players swipe to flip image pairs. This made some intuitive sense, because of the inherently human subjectivity of what I was fine tuning. Then again, maybe I should have spent a little more time on the PRD.

After a few hours of prompting and testing, I had a working React-based frontend with a Python-driven backend pulling dummy data from a database.

One more big problem: the game wouldn’t work without a sizable repo of images, and not just the handful of placeholder images I’d been working with so far.

I scoured the web for free photo repositories but hit a wall when it came to AI-generated images. Where would I find enough AI-generated content to create a game with meaningful variety?

The answer was obvious in retrospect. I'd have to create them myself.

It’s AIs all the way down

My knowledge of image generation matched my website coding expertise: near zero. I turned to ChatGPT with basic questions about finding image models and generating the thousands of images I needed.

Given that my desktop is a modestly equipped Mac mini, I had to take a crash course on server rental that lead me to runpod.io, where I could rent out powerful GPUs by the hour. On one of these pods, which were basically virtual machines, I could run Python scripts that launched ComfyUI, a node-based interface for building and executing image generation workflows.

The first time my script worked, and I opened up a png to find a (nearly) photorealistic middle-aged guy staring back at me. He had soulful eyes, a salt and pepper beard, and skin that looked as plastic as a Funko Pop. Perfect.

Next problem: each prompt generated one image, but I wanted thousands so that the game wouldn’t become immediately boring as images appeared repeatedly.

Living life on the edge without a proper PRD, I prompted Cursor to create a second project just for batch image generation. The script used ChatGPT to generate thousands of prompts that were then fed into the generative image model. Just to see what would happen, I also fed all the resulting image through another model to generate embeddings (multidimensional points in semantic space that ‘describe’ the image in a way that’s searchable based on input prompt similarity….just in case I ever wanted to add that feature to the game). AI generated a script that used AI to prompt and AI to generate images that were then characterized by AI.

I could have spent more time testing sophisticated generative image models beyond SD 1.5, but I deliberately chose not to. I wanted players to have a reasonable chance of scoring well.

After observing my friends and family identify AI-generated images from my repository with at most 90% accuracy, I believe my decision to avoid using leading-edge generative models was justified.

What’s next?

The game is live today at slopornah.com. I could have launched faster, but got sidetracked learning about website security and Google Ad Sense. With those tangents factored in, what should have been a weekend project stretched to three weekends, though it would have taken years to learn and code everything myself.

The deeper I got into this project, the more my fears about AI's impact on my career faded. AI will certainly transform my day-to-day work in profound ways I’m only beginning to grasp. But I work in tech. Did I really expect tradition and security?

I remember older developers in the 90s reminiscing about their "good ol' days" with punch cards and basement mainframes. Their experience was as foreign to me then as coding without AI assistance will seem to future programmers.

As engineers, we are agents of change. We have a responsibility to innovate and ensure our innovations make the world better. Generative AI's transformative potential is both more dangerous and promising than any technological advance in decades. That's why it's crucial for all of us to take responsibility for AI. It’s up to all of us to harness this technology for positive change to create tools and products that make people happier, safer, and more prosperous.

We've evolved from punch cards to machine code to higher-level languages, building more, experimenting further, and occasionally creating technology that genuinely transforms lives. I'm excited about the innovation and efficiency that AI unlocks. Infinitely patient, impressively competent, and occasionally confounding, it's already helping me start my next project.