AI is changing how products behave and how they are built. Product lawyering has to change too.
Product counsel have always needed to understand their products. What does the product do? What does it not do? How does it work? Two AI-driven changes complicate that job. First, many products now include LLM decision-making and output as product features, which can make product behavior unexpected. Second, product teams now use LLMs to build products, which can leave no one with a complete understanding of how the product works. Product counsel will need to adapt to both changes.
 |
| From United Editors Encyclopedia and Dictionary: A Library of Universal Knowledge and an Unabridged Dictionary of the English Language, United Editors Association, 1907. |
LLMs as Product Features
The first major change is that some products now incorporate LLM responses. LLMs differ from most previous computer algorithms in the extent to which non-determinism is both important and valuable. For product-counsel purposes, the key point is not mathematical randomness but operational unpredictability. The same or similar inputs may produce different outputs, and even great engineers cannot guarantee in advance what the model will say.
Chat interfaces, such as ChatGPT or Claude, provide the clearest examples. In the days of deterministic software, a user who typed “tell me a story” might get a predetermined response, perhaps Rumpelstiltskin or a random choice from a fixed set of stories. Prompt Claude or ChatGPT with “tell me a story,” and you will get a different story each time.
There was once a lighthouse keeper named Maren who lived at the edge of a crumbling cliff on the northern coast. Every evening, she climbed the spiral stairs and lit the lamp, and every morning, she extinguished it. The rhythm of her days was as steady as the tide. …
There was once a lighthouse keeper named Maren who lived on a rocky island so small you could walk its entire shore in seven minutes. She had no boat, no radio, and no particular desire for either. Supplies came once a month by ferry, and the ferryman never stayed longer than it took to unload the crates. …
There was once a lighthouse keeper named Maren who lived on a rock so small it didn't appear on any map. Every evening she climbed the iron stairs, lit the lamp, and watched the beam sweep across the water like a slow, bright hand waving to no one. …
A non-LLM product might convert “Tell me a story,” “Spin me a yarn,” “Please can you tell me a story,” or even “Tell me a story” (with two spaces before “story”) into the same story query. An LLM will treat all of those prompts as requests for a story, but small wording changes will still change the output.
“Tell me a story”
There was once a lighthouse keeper named Maren who lived on a narrow spit of rock three miles off the coast. Every evening, she climbed the iron spiral to the lamp room, lit the wick, and watched the beam sweep out across the dark water like a slow, bright arm reaching for something it could never quite hold.
…
Years passed. The coast guard automated the light. They told Maren she could leave. She stood at the door of the lighthouse with a bag over her shoulder and Compass at her heel, looked back at the beam still turning without her, and felt something she hadn't expected: not sadness, but gratitude — for the years of dark water, and the one bright thing that had drifted out of it.
She walked to the ferry. Compass walked beside her. Neither of them looked back again.
“Spin me a yarn”
A lighthouse keeper named Margot had one job: keep the light burning. Simple enough, until the night the fog rolled in and the light started answering back.
…
Margot laughed, alone in her tower above the black Atlantic, and kept the light on a little longer that night. Some conversations are worth the overtime.
“Please can you tell me a story”
There was once a lighthouse keeper named Maren who lived on a narrow spit of rock far from the mainland. Every evening, she climbed the iron spiral staircase to light the lamp, and every morning she climbed again to snuff it out. The rhythm of it — up and down, light and dark — had become the pulse of her life.
…
The storm broke by morning. Maren watched his ferry pull away, the case tucked safely under his arm. She climbed the stairs and snuffed the lamp, same as always — but the tower didn't feel quite so quiet anymore.
“Tell me a story”
There was once a lighthouse keeper named Maren who lived at the edge of a rocky coast where the sea never seemed to rest. Every night, she climbed the spiral stairs and lit the lamp, watching its beam sweep across the dark water like a slow, golden arm reaching for something it could never quite hold.
…
“Because," she said, lowering her arms at last, "you can't save anyone from where it's comfortable."
The same or very similar inputs can produce meaningfully different outputs. In these examples, the material difference lies in the emotional resonance of the story. In your product, the difference could create a security risk or legal jeopardy. In a customer-support product, the system might promise a refund. In a health product, it might give inaccurate medical-sounding advice. In a copyright-sensitive product, it might summarize a book or reproduce too much of it.
AI engineers use many techniques to make products with LLM features behave more like traditional software. These techniques can reduce risk, sometimes substantially, but they do not provide the same hard guarantees as traditional rule-based systems. It remains unclear whether that will change.
LLMs as Product Builders
The second major change is that teams now use LLMs to build products. More and more product-development work is being done by LLMs rather than humans. The approaches vary, but they all sit on a continuum: each transfers more or less software implementation from a human to an LLM.
Ask Antigravity to create an online store, and the LLM will make many implementation decisions. Depending on the human's involvement and the tool's expressiveness, the person “coding” the product may not know those implementation details. What user information does the product receive or store? For how long? Can two users have the same username? Does the product verify a user's email address? Many such decisions are being “made” by the LLM. Code review may let an engineer peer behind the curtain, but code can be complex, and reviews may get less thorough as LLMs get better at coding.
Each change radically alters the role of lawyers who work with products.
An Example from the Golden Age: Google Book Search
As Google Book Search's product counsel, it was my job to make the world's books searchable legally. Copyright law was central to that work. To understand which books were relevant to a query, we needed to copy them. That meant scanning books in many of the world's great libraries. That scanning created a copy, which implicated copyright law, so we had to design a product that stayed within the bounds of fair use both in how it treated those copies and in what it showed users.
Even if copyright law had not existed, we wanted to help readers find authors' books and hoped to help, not hurt, book sales. Our solution was a deterministic algorithm that showed a snippet from a book in response to a query. Because we did not want search to substitute for getting and reading the book, we took steps to keep users from stringing snippets together. The code could be guaranteed to follow the rules we established, and the engineers who wrote it knew it intimately. They could answer my questions. Google was sued over Book Search, and judges agreed that it was a fair use of the books based on those design decisions.
 |
| A "snippet" from a book in Book Search. |
OpenAI’s product counsel have a harder job. OpenAI also makes copies in order for ChatGPT to be able to respond to queries, but it is much harder for its lawyers and engineers to design its responses because they are non-deterministic.* Telling an LLM to “never return more than fifty words from a passage, never return two fifty-word chunks from any one page, and never return more than five passages per book” cannot be guaranteed the same way we could guarantee rules in Google Book Search. Indeed, in the New York Times lawsuit against OpenAI, the Times submitted an entire exhibit that purported to show ChatGPT reproducing large portions of Times content as part of its argument that OpenAI violated copyright law. Instructions to an LLM are weaker than hard guarantees. In adversarial settings like litigation, that difference can matter.
Some AGI optimists who believe AI will soon have essentially infinite intelligence will argue that LLMs will become good enough to take law into account in their responses. Or perhaps, as LLMs get better at mimicking human writing about copyright law, the product itself could respond in ways that mimic how a human might apply copyright law in each response. That would allow product counsel to shift from establishing product rules to issuing simpler behavioral statements, such as “don't violate relevant law.” Anthropic already uses what it calls a “constitutional” approach, based on broad statements, to address many model-behavior issues.
I am skeptical that “future AI will solve all problems.” Still, LLMs might let product counsel add nuance to product design. In Google Book Search, for example, we suppressed snippets for all books we believed contained short-chunk content, such as dictionaries, so users could not avoid buying those books by running searches on our site. That was a coarse determination. Perhaps LLMs would have allowed finer-grained determinations closer to our real intent: give searchers enough to know that a book would be useful, but not enough to substitute for getting the book. Rule-based algorithms struggle with that line. Even current LLMs might do better much of the time.
LLM-Generated Products
The trend in product development is toward AI-written code. Future product counsel may no longer have access to smart humans with deep knowledge of the code. Or they may have access to smart humans with less deep knowledge of it.
For example, while developing a watch interface for local London bus schedules, I wanted to show bus times for stops near a user's location. Using Claude Code, I “wrote” the application in a coding language and framework I had never used. I “developed” a working product, creatively named “BusWatch,” and shipped it without ever looking at the code. From using the product, I knew that it correctly displayed bus stops and times based on my location. I had educated guesses about how it did that, but no guarantees.
How a product works often matters to whether it is legal (or ethical). In BusWatch, the user's location is private information. How did my product get that information? How granular was it? With whom did it share the information? What other information was linked to it? Was the information stored? The creator of a product built by an LLM coding agent may have limited knowledge of those answers, or none at all.
BusWatch sits at the extreme end of the LLM-coding continuum, but every point along that continuum moves product development away from deep engineering knowledge of how a product works. As LLM-assisted development spreads inside companies, teams may produce more code than traditional review processes can absorb, and fewer people may have a full mental model of what was built.
This makes product counsel's job harder. At Google, I could ask the product manager and engineers to test my assumptions about how a product worked. The engineer who wrote the location code could have walked me through granularity and user consent. The PM could have confirmed with the team that the product did not store location or share it with anyone other than the bus-times supplier. With LLM coding tools, there may be no person with mastery of any part of the written code. That may remain true even if companies still require code review, documentation review, privacy review, or other reviews. Additionally, LLM-coded projects may shorten the time that product counsel have to review as products move from idea to prototype to launch on a compressed timeframe.
On the other hand, code is now more transparent to the average lawyer than it has ever been. An LLM coding agent could digest the entire BusWatch codebase in its context window and answer questions about it. Despite the risk of errors, the ability to interrogate code directly gives product counsel a tantalizing new tool. A lawyer who cannot read a codebase can now ask an LLM to explain data flows, identify API calls, summarize permissions, or generate tests for risky scenarios. An LLM with specific instructions about what to look for could helpfully summarize technical and legal intersection points. Those answers need verification, but they change the lawyer's practical relationship to products.
Even more transformatively, product counsel could become coauthors of code, shaping products so they achieve their missions while complying with law. Engineer time would no longer be the main bottleneck to that kind of product reworking.
Testing and Evaluation
Testing and evaluation offer even stronger answers. Product creators face the same problem as product counsel: they need to ensure that their products do what they intend. Between LLM errors in code interrogation, the relative opacity of LLMs used in products, and the non-deterministic parts of modern products, it is harder than ever to say reliably what a product does and does not do. Software engineers are putting more emphasis on testing and evaluation. Given the stakes of many product-counsel decisions, product counsel should too.
Many coding frameworks for LLM coding agents, including Jesse Vincent's excellent Superpowers framework, rely on testing-centered methodologies. These frameworks emphasize test-driven development, an older software-development methodology in which tests are written before code and code is written to pass the tests. In that process, tests stand in for specifications. Whether the code has been successfully developed depends on whether it passes the tests.
Product counsel can and should think about how to test legal constraints as applied to the products they advise. In BusWatch, for example, tests with network monitoring could verify that, at least in the tested situations, the product sends network traffic only to known destinations. Similarly, tests specifying the expected location data could verify what information the product collects. LLMs can help generate test data and test pathways, which may make this kind of testing easier. But product counsel should be careful about delegating test generation entirely to LLMs; otherwise they may recreate the same LLM problems they were trying to solve.
AI engineers are also increasingly using evals, another kind of product analysis, for products with LLM features. Whereas tests ensure product guarantees are met, evals are more useful to add qualitative assessments. In a recipe product, for example, tests might ensure that the recipes don’t contain the words “glue” or “arsenic” while evals might evaluate recipe healthiness. Tests either pass or fail. Evals could help rate the recipes and determine whether a new model actually improves the product. Before LLMs, a designated group of people or a subset of users often performed evals, and product teams integrated those results into release processes. Now LLMs often judge evals, so a human-specified goal gets scored by an LLM. Yes, it is turtles all the way down.
There Was No Golden Age
I am a big fan of humans, but some of the problems I identify here also existed in the golden age of artisan hand-coded products. Internet history is littered with products that broke the law. Some came from companies that did not care what the law was, or knew they were operating illegally. I assume my readers would not do that.
Other products broke the law because the engineers did not quite understand what their products did, or because the lawyers did not understand the products well enough to advise properly. Misunderstandings between human engineers and lawyers happen. Engineers build products with unintended consequences. Engineers and lawyers make mistakes. Humans forget things. All of that happened in deterministic software written by well-intentioned humans.
Products with non-deterministic features and products built with LLM coding agents change the types and frequency of these problems. They do not create the problems from scratch. There was no golden age of product counseling in which none of them existed.
What Product Counsel Should Do Now
Changes brought by more powerful LLMs do not eliminate product counsel's traditional job. Product counsel still need to understand what the product does, what data it uses, what rights it implicates, and what harms it may cause. But LLMs in products and LLMs used to create products make product behavior harder to predict and product implementation harder to understand.
Product counsel should therefore use tests, evals, and ongoing monitoring to understand product behavior and guard against risk. They should also use LLM tools themselves to take a more hands-on approach to understanding products and to have better conversations with product teams about how those products work. These approaches will make product counsel more empowered and more tightly integrated into product teams. They may also give product counsel more room to think creatively about how to deliver on a product's mission without increasing legal risk.
* I’m not engaging here with the separate interesting and important questions of whether ChatGPT is legal or good, or who should win the various legal challenges to LLMs and chatbots.
