As an industry, we’ve been struggling with what to do about the gap between engineers’ ability to review code and the quantity of code an LLM can produce in a day. We’re debating what will come of self-review (reading your own code before shipping it) and code review (someone else reviewing your code). In speaking with friends at various companies, I mostly hear about abandoning review. While agents generate nearly all of the code we ship to production at B12, we haven’t given up on reviewing their output and understanding how it works, and have been looking for ways to make that review process more palatable.

In the past few months, we’ve settled on a development pattern we call prototype-plan-stack. In the prototyping phase, 1-3 engineers implement a feature end-to-end in our codebase, prioritizing a working demo with reduced focus on some architecture/implementation details. Those engineers, paired with agentic coding assistants, might generate 5000-10000 lines of code without yet having a full understanding of how the code works. In a short planning phase, the engineers then take any feedback and reflections we have about the prototype and iterate with an agent to propose a plan to slice the implementation into 10+ pull requests that are stacked/dependent on one-another and small enough to each be reviewable in a sitting. In a final stacking phase, engineers review the individual PRs in the stack, shipping them in pieces they can digest and comprehend. As they review, they update the plan to include testing and architectural changes, making the stacking phase a relatively dynamic process.

In the rest of this post, I’ll cover the three phases of prototype-plan-stack in more detail before leaving some closing thoughts.

Prototype

A caveat before diving into this section: prototyping can mean many things. Some product teams prototype many ideas, discard many of those, and ship a select few. We don’t do that. We tend to vet ideas ahead of time given user feedback, what’s happening in industry, etc., and then commit to shipping the smallest version of an idea to see how users interact with it. The prototyping phase helps us identify the tricky parts and open questions around a minimal version of a feature we’ll learn from in the wild.

The prototype phase’s goal is to give 1-3 engineers 2-3 days to show an idea working on a branch in your codebase. The input to the process should be minimal: You might have a few discussions or brainstorms ahead of time about major flows, you might share a few user stories of pain in the current experience, or you might be working off a napkin sketch. You might also discuss some architectural considerations and gotchas depending on the needs of the project. It helps to have the engineers agree on 1-3 concrete stories they will be able to demo with their prototype, but don’t worry about nailing down detailed wireframes/mockups/requirements.

The focus here is the ~5-minute demo in which one of the engineers walks through the updated product. To help keep focus on the demo, it helps for engineers to have a document open where they can write down all of the sins, issues that didn’t stop the world, and ideas/questions they had that they didn’t get to. As thoughts pop up that don’t block the demo but would block shipping to production, they can live in this document.

If you properly timebox and focus on a small number of concrete user stories, then at the end of the prototype phase, you’ll come out with:

  • A 5-minute demo,
  • A list of corners the engineers had to cut to make it all come together,
  • A wishlist of features or architectural considerations that they didn’t get to,
  • A bunch of open questions that didn’t get answered, and
  • A not-fully-formed braindump of what the engineer(s) would do if they had to ship this in a few weeks.

A demo focus doesn’t mean that engineers don’t care about the code or implementation details at all. For example, when we were prototyping an upcoming project involving connectors to third-party APIs, an engineer used some of the prototype time to better understand how we’d abstract away the OAuth flow for each connector.

This form of prototype has different benefits from the traditional requirements document or clickable prototype. Most importantly, the prototype and demo live in the codebase. You’re not walking through an abstraction of the experience, but are instead faced with the constraints and reality of your product, and leave with the conviction that what you’ve seen actually does work in your product. Secondarily, the product surface area I’ve seen in these prototypes is larger than the surface area I’ve seen in wireframes/mockups generated in the same time frame, even if it’s not as refined as the experiences I’ve seen in a designer’s output. Finally, the prototyping engineer gets to face their architectural assumptions and ideas about technical constraints, coming out more informed on some of the implementation details or open engineering questions to discuss with the team.

Plan

You’re now sitting on a promising demo, a pile of unreviewed code, and a document with a wishlist and confessions of the engineers that created the prototype. You might also have a little bit of feedback from people who saw the demo. Time to figure out how to ship the first version!

Planning is a short phase in which you imperfectly figure out how to ship the prototype in pieces. Unlike a traditional engineering plan, this phase is usually completed in a morning or afternoon and is a lot less technically refined. You plan in concert with an agent, producing a plan to take the thousands of lines of code from the prototype and turn them into a stack of dependent pull requests in the range of ~hundreds of lines of code. Effectively the agent is instructed to figure out how to turn an unreviewable blob into a series of reviewable ones.

After getting any feedback on their demo, an engineer feeds the agent both the branch with the too-big pile of code and the document containing cut corners/wishlist of features/open questions with answers that came out of their prototyping effort. They ask the agent to propose a stack/series of several-hundred line pull requests. The agent or engineer also proposes a gate: something like a feature flag, experiment configuration, or stack order that hides the new feature until you’re ready as you merge intermediate pull requests. Picking the gate is as important as the order of the stack: individual PRs of the stack can only be merged independently if they won’t have a user-visible effect until later elements of the stack are merged. While some early PRs in the stack introduce dead code (a data model that’s not used, a view that isn’t called), the gate is used to protect user-visible interface updates or code that would otherwise cause a side effect from running.

Here’s a real example from a feature we shipped to allow users to schedule B12 agent tasks for the future, or on a recurring basis. Three of us built a prototype that included scheduling as one of three features during a 2-day hackathon, and I shipped this third of the prototype using the stack described below. By virtue of the prototype itself breaking into three parts that shipped independently, this is the smallest feature I’ve shipped using stacks, which makes it easiest to follow in full:

We’ve got an agent skill that guides the agent through proposing a stack and gate using B12y mechanisms for experiments and feature flags, but agents seem to be pretty good at doing this without too much direction. When the engineer and agent agree on the right list of stacked PRs and feature gate, the agent writes the plan to a markdown file inside the prototype branch, and the engineer may get feedback on the big-picture plan from peers.

Stack

And now: we ship it! At a few hundred lines apiece, the individual pull requests of the stack are bite-sized and reviewable, unlike the original several-thousand line PR that would result in a rubber stamp and no internalization of how it works. At the moment, the prototyping engineer and a second reviewer code review every PR in the stack, but we’re actively exploring ways to reduce secondary review on less risky PRs. Since each PR in a stack tends to focus on a smaller set of components, it’s easier to recommend tests and architectural improvements as engineers understand more of the implementation.

One interesting thing that happens as you review the PRs: the stack changes. I find myself reading the data model or business logic of the prototype and realizing we’re not handling something like a class of use cases, security considerations, or scalability concerns. Since there’s already a markdown file that documents the stack, it’s easy during a review to tell the agent to “add a final PR before launch to update how we do sandboxing to…” or “add a post-launch PR to rename/refactor this component…” Since review is the first time you’re deeply looking at the implementation, it’s natural for the plan to change, and the agent has a clear place to document these changes without distracting from the current PR. To make this concrete, here are some extra PRs and details we inserted as we reviewed the stack for our scheduling feature:

So we shipped the scheduling feature in a few more PRs than the original 6. There were also a few follow-on PRs after making the feature user-visible, like data model cleanups and better handling of concurrent scheduling runs.

I’ve noticed a bit of variance in how engineers approach stacking. Some engineers create the entire stack up front, so they create a series of 10+ PRs before reviewing the first line of code. This is nice as it allows a reviewer to use GitHub’s new stacked pull request interface to visualize the plan. The downside is that any change in an early PR has to be passed through to every subsequent PR that depends on it. That’s OK because an agent is doing the tedious work, but pretty slow and token-consumptive nonetheless. Since I change my mind a lot on the design and plan as I read PRs, I prefer to create 2-3 PRs at a time, allowing me to review one, send it to someone else to take a look, and start on a second or third one without getting too bogged down having to wait for the agent to update long dependent PR chains.

This isn’t new, but

Since long before agentic coding, engineers have excitedly submitted too-big pull requests for review. And in many cases, a collaborator would suggest breaking the large unreviewable PR into pieces. A few things feel different about the flavor of prototype-plan-stack that I’ve experienced recently:

  • Since writing lines of code used to be more challenging, the size/scope of the prototype used to be smaller before you decided to break it up. It used to be harder to generate PRs that rounded up to 10,000 lines of code, which made it more of a tossup whether you’d decide to break PRs up into pieces. Two concerted days of prototyping now easily gets you to a pile of code that you can either review in pieces (how else?) or merge without meaningfully reviewing (hopefully with other forms of verification).
  • The frequency with which you encounter large piles of code is higher, both due to the ease of producing lines of code and because more people feel comfortable prototyping. If you’re encountering one or more large piles of code a week, you’re naturally going to have more opportunity to flesh out the process/workflow for shipping those piles.
  • Plans are generated later and less precisely. When it was slower to generate code, we’d often research and write plans before writing much code. The plans took a long time to generate and review, and they couldn’t account for the unknowns we’d encounter after starting the implementation. Now that agents are part of the exploration process, you can learn a lot about the decisions you’ll have to make by having an agent build something imperfectly, and then coming up with a stacking plan is both less theoretical and faster to generate.

How does prototype-plan-stack generalize?

In prototype-plan-stack, the prototype helps us change our mind about what we’re building, and the stack helps us change our mind about how we ship it.

One related topic I’ve been thinking about is the costs and benefits of review. I have reasonable confidence that it’s less tiring to review a stack of small PRs, and that you’re less likely to rubber stamp smaller PRs in a stack than one big ball of code. But I can’t claim that shepherding or reviewing a stack of 10+ PRs isn’t still tiring. Internally, we’re asking ourselves what types of PRs in a stack require what level of scrutiny, and looking for ways to reduce review load around lower-risk changes.

I’m curious how companies that view prototyping or review differently address some of the problems that prototype-plan-stack solves for us. In particular, I wonder which aspects of the pattern remain useful in organizations that have bypassed self-review and code-review more completely. Presumably it’s still good to ship portions of a prototype in pieces, but the shape and size of the pull requests likely differs. If you’re at one of these teams and still stacking, I’d love to hear how you go about it!

Thank you to the B12 product team members who have introduced and iterated on this pattern.