I Tested Rails Baseline by Leaving Architecture Out of the Prompt

Aug 27, 2026

Rob

I did something a little different before shipping Rails Baseline 1.0.

Instead of spending another week adding starter features I thought somebody might need, I took the actual packaged ZIP, opened it as if I had just bought it, and handed it to fresh coding-agent sessions.

Then I gave them product prompts.

I did not tell them to use Pundit. I did not say to scope records through the current account, use public IDs, put paid access behind Entitlements, or use Solid Queue for background work. Those were all patterns already in the repository, and I wanted to see if the agent would find them without the prompt turning into an architecture checklist.

One test used Claude. The other used Codex.

Both went pretty well.

That sentence is intentionally boring. I was not trying to figure out which model could win a Rails benchmark. I wanted to test the codebase.

Why test the starter this way?

Rails Baseline is a normal Rails application, but part of the reason I built it was for coding agents.

I have spent years building SaaS apps and eventually wrote Build A SaaS App in Ruby on Rails 8. A lot of the hard-won stuff from doing that is not a particular controller action or gem install. It is knowing where certain decisions belong and trying not to make a different version of the same decision six months later.

Coding agents make this more obvious because they can produce so much code so quickly.

If the repository has a good pattern, the agent can usually find and continue it. If the repository is silent, the agent still has to build something. At that point it either asks, guesses, or follows whatever general Rails pattern seems reasonable from training.

A starter that claims to work well with agents should probably be tested by giving an agent incomplete product instructions and seeing what happens.

So that is what I did.

Test one: build a feedback board

The first test started from the RC1 release ZIP, outside the Rails Baseline source repository. I onboarded it as a new application and gave a fresh Claude session a prompt to build a small feedback board product.

The basic product was pretty normal SaaS stuff. Accounts could create boards, anonymous visitors could leave feedback, feedback had statuses, and plan limits changed how many boards an account could have. Paid accounts also got a CSV export.

I intentionally described those things as product requirements.

I did not say something like:

Use Current.account for tenancy.
Use Pundit for authorization.
Use Entitlements.enabled? for paid capabilities.
Use Limits.value for board counts.
Use HasPublicId for URLs.

That would prove I can write a detailed prompt. I already know I can do that.

The more interesting question was whether Claude would inspect the app and reach those conclusions itself.

It mostly did.

The board records were account-owned. Authorization followed the existing policy shape. The free and paid board counts went through the application’s plan-limit boundary instead of being scattered through controllers. The paid CSV feature used the entitlement layer. Public URLs followed the existing public-ID convention.

There was also a policy-scope mistake during the build. Claude caught it and fixed it during its own pass.

Good. That is much closer to what I actually want from an agent session than a first pass that looks impressive but quietly breaks account isolation.

The useful miss

The CSV export exposed something Rails Baseline had not really answered yet.

Where should a generated file live?

It would have been easy to react by adding Active Storage, S3 setup, an Export model, and some generic file-delivery service to the starter. That felt like exactly the kind of thing I did not want Baseline to become.

A lot of products do need durable generated files. Plenty do not. A CSV that can be generated quickly and downloaded immediately may not need to be stored at all.

So the answer became documentation instead of another core subsystem. I added a generated-artifacts recipe that talks through when synchronous generation is enough and when durable storage becomes a product decision.

That gave me something else to test later.

Test two: build an endpoint monitor

The second test started from RC2 and used a fresh Codex session.

This prompt was more demanding. Build an account-owned endpoint monitor that checks public HTTP/HTTPS URLs on a schedule. Free accounts get three monitors checked every 15 minutes. Paid accounts get 25 monitors checked every five minutes. Keep recent check history, support a paid CSV export, and optionally expose a public status page.

Again, the prompt described the product. I did not name the Rails Baseline implementation pieces I expected it to discover.

Codex found most of them.

It used account scoping and the existing Pundit context. It used persisted public IDs, Entitlements, Limits, tenant-aware jobs, and the Solid Queue recurring configuration. For the CSV export, Codex found the generated-artifacts recipe from the first dogfood and chose not to invent persistent storage for a file that could be generated on demand.

That last part made me pretty happy. The first test found a missing decision. I documented it, and a completely fresh agent used that documentation in the second test.

That is basically the behavior I wanted from Rails Baseline in the first place.

The second app found harder gaps

An endpoint monitor has one security problem that a feedback board does not have: the application has to make HTTP requests to URLs supplied by users.

Codex correctly treated that as an SSRF problem instead of just checking whether the string started with https://.

Its implementation resolved DNS, rejected private and other unsafe address ranges, pinned the connection to the validated address, used explicit timeouts, verified TLS, disabled environment proxies, and did not follow redirects.

Rails Baseline did not teach any of that.

I do not really consider that a failure of the starter. Safe outbound HTTP is one of those capabilities that only some products need, and there are enough security details that “just use Net::HTTP” is not much of a recipe.

Still, the test found a real boundary where an agent had to leave the existing precedent and design something new. That became one of the small recipes added before 1.0.

The scheduling side found another one.

Baseline already showed recurring Solid Queue work and tenant context inside jobs, but it did not show the shape needed when each account’s schedule depends on product state. Codex ended up with a minute-level dispatcher that finds monitors which are due and enqueues their actual check jobs.

That also became a short recipe instead of a new scheduling framework.

And, yes, the agent still wrote a bug

The endpoint monitor app ended up calling the model HttpMonitor because Monitor collides with a Ruby constant and EndpointMonitor collided with the onboarded application namespace.

The routes still used the product-facing name:

resources :monitors

At one point, a view used Rails’ polymorphic routing shortcut:

<%= link_to monitor.name, monitor %>

Rails saw an HttpMonitor instance and looked for http_monitor_path, which did not exist. The route helper was monitor_path.

That one made it through the agent’s test suite, and I hit it manually while clicking around the app.

The fix was easy, but I liked finding this too. A coding agent following the architecture well does not mean it stops making normal implementation mistakes. It also gave Baseline one more small Rails-specific recipe: if your model class and route resource intentionally use different names, either avoid polymorphic routing there or test a navigation path that exercises it.

No framework required.

What I actually learned from the tests

The useful result was not “Claude passed” or “Codex passed.”

Both agents were capable enough that the interesting differences came from what the repository gave them to work with.

When Rails Baseline had a clear local pattern, the agents usually found it. Account ownership, authorization, plan access, limits, jobs, IDs, and normal Rails/Hotwire UI conventions all carried through into new product code without me naming those pieces in the prompt.

When the app did not have a pattern, the agent had to make a product decision. Sometimes that was exactly right. Other times it exposed a missing piece of documentation that was general enough to add back to the starter.

That seems like a healthier way to build an agent-friendly codebase than trying to anticipate every future feature with another abstraction.

I would rather have 20 boring, well-used patterns and a handful of good recipes than 80 generic service objects waiting for a use case.

I will probably keep doing this

The best part of the process was how close it felt to an actual customer using the product.

I was not running tests inside the Baseline source repo and congratulating myself because they were green. The ZIP was packaged, copied somewhere else, onboarded into a new app, and then handed to an agent that had no history from building Rails Baseline.

That caught setup issues, product gaps, documentation gaps, and one wonderfully ordinary Rails routing bug.

For future Baseline releases, I think this kind of test is more valuable than adding another sample feature just so the feature list gets longer.

Give a fresh agent a product request. See what it copies. See where it guesses. Fix the parts that should have been obvious from the repository.

Then stop before the starter turns into the product it is supposed to help you build.

If you want to see the Rails app these tests were run against, Rails Baseline is at railsbaseline.com.