Automated tests for AI SaaS expose salvageable code

Automated tests can tell you whether an AI-built SaaS is worth salvaging, but only if the tests measure the business invariants that make the product worth operating. A hundred green component tests do not answer whether a customer can pay once, see only their own records, survive a retry, and receive the same application after a clean deployment.

I have seen teams treat a generated test suite as a vote of confidence in generated code. That reverses the burden of proof. The existing code does not get credit for being easy to test. It earns remediation money by producing a small, repeatable body of evidence around revenue, authority, data, and release behavior.

That evidence should be small enough for a founder to understand and strict enough for an engineer to distrust. It should also fail for useful reasons. If every failed test collapses into an unexplained timeout, you have learned little about the system. If a test says that a second webhook created a second paid order, or that a member read another tenant's invoice, it has exposed both a defect and the boundary of the repair.

The aim is not to prove that the SaaS has no bugs. No practical test suite proves that. The aim is to decide whether the product's valuable behavior can be isolated, specified, repaired, and deployed without paying to rediscover the application one screen at a time.

A green suite is evidence only when it protects an invariant

A passing test matters when you can name the promise it protects, the failure it would catch, and the artifact it leaves for review. Without those three parts, a green result is often ceremony.

AI app builders tend to generate tests around visible behavior because visible behavior is easy to describe: a page loads, a button responds, a modal closes. Those tests can catch regressions, but they say almost nothing about whether the code has a coherent domain model. A checkout page can display a success message while the webhook handler creates duplicate subscriptions. An admin button can disappear while its API endpoint accepts a member's request. A form can show the new address while the database saved only half of it.

Use a four-part evidence set. Each part answers a different salvage question:

• Revenue flows: can money-related state move forward exactly once, including retries and failures?
• Permissions: does the server enforce role and tenant boundaries for both allowed and forbidden actions?
• Data integrity: do writes remain valid under partial failure, concurrency, and repeated requests?
• Deployment repeatability: can an empty environment become a working release through one documented path?

The distinction between a regression suite and a salvage suite matters. A regression suite protects known behavior after a team trusts the architecture. A salvage suite discovers whether there is enough stable behavior to justify repairing an architecture nobody yet trusts. Reusing existing tests is fine, but inherited assertions should receive no special status. Trace each one to an invariant or remove it from the decision set.

OWASP's Application Security Verification Standard makes a useful separation here. It has distinct areas for authentication, access control, business logic, data protection, APIs, and configuration. That structure is better than a generic claim that the app is "secure," but applying every ASVS requirement before a salvage decision would bury the decision in work. Use the relevant requirements to sharpen boundary tests, then keep the approval set tied to the product's actual risks.

A useful test name reads like a policy. second_paid_webhook_does_not_create_another_entitlement says more than payment_test_03. When a non-technical owner reads the report, the name should tell them what survived and what did not.

Define the decision before repairing the code

Write the salvage threshold before anyone fixes the first failing test. Otherwise every repair changes the standard, and sunk cost starts making the decision.

The threshold needs outcomes, evidence, and stop conditions. It does not need a forecast built from guessed engineering points. For each evidence area, record one critical journey, the state that must remain true, the observable proof, and the result that would stop remediation. Keep that document beside the tests so reviewers can distinguish a business failure from a test harness failure.

A compact file can carry the contract:

salvage_gate:
  revenue:
    journey: paid checkout plus duplicate webhook
    invariant: one charge maps to one entitlement
    proof: order row, entitlement row, provider event id
    stop_if: duplicate processing cannot be made atomic
  permissions:
    journey: member requests another tenant's invoice
    invariant: server denies the request without revealing existence
    proof: 404 response and unchanged audit record
    stop_if: tenant ownership is absent from the data model
  data_integrity:
    journey: two workers claim the same queued job
    invariant: one worker owns the job
    proof: one claim token and one completed result
    stop_if: authoritative state exists only in browser storage
  deployment:
    journey: empty environment to smoke-tested release
    invariant: migrations and configuration are deterministic
    proof: commit id, migration log, smoke-test report
    stop_if: production requires an undocumented manual edit

The exact syntax does not matter. The specificity does. "Payments work" gives a repair team room to demonstrate the easiest path. "A duplicate paid webhook leaves one entitlement" forces the test through a common failure boundary and produces state that a reviewer can inspect.

Choose stop conditions that expose missing foundations, not ordinary defects. A wrong conditional can be repaired. Tenant ownership absent from every table can turn a small remediation into a data-model rebuild. A migration that fails because of one bad default is a bounded task. A production database whose schema differs from every migration file means the repository is not the source of truth.

Set a time box for gathering evidence, but do not turn elapsed time into the verdict. A slow test environment may say more about deployment repeatability than code quality. Record blocked tests separately from passed and failed tests. A blocked revenue test because nobody knows the webhook secret is deployment evidence, not a neutral omission.

The approval result should be one of three states: remediate, rebuild a bounded subsystem, or stop and re-scope the product. Avoid a single percentage score. Scores hide vetoes. A product can pass nine minor checks and still be unsafe to operate because tenant isolation fails.

Revenue tests must follow state, not the checkout screen

A revenue test should prove the complete commercial state transition, including the awkward paths after the browser reports success. The browser is one participant in that transaction, and usually not the authoritative one.

Start with the smallest journey that creates or preserves revenue: a customer begins checkout, the payment provider confirms an event, the application records the purchase, and the customer receives the correct entitlement. Then run the same provider event again. The second delivery must produce the same business state without a second order, credit, email-trigger record, or entitlement.

Do not call the test finished when it receives a success status. Capture evidence from each authoritative boundary:

• The provider event identifier is stored under a uniqueness rule.
• The order and entitlement refer to the same customer and tenant.
• A retry returns a stable result rather than replaying side effects.
• A failed downstream action can resume without repeating the charge-side transition.
• A refund or cancellation removes only the access that the product's policy says it removes.

This is where generated applications often reveal split authority. The browser writes a paid=true flag, a serverless function writes an order, and a webhook writes a subscription. Each path looks reasonable in isolation. Together they allow contradictory states. Automated tests expose the split only when they assert across all three stores and identify which record owns the truth.

Walk through one failure in detail. Send event evt_test_42 and force the entitlement insert to fail after the order insert. Retry the same event without the fault. The acceptable outcome is one order and one entitlement associated with evt_test_42. Two orders reveal missing idempotency. One order with no entitlement reveals a transaction or recovery gap. A permanently rejected retry reveals that the code marks an event complete too early.

The test report should show business-shaped output rather than a wall of assertions:

REVENUE_GATE duplicate_webhook
provider_event=evt_test_42
orders=1 entitlements=1 notifications=1
first_attempt=rolled_back retry=completed
result=PASS

Do not use a real payment card or depend on a production account for salvage testing. Use the provider's test mode or a controlled adapter, but keep the application's real persistence and idempotency logic in the path. Mocking the entire payment boundary proves that the mock agrees with itself.

A popular recommendation is to begin with end-to-end tests for every pricing plan. That feels comprehensive and produces impressive reports. It is the wrong first investment when the revenue state model may be broken. Prove one representative plan through retries, cancellation, and fault recovery. Add breadth after the state transition has one owner.

Permission tests need two tenants and several deliberate denials

Permission tests should prove that the server rejects forbidden access even when a caller bypasses the interface. Hiding a button is presentation logic, not authorization.

Create two tenants with stable fixtures. Give the first tenant an owner and a member; give the second tenant a record with an identifier the first tenant can guess or obtain. Exercise direct requests against reads and writes. The first tenant's member must not read, update, delete, export, or attach data to the second tenant's record.

Test allowed actions too. A suite containing only denials can pass because every endpoint is broken. Each protected operation needs a paired assertion: the correct role succeeds on its own object, while the wrong role or tenant fails on the same route. Check the database after both requests. A server that returns 403 after performing the write is still compromised.

Decide whether the application uses 403 or 404 for cross-tenant objects and apply that rule consistently. Returning 404 often avoids confirming that another tenant's record exists. The status alone is not enough. Compare response shape, timing within a sensible tolerance, and side effects so an error handler does not leak a title, owner name, or storage path.

OWASP ASVS treats access control and business logic as separate verification areas, and that is a useful warning. A user may have permission to issue a refund but still violate a business rule by refunding the same payment twice or approving their own request. Role checks answer "may this actor call the operation?" Business invariants answer "may this valid operation change this object now?" A salvage suite needs both.

Generated code often scatters role strings across route handlers, UI conditions, and database queries. Do not immediately refactor those strings into a polished policy layer. First use the tests to map where enforcement actually occurs. If one server-side boundary can be made authoritative, remediation may be bounded. If every query relies on a client-supplied tenantId, estimate a deeper repair.

Treat service roles and background jobs as actors in the same matrix. A queue worker with unrestricted database credentials can erase the guarantees tested through the public API. Give a background operation a fixture from each tenant and verify that its selection query and update both preserve ownership.

One failed denial is a veto for production approval, but it is not automatically a veto for salvage. The diagnostic question is whether the data model contains the ownership facts needed to enforce the denial. If every invoice has a trustworthy tenant reference, the repair has a foundation. If ownership must be inferred from mutable email addresses, the test has found a rebuild boundary.

Integrity appears when requests repeat or collide

Data-integrity tests should force the application into states that ordinary happy-path tests politely avoid. Retries, concurrent writes, process termination, and partial downstream failures reveal whether the database protects the product's rules or merely stores whatever the application sends.

Begin by writing each important invariant as a database fact. An invitation token can be consumed once. An email belongs to at most one active account within the chosen scope. A job has one current owner. An invoice total equals the persisted line items under the product's rounding rule. Then find the lowest layer that can enforce each fact. Prefer a uniqueness constraint, foreign key, check, or transaction where the database can express the rule.

Application checks such as "query first, insert if absent" are vulnerable when two requests run together. A deterministic concurrency test makes the race visible: pause both requests after they observe absence, release them together, and inspect the final rows. If the database constraint rejects one insert and the application converts that rejection into the expected response, the invariant holds. If two rows appear, repeated test runs are not a substitute for a constraint.

PostgreSQL's documentation explains that transaction isolation controls which concurrent changes a transaction can observe. Teams often cite isolation levels as if selecting a stronger label solves every race. It does not. The application still needs a correct transaction boundary, constraints, and a deliberate response to serialization or uniqueness failures. A generated handler that starts a transaction after its first write has already lost the useful boundary.

Test recovery as state, not merely as an error code. Kill a worker after it claims a job but before it records completion. Advance the lease or recovery clock through an injected clock, start another worker, and verify that one final result exists. This distinguishes at-least-once delivery, which permits repeated attempts, from duplicate business effects, which the application must prevent.

Do not assert entire database snapshots for every case. Snapshot tests make harmless fields look important and hide the invariant among noise. Query the records that own the decision and print a compact before-and-after diff. Keep timestamps, random identifiers, and generated ordering deterministic where the behavior allows it.

If the tests require global cleanup scripts that delete tables between cases, inspect that dependency. It may conceal missing tenant filters and order dependence. A stronger fixture creates isolated tenant identifiers, runs safely beside another fixture, and removes only its own records. Parallel test failures are often architecture evidence wearing a test-runner costume.

Integrity failures define repair size. Missing constraints around an otherwise coherent schema are often salvageable. Several competing identifiers for the same user, unversioned serialized blobs, or business state held only in browser storage point toward a bounded rebuild. Record that boundary instead of writing ever more test setup to imitate coherence.

A repeatable deployment starts with nothing

Deployment repeatability means a documented command can turn a clean environment and declared configuration into the same tested application. Redeploying the already-working production machine does not prove it.

Use an empty database and a newly created runtime environment. Pin the source revision and dependency lockfile. Supply configuration through the documented mechanism, run every migration in repository order, build the application, start it, and execute one smoke journey from each evidence area. Save the migration log, build output, release identifier, and smoke-test results together.

The first run matters, but the second run catches hidden non-repeatability. Destroy the environment and repeat the process from the same revision. Compare schema state, seeded reference data, generated assets, and observable configuration. A release that succeeds only after someone opens a console and edits a row has failed even if the application eventually responds.

Do not copy production secrets into the test. The evidence you need is that every required secret has a declared name, a responsible source, and a failure mode when absent. A startup check should identify missing configuration before serving traffic. A secret embedded in source code is both a security defect and evidence that environments cannot be recreated safely.

GitHub Actions documentation says environment protection rules can gate a job and withhold environment secrets until the rules pass. That is a useful release control, but it cannot repair an unreproducible build. Apply approval after the pipeline has created inspectable evidence. A manual approval placed before opaque scripts only formalizes guessing.

Playwright's documentation recommends recording traces on the first retry of failed tests in continuous integration. I agree with the intent because a trace can preserve screenshots, DOM snapshots, and network activity from an intermittent browser failure. For a salvage gate, also retain artifacts from the first failure, and never let a retry replace the original result. A test that fails once and passes on retry is flaky evidence, not clean evidence.

Make rollback part of the release test when migrations can change customer data. You may discover that rollback means deploying compatible application code forward rather than reversing a destructive migration. That is acceptable if the runbook states it and the test proves it. An imaginary down migration is worse than an honest forward-recovery plan.

Deployment failures often settle the salvage decision faster than style complaints. If the repository contains its schema, build inputs, and configuration contract, engineers can repair toward a known release. If the only working system depends on console edits, untracked functions, and one person's local environment, price discovery before remediation. The code archive alone is not the product.

Failure shape tells you whether repair has a boundary

The pattern of failures matters more than the raw count. Ten failures caused by one missing authorization boundary may be cheaper and safer to repair than two failures caused by contradictory sources of truth.

Classify each failure by ownership. A localized failure has one responsible component and one authoritative record. A cross-cutting failure passes through several layers but still has a clear owner. A foundation failure has no trustworthy owner, such as subscription status independently controlled by browser state, a webhook table, and an admin override with no precedence rule.

Then inspect determinism. A deterministic failure is an asset because engineers can reproduce it and prove the repair. An intermittent failure needs artifacts that reveal scheduling, state, inputs, and environment. If the team can make an intermittent race deterministic with barriers or an injected clock, the system becomes more salvageable even before the production code changes.

Do not reward easy fixes during the assessment. It is tempting to repair syntax errors, update dependencies, and adjust selectors because the passing count rises quickly. Those changes can wait unless they unblock a critical journey. The decision needs information about the expensive uncertainty first.

Use a short remediation ledger with four fields: failed invariant, suspected authority, smallest repair boundary, and proof required after repair. Avoid estimates until the suspected authority is credible. "Fix billing service" is not a boundary when billing truth lives in four tables and two browser stores. "Make processed provider event IDs unique and commit the entitlement in the same transaction" is testable enough to estimate.

A test harness can also fail the assessment. Warning signs include fixtures that call production, assertions that depend on current clock time, retries that erase failures, shared accounts across cases, and mocks that duplicate the implementation. Repair the harness only far enough to obtain trustworthy evidence. A pristine testing framework around an incoherent product is still sunk cost.

The decision becomes favorable when failures cluster around a few boundaries, the data model contains the facts needed for enforcement, and clean deployments reproduce the same results. It becomes unfavorable when each test requires a new interpretation of business state. That pattern means remediation would include rediscovering the product specification while changing it.

Approval needs vetoes, owners, and a price for uncertainty

Approve remediation when every critical evidence area either passes or has a bounded failure with an identified owner, a repair plan, and a test that will prove the change. Do not approve it from an aggregate pass rate.

Revenue duplication, cross-tenant access, unrecoverable data corruption, and an unrecreatable production environment deserve explicit veto status. A veto can lead to a subsystem rebuild rather than cancellation, but someone must name that scope before work begins. If the team cannot agree which state is authoritative, the scope is not ready for a fixed remediation commitment.

Ask for an evidence packet, not a presentation. It should contain the salvage-gate file, exact source revision, test command, fixture description, first-failure artifacts, database diffs for stateful checks, deployment logs, and the remediation ledger. Another engineer should be able to rerun it without receiving oral instructions.

Require the packet to record what the team could not test. Missing access to a payment sandbox, an unknown production migration, or an unavailable signing secret changes the confidence of the decision. An untested boundary does not count as a failure, but it cannot quietly count as a pass. Assign an owner and a resolution date to each gap, then decide whether the gap blocks approval or increases the discovery allowance.

Non-technical owners should review the invariant names and business outcomes, while engineers review isolation, fixtures, and artifacts. This division does not ask a founder to judge SQL or a developer to invent commercial policy. It exposes disagreements early. If cancellation should preserve access until period end, write that policy before a test encodes a different answer.

FixMyMess uses codebase diagnosis and expert verification alongside AI-assisted tooling for this kind of inherited application assessment. A free code audit can help identify whether the failures point to logic repair, security hardening, refactoring, deployment preparation, or a rebuild boundary, but the approval should still rest on the evidence packet.

Price uncertainty openly. A bounded authorization repair can carry a clear acceptance test. An unknown production schema, missing provider configuration, or absent ownership model needs a discovery allowance or a separate rebuild decision. Hiding uncertainty inside a confident estimate creates the worst salvage project: one that stays almost finished while every repaired path exposes another source of truth.

Automated tests earn the decision when they make refusal possible. If the evidence set cannot stop remediation after duplicate revenue effects, tenant leakage, corrupt retries, or a deployment that nobody can reproduce, it is a sales prop. Build the small gate, preserve its failures, and fund only the boundaries it makes visible.