← All posts

Testing a UI that doesn't exist until runtime

A metadata-driven UI has no hand-written ids, no stable labels and no fixed layout. Here is what we had to build into the framework to make it testable — and the four layers of tests that keep it honest: unit tests against real engines, docs-driven e2e, clean-VM journeys and a 7,000-user stress test.

Updated on 13 October 2026: two more layers (unit tests against four real database engines, and a stress test that reached 7,000 simultaneous users), the docs-driven suite running on clean virtual machines, what the journeys learned since September, and an honest list of what none of it covers yet.

Most end-to-end test advice assumes something that isn't true for us: that somebody wrote the page. Pick a data-testid, query a button by its label, assert the third row of the table. That works when a developer typed the markup and can type an attribute next to it.

In WUIC nobody typed the page. A screen is generated at runtime from metadata rows: which columns exist, which widget renders each one, what's editable, what's hidden, in what order. The labels come from a translations table and change with the user's language. The column order is a number in a database. There is no hand-written id anywhere, because there is no hand-written markup.

So the first question of the test strategy wasn't "which framework do we use" — it was what does a generated UI owe to the person testing it.

The contract had to be part of the product

The answer we landed on is that the components expose a small, deliberate contract, and that contract is a feature of the framework like any other — documented, versioned, and not allowed to change by accident.

Every field rendered in a form sits inside a wuic-field-editor that writes onto its own host element: the logical column name, the resolved widget type, whether it's editable, whether it's hidden in edit, whether a record is bound, and the current value. A test anchors on the name of the field, which is a metadata fact, instead of on a position or a label, which are presentation facts:

const host = document.querySelector('wuic-field-editor[data-field-name="StateProvinceID"]');
const value = host.getAttribute('data-field-value');

One detail in there took a while to get right. Those attributes are written with an imperative setAttribute inside a sync method called from ngOnChanges and ngDoCheck — not with @HostBinding. In zoneless Angular a binding getter is re-evaluated when change detection touches that component, and the record arrives through an async subject: the test's first read kept landing before the binding refreshed. The imperative write is uglier and it is correct.

"Loaded" is two different questions

The second thing a generated UI owes you is an honest answer to "is it ready".

We had one attribute, data-has-record, and it meant "the record object has been assigned". Tests treated it as "the values are here". They are not the same moment: the form builds its skeleton as soon as the object exists, and fills it when the data call comes back. On a laptop the gap is invisible. On a real installation — eighteen metadata calls in flight, the slowest at 2.8 seconds — the gap is more than a second, and the test reads an empty string from a field that is full an instant later.

The fix was not a longer sleep. It was a second, distinct signal on the dialog that says the data response arrived and was applied, so a test can wait for the thing it actually cares about:

Signal Meaning
wuic-field-editor[data-has-record="true"] skeleton ready, values not guaranteed
wuic-parametric-dialog[data-record-loaded="true"] the data response arrived and was applied

There's a residue even so: lazy widgets — the code editor, the lookup — propagate their value one tick after the dialog. So the wait allows a short window after "record loaded" for the value to appear, and then proceeds anyway, because a field can legitimately be empty. Testing an async UI is largely the work of naming the moments precisely enough to wait for the right one.

The tools have to exist on the build customers run

This is the one that cost us the most, and it's the one I'd tell anyone else first.

Some tests need to look inside a component: what chart type did it bind, how many datasets did it receive, what does the menu model actually contain. The obvious tool is Angular's window.ng.getComponent(). It works beautifully in development.

It does not exist in a production build. enableProdMode strips the debug tooling.

We found out by running the docs-driven suite against a real installation — the artifact a customer downloads, built the way a customer builds it — and watching dozens of tests fail with "runtime introspection failed". Not one of them had found a defect. They had all found the absence of a tool. Worse, for a while those failures looked like product regressions, and they cost triage time that belonged elsewhere.

The fix is a handle the framework attaches itself:

const host = document.querySelector('wuic-chart-list');
const cmp = host.__wuicCmp || window.ng?.getComponent?.(host);

With one rule that matters: it is not always attached. Exposing a component instance makes it reachable from any script on the page, and on a customer's installation that serves nobody. So it attaches in development, where window.ng exists anyway, and in production only if the page declared a flag before the application booted — which the test bootstrap does, and the product never does. Flag off, the cost is a boolean check per instance.

That's the general shape of the lesson: if your tests need a capability, ship the capability deliberately, gated, and documented — don't borrow one that evaporates in the build that matters.

Four layers, four different questions

We run four kinds of test, and each one answers a question the others can't:

Layer The question it answers Runs against
Unit tests does this piece of code generate the right SQL, and does that SQL run on each engine the code, plus real local instances of the four engines
Docs-driven e2e does the feature behave as its documentation page says a running backend, from a dev machine or from a clean VM install
Journeys can someone with only the docs go from an empty machine to a working app a clean virtual machine and the published package
Stress what happens when thousands of people use it at once a clean VM install, driven by virtual users over the API

The first two existed when this post was first written; the last two grew up alongside them. They are described here roughly in the order a defect meets them.

Docs-driven e2e asks does the feature behave as documented. Every documented feature page has tests derived from what the page promises. The catalog is not a folder of spec files: it's rows in a database — the test file, the route it runs against, and a metadata patch payload. A dispatcher applies the patch (this chart is a bar chart, this column uses the code editor widget, this kanban column has a hard WIP limit), runs the test against a real backend, then restores the metadata baseline. The patch is the point: the same test file exercises a different configuration of the same generated screen, which is what a metadata-driven product actually has to get right.

Journeys ask a different question: can a person who has only the documentation get from nothing to a working application. A journey takes a virtual machine with nothing on it, installs the published package the way the tutorial says to, and then uses the product the way the docs tell the reader to use it. No fixtures, no dev database, no warm node_modules, no shortcuts that only the author of the feature knows.

What a journey actually does

A journey is a sequence of modules, and the numbering is the plot: install, first-run wizard, getting started, apply the license, the business features one archetype at a time, the dashboard designer, a workflow, a report, the RAG chat, admin operations, teardown. Each module is a chapter of the tutorial, executed for real against a real browser on a real installation.

There is a second kind of reader, and it gets its own journeys: the developer who installs from sources and works in VS Code. Those journeys install the source kit, open the workspace, drive VS Code itself (debugger, Swagger page, the WUIC Assistant extension, the MCP clients), run the first-run wizard against a database of their own, and then build the same feature five times — once for each of the five ways the docs say framework and custom code can be combined, from fully generated to fully hand-written. A pattern that only works when you follow the author's mental model, and not the page, fails here.

Two design decisions make it worth the minutes it costs.

Each module names the documentation pages it is enacting. A module declares docs: ['list-grid', 'parametric-dialog', 'kanban-list', …], loads those pages, and checks the strings they quote against what's actually on screen. If the page says the menu path is Samples > Cities, the journey opens the menu and looks for it. If the doc quotes a button label that no longer exists, that is a finding — against the documentation, not against the code. Docs rot silently; this is the only mechanism we have that makes them fail loudly.

A finding is not just pass or fail. The evidence collector records four classes:

Class What it means
broken an assertion failed, a console or page error, a 4xx/5xx
slow the step took longer than the budget declared for it
unintuitive friction: the natural selector didn't match and a fallback did
doc-error the documentation quotes something that isn't on screen

That third one is the one I didn't expect to value so much. Every module declares the selector the documentation suggests first, and a fallback. When the fallback is what matches, nothing is broken — the feature works — but the path a reader would take didn't, and that gets recorded. It's the closest thing to an automated usability complaint I know how to write.

Each step is also timed against a declared budget (renderMs, interactionMs), so "it still works, it's just three times slower than it was" shows up as a finding instead of as a feeling.

The discipline around the run matters as much as the run. A journey that dies at minute forty is not restarted from scratch: the logs come off the guest first, and it resumes from the broken step, because the thirty-five minutes of installation before it were sane and the state they produced is the thing under test. Only a brand-new run is allowed to roll the VM back to its snapshot — restoring a snapshot on top of a run in progress destroys exactly the evidence the run existed to collect. And every deviation from the published artifact (running the installer from the repo instead of the site, patching a step script) has to declare itself as friction in the report, so nobody reads a green journey and believes the published package was the thing that passed.

How much of it there is

The docs-driven catalog is currently 315 tests over 82 documentation pages (it was 266 in September) — roughly one family per documented feature, sized by how much surface that feature has. The heavier families are the ones with the most configuration to get wrong: application settings (19), the grid (15), multi-tenant (14), metadata integration (12), the RAG chatbot (12), and then the pivot builder, the workflow designer, authentication and authorisations and import at 11 each, the dashboard designer (9), charts, timelines, licensing and OData (8 each).

By category, that catalog covers:

  • the data archetypes — grid, kanban, scheduler, chart, map, tree, carousel, timeline, spreadsheet, pivot, data repeater;
  • the edit form and its widgets — the parametric dialog plus one family per field widget (lookup, upload, code editor, HTML area, dictionary, number, many-to-many, button, and the rest), each with its own configurations;
  • the designers — dashboard, report, workflow, scene 3D, view builder;
  • the platform behaviours that cut across every screen — authentication and authorisations, multi-tenant, optimistic concurrency, logic delete, conditional styling, validations, custom actions, callbacks and events, translations (interface and data), themes, responsive mobile layout, accessibility;
  • the server side you reach through the UI — OData, Swagger, webhooks, scheduling, mailing, notifications, import/export, reporting, audit, SQL retry policy, custom exception handling;
  • the paths a developer takes — initial scaffolding, the first-run wizard, the five framework/custom composition patterns, licensing, the performance inspector, the RAG chatbot and its tools.

The journeys are four packages: the packaged install on Windows with IIS and on Linux with nginx (a chain of twelve modules, plus an optional exploratory one), and the sources-and-VS-Code kit on both (fourteen modules). Together they enact 31 documentation pages end to end, from install and first-run wizard through getting started, licensing, every business archetype, the dashboard designer, a workflow, a report, the RAG chat, Swagger, the five composition patterns and the admin operations, down to teardown. Four packages times four engines is sixteen cells, and a cell only counts as closed under a rule strict enough to get its own section further down.

Underneath all of it sit roughly 2,800 .NET test methods — about 5,400 cases once parameterised theories expand — and about 1,900 cases in 131 spec files for the Angular library. The stress test is a different animal: one scenario mix, run at 100, 500, 1,000 and up to 7,000 simultaneous users.

The suites deliberately overlap. A grid tested by the e2e suite in forty configurations is also opened once by a journey, on a machine where nobody has ever run npm install — and the second one fails for reasons the first cannot see.

Four engines, two operating systems

None of this means much on one database. The framework generates SQL, and the four engines it supports disagree about almost everything worth disagreeing about: identifier quoting, pagination, date handling, what comes back from an insert, how a BLOB gets loaded, whether a bulk read is even the same statement. A feature that works on SQL Server and has never been run on Oracle is a feature with an unknown.

So the matrix is four engines — SQL Server, MySQL/MariaDB, PostgreSQL, Oracle — times two operating systems, Windows and Linux, and every suite runs across it. The dispatcher carries its own per-engine configuration; the journeys carry the engine as a parameter of the run and install the product against it from scratch.

Since September the docs-driven suite also stopped running only against a developer's backend. A docs-driven cell is a clean VM where a journey has installed the locally built package and walked the first-run wizard, snapshotted at that point. The dispatcher, the browser and the test files stay on the development machine; the application, its engine and its data are the ones a customer would have, reached through a tunnel. A campaign is every engine once, split across two lanes in parallel — a Windows VM with IIS and a Linux VM with nginx — and from one campaign to the next each engine changes lane, so after two campaigns every engine has been through both operating systems. Multi-tenant tests get a round of their own, because a tenant-aware install has to be prepared differently (cloned tenant databases on three engines, cloned schemas on Oracle).

The cells are not interchangeable, and that is the point. A defect can live in exactly one of them: the upload path story further down is a Windows-package defect that no Linux run would ever have produced, because the Linux packages ship an absolute path. Report queries that looked engine-neutral turned out to be SQL Server dialect running against MySQL. A test can pass on three engines and fail on the fourth for a reason that has nothing to do with the feature under test, and only running the fourth tells you.

What each one caught

The e2e suite, once it could see, found real product defects. The journeys found the ones I'd never have found otherwise.

The best example is an image that wouldn't display in a grid. The e2e failure was blunt: the thumbnail element loaded but had zero width. On a developer machine it worked. On the installed package it didn't.

Underneath there were four copies of the same path calculation — where does an uploaded file live — that had drifted apart. One in the upload writer, one in the controller that serves the file back, one in the code that reads it into a BLOB column, and a fourth in the step that moves a file from its temporary folder to its final one once the record has an id. Three of them resolved a relative uploadFolder setting against the project's data folder. The fourth resolved it against the web root.

The Windows package ships "uploadFolder": "Upload" — relative, because you can't ship the build machine's absolute paths. So on that package, and only on that package, the fourth copy looked for the file in a tree where it had never been, found nothing, and skipped the move in silence. The image stayed in its temporary folder, the grid asked for it by record id, and got a 404.

The smoking gun was nine empty directories. That fourth copy created its target folder before checking, so the installation had a little museum of folders under the web root — uploadsample/2124, uploadsample/2125, one per record that had ever been saved — all empty, all in the wrong tree. Once you see those, the bug takes ten minutes.

The other one was less subtle and more embarrassing: the sample data that ships with the package points at image files on the machine that built it. Rows in the demo table referencing C:\Users\...\Dropbox\pictures\.... On any clean installation those files don't exist, so a screen about image previews cannot work, ever, and not because of a defect in the product. That isn't a bug in code; it's a bug in what we ship, and only a clean VM was ever going to say so.

The pattern behind both is the same, and it's the reason the journeys exist: a developer machine is a machine where every mistake about the environment has already been forgiven. Absolute paths that happen to resolve. Files that happen to be there from six months ago. A database that was migrated by hand once and never again. Services already running. The installer never ran, so nothing the installer does was ever tested. A clean VM forgives nothing, which is the entire point.

The categories a journey catches that an e2e suite structurally cannot:

  • What only exists on an installation. Packaged settings that differ from dev settings — like a relative uploadFolder — and the code paths that only that value reaches.
  • What the installer does. First-run scripts, database creation, the wizard, the license, the folder layout. A suite that starts from an already-installed app tests none of it.
  • What the documentation says. Menu paths, button labels, the order of steps. Those are only "true" if somebody follows them literally, and the journey is the only reader we have that never skips ahead.
  • What ships in the box. Sample data, templates, report definitions. They're content, they're not covered by unit tests, and they break quietly.
  • What got slower. With per-step budgets, a regression in startup or render time is a finding rather than something somebody eventually grumbles about.

One cause wearing five costumes

The single most useful hour of the whole campaign was realising that five unrelated-looking failures were one bug in the tests.

The symptoms: a chart's settings dialog never opened. A kanban card refused to move when dragged. A grid's row menu never appeared. A save seemed to never start, and timed out fifteen seconds later. A lookup showed no suggestions. Five features, five plausible product defects, five different-looking stack traces.

The cause: while the shell is loading it covers the page with an overlay, and every one of those interactions was a Playwright click with force: true. That option skips the actionability check — but it still delivers the event at the target's coordinates, which is where the overlay is. The click lands on the overlay, silently, and the test fails much later on the consequence.

Two things made it hard to see, and both are worth knowing:

  • The failure screenshot lied. The scenario runner takes its screenshot after running the restore callback in finally — and the restore saves for real. So the picture showed a closed form and a cheerful "updated" toast, fifteen seconds after the test had died on a click that never landed. The screenshot was of the cleanup, not of the failure.
  • A swallowed catch moves the failure somewhere innocent. The lookup helper waited for the suggestion panel with an 8-second timeout wrapped in .catch(() => {}), then clicked the option with a 30-second timeout. The error you get points at the option click — "no suggestions" — when the truth is the panel was never asked to open.

One shared helper now waits for the overlay to clear and verifies, via document.elementFromPoint, that the thing under the cursor is the thing we mean to click. Five tests went green. No product code changed, because there was nothing wrong with the product.

The honest version of that story is that it took three rounds of triage, and in two of them I was confidently wrong about the cause. The thing that finally worked wasn't cleverness, it was refusing to accept a diagnosis that hadn't been reproduced: open the page, click the button by hand, print what's actually under the pointer.

The layer underneath: unit tests against real engines

The unit tests are the cheapest layer and, for a product that writes SQL, the one most easily fooled. A provider test that asserts "the generated statement contains IS NOT NULL" proves the string, not the query. So the unit suite has two halves.

The first half is synthetic: build the metadata of a table in memory, ask each provider for the statement, assert the shape. It runs anywhere, in seconds, and it catches the regressions of a string builder — a missing quote, a parameter that became a literal, an operator that fell through to a default.

The second half runs what it generates. The database tests open a real connection and execute the statement against a real instance of each engine: SQL Server Express, MySQL 8.0 and PostgreSQL 16 running locally from a script that brings them up on demand, and Oracle Free inside the same Linux VM the docs-driven cells use, through an ssh tunnel. Two rules took a while to learn:

  • Never against a shared or source database. The databases that feed the release packages are not test fixtures. A test that writes there can poison the next build, and nobody will connect the two.
  • One engine per test process. The providers keep some static state, and when four engine factories share one process the tests collide. They pass on their own and fail together. Twenty-five of them still do when the whole suite runs in a single process, and the report says so in exactly those words rather than counting them as green or as product failures.

The pressure on this layer is what changed most since September, and the reason is the next section.

A feature on SQL Server is a promise on Oracle

WUIC was written on SQL Server first. Over the years every feature arrived there first, and most were ported to the other three engines; "most" is the dangerous word. In October we went through every feature that exists on SQL Server, looked for the ones that behaved differently — or not at all — on MySQL, PostgreSQL or Oracle, and closed them one by one, each with a unit test that runs on all four engines and a docs-driven test that runs on all four cells.

Some of the gaps were plain to see: filter operators the other providers didn't translate (not equal, between, is not null); pivot filters that only SQL Server understood; database triggers for notifications that only existed in T-SQL. On SQL Server a trigger can enqueue a notification directly; on the other three engines the trigger writes into an outbox table next to the data, and a background pump in the application delivers it. Same contract, two mechanisms, one test that checks the notification arrives.

The interesting ones were the silent ones, and they are the reason the docs-driven half of the work mattered as much as the unit half:

  • The scheduler on Oracle had never run a single job. It looked for its own tables under an owner called DBO — a SQL Server habit — found none, and concluded there was nothing to do. No error, no log line that looked like a failure. Scheduled jobs, mailings, workflow timers, the notification outbox: none of it ever fired on Oracle.
  • Restoring a record from its change history marked the history entry "restored" while the row stayed logically deleted on three engines, because their providers cleared a hard-coded column name instead of the table's own deletion flag.
  • Saving a workflow graph on the three non-SQL Server engines silently dropped its SLA timers: the graph was saved, the timers never existed.
  • When a table had more than one "last modified" column, the update wrote the right columns to the database but handed back a record with a single field literally named "fld": the loop variable's name in quotes instead of its value. That one was on SQL Server too.

And a few defects the campaign found on every engine at once, SQL Server included, because a new test looked where nobody had looked before: a PDF export that answered 500 when the report had no translation for the requested language; "Add table" from the metadata editor failing on every new table; a range filter in the grid that sent an empty filter list because the echo of its own descriptor reset the value the user had just typed.

Around twenty product defects in all, each fixed with a test that reproduces it. The final docs-driven round ran the regression set plus nine new parity tests on each engine, and all four came back green. Two tests didn't run at all on three of the engines: they need a table with at least 150,000 rows to observe an export in the background, and those engines' first-run data has a thousand per table. They are reported as not verified, not as passed.

That last distinction turned into a rule for every suite: a check that didn't run is not a check that passed. A skipped step, an assertion behind an if that was false, a "could not determine" — all of them count as open.

The machine under the test

The clean VMs taught us that "clean" is a moving target.

A snapshot freezes the guest's clock for its maintenance jobs. Restore it a month later and Windows decides that everything is overdue: the scheduled defragmentation, the compatibility appraiser, the .NET image optimiser, the updaters of two browsers and — the big one — Defender's "aggressive catch-up" scan, which kicks in when the last scan is more than thirty days old, regardless of the setting that supposedly disables catch-up scans. They all start in the first ten minutes after boot, which is exactly when a journey is installing the product and a stress test is ramping up. On one guest that window made the backend a hundred times slower for six minutes; on an Oracle journey it failed two modules with a wizard connection test over sixty seconds, and the same steps took a fraction of a second once the maintenance was over.

The answer was not a longer timeout. Defender is now removed from the test VMs (new snapshots, the old ones kept), and a single hygiene script — the same one for journeys, docs-driven cells and stress cells — switches off Windows Update, automatic maintenance, scheduled scans and the idle-time tasks after every boot, and on Linux masks the apt timers, snap refreshes and the other periodic units for the duration of the run.

The network between the test and the guest is part of the measurement too. Windows' built-in port proxy, which we used to expose the guest's dev servers, refuses new connections beyond six to ten at a time; behind NAT that became a thirty-second wait followed by ERR_CONNECTION_RESET on chunks of the dev server — a failure in the test bench that looked exactly like a failure in the product. A forty-line TCP forwarder in Node with a proper backlog, and a log line per connection, replaced it. For the stress test even a good forwarder is too much: every proxy between the load generator and the guest adds 40–60 ms to every request, so stress cells are measured only over a bridged network.

And then the rule for calling a journey cell closed, which exists because it was broken once. A source-kit journey was reported as "fourteen of fourteen modules, zero broken" while one of its verifications — the idempotency check of the first composition pattern — had never executed. Since then a cell is closed only when the teardown module ends green, nothing is classified as broken, and nothing was skipped: the report counts verifications that didn't run, and a non-zero count keeps the cell open.

Stress: how many people at once

The other three layers ask whether something works. The stress test asks how it behaves when a lot of people want it at the same time, which is a different kind of question with a different kind of answer.

The virtual users don't drive a browser — at thousands of users the browsers would be the thing under test. Each one logs in with its own account and uses the API the way the SPA does: 80% are readers (open a list, filter, page, sort, open a record, look at a dashboard, print a report) and 20% do a bit of everything, including inserts, updates, deletes and exports. One user in twenty, at a random moment, downloads an Excel export, "edits it" for a minute or two and imports it back. Routes are the real menu of the tutorial application minus the administration area, think times are human, the seed is fixed so that two runs with the same users make the same moves at the same moments, and logins ramp over about four minutes at every level so the report doesn't mix a half-empty system with a full one.

It started on the public demo server with twenty users, and the first finding wasn't about concurrency at all: exports were the bottleneck. A 230,000-row export took 311 seconds with seven to nine running at once, and the web process, not SQL Server, was the one sweating — SQL Server spent almost all its time waiting for the application to consume rows. A benchmark without a database took the same export from 28.5 seconds and 12.2 GB allocated to 7.4 seconds and 2.3 GB, with byte-identical output (the per-cell hook used to scan every loaded assembly, for every cell). On top of that, exports now go through a queue: at most half the cores run at once, the rest wait in order, and the user's dialog and notification bell show the position in the queue. The probe also found that two exports of the same table started in the same second wrote the same file, and every one but the first answered 500.

The import round trip — export a table, change it, import it back — found more than the export did: date cells read as numbers, lookups whose description wasn't unique, a primary key that was also a lookup, two columns with the same caption. All fixed on the four engines with shared rules and their own tests.

Then the dedicated Linux cell, MySQL, 16 vCPU and 32 GB, bridged network:

Users Operations/s Hard errors p50 / p95 interactive (ms)
1,000 155 0.01% 6 / 36
2,000 310 0 7 / 51
3,000 465 0 9 / 75
4,000 619 0 13 / 123
5,000 776 0 11 / 64
6,000 931 0 12 / 74
7,000 1,086 0 15 / 85

Linear — about 155 operations per second for every thousand users — but that table hides the two weeks that made it possible, and its rows are not all the same build: up to 5,000 users it is release 1.7.23, while 6,000 and 7,000 include the fixes described below.

With the MySQL driver the product used before, the process collapsed at around 560 users and fell over at a thousand; the release that switched to MySqlConnector is the reason the first row exists. At 4,000 users the limit was not the server, which sat at 40% CPU, but the load generator: the shared API client never disposed of its responses, so every virtual user kept the body of every reply it had ever received — about 97 KB per operation, growing for the whole run — until the generating machine was down to its last few hundred megabytes. Fixed, a runner for 500 users went from 3.8 GB to under half a gigabyte.

At 6,000 users the first ninety seconds were a stall: 1,500 logins timed out, then a burst of "all pooled connections are in use". A heap dump taken with the ramp in progress told the story: the .NET thread pool was starved. A synchronous database open in the authentication of the notification WebSocket, a system-info cache without single-flight, so every caller that missed it went to the database at the same moment, and about 800 pool connections held by the initial notification snapshots, waiting for a continuation that no free thread could run. Two fixes in the product — single-flight on the cache, a cap of 32 concurrent initial snapshots — plus a minimum thread count of 256 in the runtime configuration of the test install (a deployment setting, not yet a default of the package), and 6,000 and 7,000 passed with zero errors, with the process's memory going down during the stable window instead of climbing.

At 7,000 users the guest is at 81% CPU on average, so the next limit is the server's CPU, probably between 8,000 and 9,000 users. We haven't measured it: past 6,000 users even two machines generating load are close to their own limit, and a stress test that measures its generator measures nothing.

What none of this covers yet

A testing post that ends with everything green is a sales brochure. Here is what isn't covered, by layer.

  • Unit tests. The static state shared between providers means the full suite can't run all four engines in one process yet; the collisions are counted and reported, not fixed. Some database tests still point at the source databases and are excluded from routine runs. A few code paths — for example the role-based record restriction on PostgreSQL — are tested only against generated SQL, not against a real database.
  • Docs-driven. The RAG chatbot family is excluded from the VM campaigns, because its answers depend on a language model rather than on the product. Scheduled mailings can't be checked on cells without a mail server. The two background-export tests need a large table that only the SQL Server cell has. And some checks are weaker than they look: the report-designer test proves that a scaffolded report renders data by comparing the page count of a PDF filtered on one state with one filtered on a single city, which shows that rows arrive and filters apply, but it doesn't read the text in the PDF, and two of the three scaffolded layouts are generated but never rendered.
  • Journeys. The sixteen cells were last measured on different package versions, and not all of them are closed on today's modules: the source-kit journeys grew from eleven to fourteen modules in October, and of their eight cells two are still green only on the old eleven and one has never completed a full run. The exploratory module runs separately and isn't part of the chain; and every deviation from the published package (a DLL from sources, a script from the repository) is declared in the report precisely because it means the published artifact is not what passed.
  • Stress. At scale, only MySQL on Linux has been measured. The Windows Server cells — SQL Server and MySQL behind IIS — are prepared and verified with a control run, but the real ramp hasn't run. Oracle Free is capped by its edition at two CPUs and is already at that cap with a hundred users, so it says something about Oracle Free and little about the product. And the ceiling above 7,000 users is an estimate.

Where this leaves us

A red test on a generated UI is not evidence of a broken product. It's evidence that something in the chain between a metadata row and a pixel didn't behave — and that chain has a lot of links: the metadata, the code that reads it, the component that renders it, the environment it was installed into, the data that shipped with it, the documentation that described it, the virtual machine it ran on, the machine generating the load, and the test itself. Every campaign we run produces failures in all of those categories, and the ones that are genuine product defects are a minority.

Deciding which link is the whole job. The only way to make that decision cheap is to build the product so it can be asked — which is what the attributes, the readiness signals and the inspection handle are for — and to keep the diagnosis honest: reproduce before you conclude, read the error rather than the screenshot, and never let a swallowed exception pick the suspect for you.

The hooks are documented, including the ones this post used: the testability and e2e hooks page lists the DOM attributes, the readiness signals, the introspection handle and its flag, and the selectors that will betray you — generated Angular ids, translated labels, positions in a table.

If you're building something that renders itself from data, the summary is short: decide what your UI tells the outside world about itself, make that a contract, and make sure the contract survives the production build. Everything else is downstream of that.