Most end-to-end test advice assumes something that isn't true for us: that somebody wrote the page. Pick a data-testid, query a button by its label, assert the third row of the table. That works when a developer typed the markup and can type an attribute next to it.
In WUIC nobody typed the page. A screen is generated at runtime from metadata rows: which columns exist, which widget renders each one, what's editable, what's hidden, in what order. The labels come from a translations table and change with the user's language. The column order is a number in a database. There is no hand-written id anywhere, because there is no hand-written markup.
So the first question of the test strategy wasn't "which framework do we use" — it was what does a generated UI owe to the person testing it.
The contract had to be part of the product
The answer we landed on is that the components expose a small, deliberate contract, and that contract is a feature of the framework like any other — documented, versioned, and not allowed to change by accident.
Every field rendered in a form sits inside a wuic-field-editor that writes onto its own host element: the logical column name, the resolved widget type, whether it's editable, whether it's hidden in edit, whether a record is bound, and the current value. A test anchors on the name of the field, which is a metadata fact, instead of on a position or a label, which are presentation facts:
const host = document.querySelector('wuic-field-editor[data-field-name="StateProvinceID"]');
const value = host.getAttribute('data-field-value');
One detail in there took a while to get right. Those attributes are written with an imperative setAttribute inside a sync method called from ngOnChanges and ngDoCheck — not with @HostBinding. In zoneless Angular a binding getter is re-evaluated when change detection touches that component, and the record arrives through an async subject: the test's first read kept landing before the binding refreshed. The imperative write is uglier and it is correct.
"Loaded" is two different questions
The second thing a generated UI owes you is an honest answer to "is it ready".
We had one attribute, data-has-record, and it meant "the record object has been assigned". Tests treated it as "the values are here". They are not the same moment: the form builds its skeleton as soon as the object exists, and fills it when the data call comes back. On a laptop the gap is invisible. On a real installation — eighteen metadata calls in flight, the slowest at 2.8 seconds — the gap is more than a second, and the test reads an empty string from a field that is full an instant later.
The fix was not a longer sleep. It was a second, distinct signal on the dialog that says the data response arrived and was applied, so a test can wait for the thing it actually cares about:
| Signal | Meaning |
|---|---|
wuic-field-editor[data-has-record="true"] |
skeleton ready, values not guaranteed |
wuic-parametric-dialog[data-record-loaded="true"] |
the data response arrived and was applied |
There's a residue even so: lazy widgets — the code editor, the lookup — propagate their value one tick after the dialog. So the wait allows a short window after "record loaded" for the value to appear, and then proceeds anyway, because a field can legitimately be empty. Testing an async UI is largely the work of naming the moments precisely enough to wait for the right one.
The tools have to exist on the build customers run
This is the one that cost us the most, and it's the one I'd tell anyone else first.
Some tests need to look inside a component: what chart type did it bind, how many datasets did it receive, what does the menu model actually contain. The obvious tool is Angular's window.ng.getComponent(). It works beautifully in development.
It does not exist in a production build. enableProdMode strips the debug tooling.
We found out by running the docs-driven suite against a real installation — the artifact a customer downloads, built the way a customer builds it — and watching dozens of tests fail with "runtime introspection failed". Not one of them had found a defect. They had all found the absence of a tool. Worse, for a while those failures looked like product regressions, and they cost triage time that belonged elsewhere.
The fix is a handle the framework attaches itself:
const host = document.querySelector('wuic-chart-list');
const cmp = host.__wuicCmp || window.ng?.getComponent?.(host);
With one rule that matters: it is not always attached. Exposing a component instance makes it reachable from any script on the page, and on a customer's installation that serves nobody. So it attaches in development, where window.ng exists anyway, and in production only if the page declared a flag before the application booted — which the test bootstrap does, and the product never does. Flag off, the cost is a boolean check per instance.
That's the general shape of the lesson: if your tests need a capability, ship the capability deliberately, gated, and documented — don't borrow one that evaporates in the build that matters.
Two suites, two different jobs
We run two things, and they answer different questions.
Docs-driven e2e asks does the feature behave as documented. Every documented feature page has tests derived from what the page promises. The catalog is not a folder of spec files: it's rows in a database — the test file, the route it runs against, and a metadata patch payload. A dispatcher applies the patch (this chart is a bar chart, this column uses the code editor widget, this kanban column has a hard WIP limit), runs the test against a real backend, then restores the metadata baseline. The patch is the point: the same test file exercises a different configuration of the same generated screen, which is what a metadata-driven product actually has to get right.
Journeys ask a different question: can a person who has only the documentation get from nothing to a working application. A journey takes a virtual machine with nothing on it, installs the published package the way the tutorial says to, and then uses the product the way the docs tell the reader to use it. No fixtures, no dev database, no warm node_modules, no shortcuts that only the author of the feature knows.
What a journey actually does
A journey is a sequence of modules, and the numbering is the plot: install, first-run wizard, getting started, apply the license, the business features one archetype at a time, the dashboard designer, a workflow, a report, the RAG chat, admin operations, teardown. Each module is a chapter of the tutorial, executed for real against a real browser on a real installation.
Two design decisions make it worth the minutes it costs.
Each module names the documentation pages it is enacting. A module declares docs: ['list-grid', 'parametric-dialog', 'kanban-list', …], loads those pages, and checks the strings they quote against what's actually on screen. If the page says the menu path is Samples > Cities, the journey opens the menu and looks for it. If the doc quotes a button label that no longer exists, that is a finding — against the documentation, not against the code. Docs rot silently; this is the only mechanism we have that makes them fail loudly.
A finding is not just pass or fail. The evidence collector records four classes:
| Class | What it means |
|---|---|
broken |
an assertion failed, a console or page error, a 4xx/5xx |
slow |
the step took longer than the budget declared for it |
unintuitive |
friction: the natural selector didn't match and a fallback did |
doc-error |
the documentation quotes something that isn't on screen |
That third one is the one I didn't expect to value so much. Every module declares the selector the documentation suggests first, and a fallback. When the fallback is what matches, nothing is broken — the feature works — but the path a reader would take didn't, and that gets recorded. It's the closest thing to an automated usability complaint I know how to write.
Each step is also timed against a declared budget (renderMs, interactionMs), so "it still works, it's just three times slower than it was" shows up as a finding instead of as a feeling.
The discipline around the run matters as much as the run. A journey that dies at minute forty is not restarted from scratch: the logs come off the guest first, and it resumes from the broken step, because the thirty-five minutes of installation before it were sane and the state they produced is the thing under test. Only a brand-new run is allowed to roll the VM back to its snapshot — restoring a snapshot on top of a run in progress destroys exactly the evidence the run existed to collect. And every deviation from the published artifact (running the installer from the repo instead of the site, patching a step script) has to declare itself as friction in the report, so nobody reads a green journey and believes the published package was the thing that passed.
How much of it there is
The docs-driven catalog is currently 266 tests over 82 documentation pages — roughly one family per documented feature, sized by how much surface that feature has. The heavier families are the ones with the most configuration to get wrong: application settings (18), the grid (12), the RAG chatbot (12), the pivot builder (11), metadata integration (10), the workflow designer (10), the dashboard designer (9), charts (8), timelines (8), kanban (7).
By category, that catalog covers:
- the data archetypes — grid, kanban, scheduler, chart, map, tree, carousel, timeline, spreadsheet, pivot, data repeater;
- the edit form and its widgets — the parametric dialog plus one family per field widget (lookup, upload, code editor, HTML area, dictionary, number, many-to-many, button, and the rest), each with its own configurations;
- the designers — dashboard, report, workflow, scene 3D, view builder;
- the platform behaviours that cut across every screen — authentication and authorisations, multi-tenant, optimistic concurrency, logic delete, conditional styling, validations, custom actions, callbacks and events, translations (interface and data), themes, responsive mobile layout, accessibility;
- the server side you reach through the UI — OData, Swagger, webhooks, scheduling, mailing, notifications, import/export, reporting, audit, SQL retry policy, custom exception handling;
- the paths a developer takes — initial scaffolding, the first-run wizard, the four framework/custom composition patterns, licensing, the performance inspector, the RAG chatbot and its tools.
The journeys are four packages — the packaged install on Windows with IIS and on Linux with nginx, plus the sources-and-VS-Code kit on both — of thirteen modules each. Those modules enact 25 documentation pages end to end, from install and first-run wizard through getting started, licensing, every business archetype, the dashboard designer, a workflow, a report, the RAG chat and the admin operations, down to teardown.
The two suites deliberately overlap. A grid tested by the e2e suite in forty configurations is also opened once by a journey, on a machine where nobody has ever run npm install — and the second one fails for reasons the first cannot see.
Four engines, two operating systems
None of this means much on one database. The framework generates SQL, and the four engines it supports disagree about almost everything worth disagreeing about: identifier quoting, pagination, date handling, what comes back from an insert, how a BLOB gets loaded, whether a bulk read is even the same statement. A feature that works on SQL Server and has never been run on Oracle is a feature with an unknown.
So the matrix is four engines — SQL Server, MySQL/MariaDB, PostgreSQL, Oracle — times two operating systems, Windows and Linux, and the two suites run across it. The dispatcher carries its own per-engine configuration; the journeys carry the engine as a parameter of the run and install the product against it from scratch.
The cells are not interchangeable, and that is the point. A defect can live in exactly one of them: the upload path story further down is a Windows-package defect that no Linux run would ever have produced, because the Linux packages ship an absolute path. Report queries that looked engine-neutral turned out to be SQL Server dialect running against MySQL. A test can pass on three engines and fail on the fourth for a reason that has nothing to do with the feature under test, and only running the fourth tells you.
What each one caught
The e2e suite, once it could see, found real product defects. The journeys found the ones I'd never have found otherwise.
The best example is an image that wouldn't display in a grid. The e2e failure was blunt: the thumbnail element loaded but had zero width. On a developer machine it worked. On the installed package it didn't.
Underneath there were four copies of the same path calculation — where does an uploaded file live — that had drifted apart. One in the upload writer, one in the controller that serves the file back, one in the code that reads it into a BLOB column, and a fourth in the step that moves a file from its temporary folder to its final one once the record has an id. Three of them resolved a relative uploadFolder setting against the project's data folder. The fourth resolved it against the web root.
The Windows package ships "uploadFolder": "Upload" — relative, because you can't ship the build machine's absolute paths. So on that package, and only on that package, the fourth copy looked for the file in a tree where it had never been, found nothing, and skipped the move in silence. The image stayed in its temporary folder, the grid asked for it by record id, and got a 404.
The smoking gun was nine empty directories. That fourth copy created its target folder before checking, so the installation had a little museum of folders under the web root — uploadsample/2124, uploadsample/2125, one per record that had ever been saved — all empty, all in the wrong tree. Once you see those, the bug takes ten minutes.
The other one was less subtle and more embarrassing: the sample data that ships with the package points at image files on the machine that built it. Rows in the demo table referencing C:\Users\...\Dropbox\pictures\.... On any clean installation those files don't exist, so a screen about image previews cannot work, ever, and not because of a defect in the product. That isn't a bug in code; it's a bug in what we ship, and only a clean VM was ever going to say so.
The pattern behind both is the same, and it's the reason the journeys exist: a developer machine is a machine where every mistake about the environment has already been forgiven. Absolute paths that happen to resolve. Files that happen to be there from six months ago. A database that was migrated by hand once and never again. Services already running. The installer never ran, so nothing the installer does was ever tested. A clean VM forgives nothing, which is the entire point.
The categories a journey catches that an e2e suite structurally cannot:
- What only exists on an installation. Packaged settings that differ from dev settings — like a relative
uploadFolder— and the code paths that only that value reaches. - What the installer does. First-run scripts, database creation, the wizard, the license, the folder layout. A suite that starts from an already-installed app tests none of it.
- What the documentation says. Menu paths, button labels, the order of steps. Those are only "true" if somebody follows them literally, and the journey is the only reader we have that never skips ahead.
- What ships in the box. Sample data, templates, report definitions. They're content, they're not covered by unit tests, and they break quietly.
- What got slower. With per-step budgets, a regression in startup or render time is a finding rather than something somebody eventually grumbles about.
One cause wearing five costumes
The single most useful hour of the whole campaign was realising that five unrelated-looking failures were one bug in the tests.
The symptoms: a chart's settings dialog never opened. A kanban card refused to move when dragged. A grid's row menu never appeared. A save seemed to never start, and timed out fifteen seconds later. A lookup showed no suggestions. Five features, five plausible product defects, five different-looking stack traces.
The cause: while the shell is loading it covers the page with an overlay, and every one of those interactions was a Playwright click with force: true. That option skips the actionability check — but it still delivers the event at the target's coordinates, which is where the overlay is. The click lands on the overlay, silently, and the test fails much later on the consequence.
Two things made it hard to see, and both are worth knowing:
- The failure screenshot lied. The scenario runner takes its screenshot after running the restore callback in
finally— and the restore saves for real. So the picture showed a closed form and a cheerful "updated" toast, fifteen seconds after the test had died on a click that never landed. The screenshot was of the cleanup, not of the failure. - A swallowed
catchmoves the failure somewhere innocent. The lookup helper waited for the suggestion panel with an 8-second timeout wrapped in.catch(() => {}), then clicked the option with a 30-second timeout. The error you get points at the option click — "no suggestions" — when the truth is the panel was never asked to open.
One shared helper now waits for the overlay to clear and verifies, via document.elementFromPoint, that the thing under the cursor is the thing we mean to click. Five tests went green. No product code changed, because there was nothing wrong with the product.
The honest version of that story is that it took three rounds of triage, and in two of them I was confidently wrong about the cause. The thing that finally worked wasn't cleverness, it was refusing to accept a diagnosis that hadn't been reproduced: open the page, click the button by hand, print what's actually under the pointer.
Where this leaves us
A red test on a generated UI is not evidence of a broken product. It's evidence that something in the chain between a metadata row and a pixel didn't behave — and that chain has a lot of links: the metadata, the code that reads it, the component that renders it, the environment it was installed into, the data that shipped with it, the documentation that described it, and the test itself. Every campaign we run produces failures in all of those categories, and the ones that are genuine product defects are a minority.
Deciding which link is the whole job. The only way to make that decision cheap is to build the product so it can be asked — which is what the attributes, the readiness signals and the inspection handle are for — and to keep the diagnosis honest: reproduce before you conclude, read the error rather than the screenshot, and never let a swallowed exception pick the suspect for you.
The hooks are documented, including the ones this post used: the testability and e2e hooks page lists the DOM attributes, the readiness signals, the introspection handle and its flag, and the selectors that will betray you — generated Angular ids, translated labels, positions in a table.
If you're building something that renders itself from data, the summary is short: decide what your UI tells the outside world about itself, make that a contract, and make sure the contract survives the production build. Everything else is downstream of that.