In-Application Integration Tests — Architecture

Status: v1 implemented (2026-08-02). ZuloOne's native equivalent of the MIQS IntegrationTest subsystem: author C# tests in the browser, run them against the live system, and see pass/fail with timing — every case isolated in a transaction that is rolled back so nothing a test writes persists.

Component map#

Browser (React)
  /tests page ......... TestsPage.tsx      (list + Monaco + Check/Save/Run)
  Auto test tab ....... AutoTestPanel.tsx  (Enable/Regenerate + Run) in the
                        document & dictionary designers
  results ............. TestResultsView.tsx (pass/fail tree, duration, GC)
        │  api/metadata/tests  (axios, JWT)
        ▼
ZuloOne.Core (ASP.NET API)
  IntegrationTestsController .. CRUD + run + run-all + generate + check
  IntegrationTestRunner ....... compile → reflect → run each case in a
                                rolled-back TransactionScope; collect results
  TestHarness ................. ITestHarness over the real services
  AutoEntityTestGenerator ..... emits CRUD test source for a dict/doc type
        │ compiles via
        ▼
ZuloOne.Runtime
  RuntimeCompiler ............. Roslyn → collectible AssemblyLoadContext
  Testing/ .................... IntegrationTestScriptBase, [IntegrationTest],
                                Assert, TestContext, ITestHarness, result DTOs
        │ persisted as
        ▼
ZuloOne.Data / SQL Server
  MetaIntegrationTests table .. (EF migration AddMetaIntegrationTests)

The test model#

A test is a single MetaIntegrationTest : BaseExtension record that carries its own C# source (Code) — a deliberate simplification of the MIQS four-table graph (IntegrationTest → Script → Run → Result). Fields: Code, TargetObjectType ("Dictionary"/"Document"/null), TargetObjectMetaId, IsAutoGenerated, IsActive, GroupName, LastRunStatus, LastRun.

Runs and results are transient in v1 — returned by the API and rendered, not stored. (Persisting run history is a v2 item.)

Authoring surface (ZuloOne.Runtime/Testing)#

A test script subclasses IntegrationTestScriptBase and marks cases with [IntegrationTest("name")]. Write it through the managers, with typed entities — the same door production code uses. A test that reaches for the data-shaped harness proves something about the harness, not about the system:

using ZuloOne.Runtime.Testing;
using ZuloOne.Managers;
using ZuloOne.Runtime.Generated;   // JournalEntry, Currency, …TablePartRow — see below
using System.Collections.Generic;
using System.Threading.Tasks;

public class MyTests : IntegrationTestScriptBase
{
    // Managers as ambient properties (MIQS keeps them on the script base; here they
    // are declared per-script because ZuloOne.Runtime cannot name Core's interfaces).
    private static IDictionaryManager DictionaryManager => GetService<IDictionaryManager>();
    private static IDocumentManager DocumentManager => GetService<IDocumentManager>();
    private static ITotalsManager TotalsManager => GetService<ITotalsManager>();

    [IntegrationTest("posts a journal entry")]
    public async Task Post()
    {
        var acct = DictionaryManager.NewRecord<ChartOfAccounts>();
        acct.Code = "1000";
        acct.AccountType = AccountType.Asset;      // generated enum, not a string
        acct = await DictionaryManager.SaveRecordAsync(acct);

        // No subtype passed — the document starts in the type's INITIAL subtype.
        var entry = await DocumentManager.NewDocumentAsync<JournalEntry>();
        entry.Lines.Add(new JournalEntryLinesTablePartRow { Account = acct.MetaId, Debit = 100m });
        await DocumentManager.SaveDocumentAsync(entry);

        // A draft carries no movements — assert it BEFORE the transition, or the
        // assertions after it pass even when the document posted itself on save.
        Assert.IsTrue((await TotalsManager.QueryMovementsAsync("GL")).Count == 0);

        // Posting is an ASSIGNMENT plus a save (MIQS `doc.SubtypeID = …; Save(doc)`),
        // not a name-string call. SaveDocumentAsync routes the changed subtype
        // through the posting engine.
        entry.Subtype = JournalEntry.Subtypes.Posted;
        await DocumentManager.SaveDocumentAsync(entry);

        Assert.IsTrue((await TotalsManager.QueryMovementsAsync("GL")).Count == 1);
    }
}

Four rules this example encodes, each of which cost a real debugging session:

1. using ZuloOne.Runtime.Generated; is mandatory in a non-partial test. A plain class X : IntegrationTestScriptBase compiles through RuntimeCompiler.CompileAsync and receives NO global usings; without the directive every generated class is CS0246, and Currency binds to an unrelated inaccessible type so the error (CS0122) misdescribes the cause. A partial class test goes through CompileMultipleAsync(withScriptFramework: true) and does get ZuloOne.Managers and ZuloOne.Runtime.Generated injected — the directive is harmless there, so writing it always is the simpler rule.
2. Assert the pre-state. A document type with postOnSave posts as soon as it is saved. Without an "empty before the transition" assertion, a test that only checks the end state passes whether or not the transition did anything.
3. Do not touch the database after catching an expected failure. Guards refuse by throwing, and the throw dooms the runner's ambient TransactionScope. Any query after the catch fails with "the operation is not valid for the state of the transaction" — which masks the assertion you meant to make. Assert on the refusal itself.
4. Follow the declared workflow. Jumping straight to a terminal subtype only works when the document has no subtype to transition from; once it starts in its initial subtype the transition table is enforced (Draft → Received is rejected when the type declares Draft → Ordered → Received).

Db (ITestHarness) remains available for the cases a manager does not cover — NewId(), CreateSampleRecordAsync(table) (required fields by EDT type, required references resolved; the MIQS CreateSampleRecord equivalent), and raw InsertAsync/QueryAsync for probing storage directly. Reach for it deliberately, not by default.

• Assert — IsTrue/IsFalse/IsNull/IsNotNull/AreEqual/AreNotEqual/Fail/Throws<T>; failures throw IntegrationTestException.
• TestContext — Log/Write/WriteLine (per-case output buffer, AsyncLocal) and Skip/SkipClass/AbortAll (throw SkipTestException).
• GetService<T>() — any platform service or model service contract.

Because RuntimeCompiler.GetDefaultReferences() references ZuloOne.Runtime, ZuloOne.Shared and ZuloOne.Core, scripts can using ZuloOne.Runtime.Testing; and even reference Core/Shared types.

The runner (ZuloOne.Core/Testing/IntegrationTestRunner)#

For each test:

1. Compile Code via RuntimeCompiler.CompileAsync under a unique assembly name (ITest_{metaId}_{guid}) so the compiler's type cache never returns a stale build.
2. Find the non-abstract IntegrationTestScriptBase subclass in that assembly; reflect its [IntegrationTest] methods, ordered by MetadataToken (declaration order).
3. For each case:
   • a fresh DI scope → fresh scoped services (DbContextFactory, DataService…);
   • new TransactionScope(RequiresNew, ReadCommitted, AsyncFlowEnabled);
   • TestContext.Recreate(), then invoke (awaiting if it returns Task);
   • record outcome (Passed / Failed / Skipped), duration, and the output/error;
   • dispose the scope WITHOUT Complete() → the transaction rolls back, so every write the test made is undone.
4. Aggregate GC stats (gen0/1/2 counts + allocated bytes) and total duration.

Why TransactionScope instead of savepoints#

MIQS wraps each case in a SQL SAVEPOINT and rolls back. ZuloOne reuses the existing DataService/DocumentPostingService/RegisterMovementService unchanged by relying on TransactionScope ambient enlistment: SqlConnection auto-enlists, and because the services open/close one connection per call sequentially, the transaction stays lightweight (no MSDTC promotion on LocalDB / SQL Server). Not calling Complete() discards all of it. This is the key design choice that let the whole subsystem land without touching the data layer.

Auto entity test (AutoEntityTestGenerator)#

POST /api/metadata/tests/generate builds a CRUD test without the user writing code, mirroring the MIQS "Auto entity test" tab:

• Dictionary → Create / Update / Delete cases; required fields are populated with type-appropriate sample values (by EDT type).
• Document type → a Create case via Db.CreateDocumentAsync. Re-generating replaces the existing auto test for that object in place.

API (api/metadata/tests)#

TestRunResult → scripts[] { cases[] { outcome, durationMs, output, errorDetail } } plus totals and GC stats. TestOutcome serializes as the string "Passed"/"Failed"/"Skipped" (JsonStringEnumConverter).

Regression coverage#

src/ZuloOne.Tests/IntegrationTestRunnerTests.cs (live-API xUnit):

• Runner_reports_pass_and_fail_and_rolls_back_writes — asserts pass+fail counts, the failing case's message, and that the inserted row was rolled back.
• Auto_generated_dictionary_test_passes — generate + run the auto CRUD test.
• CreateSampleRecord_resolves_a_required_reference — the sample builder resolves a required reference to a seeded record (not Guid.Empty).

Self-cleaning tests#

Every ZuloOne.Tests regression cleans up after itself: await using var _cleanup = _fx.NewCleanup(s) deletes all metadata rows and physical tables named with the test's random suffix on dispose (even on failure), via DELETE /api/dev/test-artifacts/{suffix} (guarded to suffix length ≥ 6). This keeps the dev DB free of test garbage — verified net-zero across a full run.

Disposable metadata (widgets, placements, print forms, documents created by the test) must stamp TestBench (ApiFixture.TestModelId), not Core. Core is sync-down-only: anything the watcher exports there becomes a platform file. The sweeper discovers widget / assignment / placement / print-form names (and JSON name inside files named by host object id). Without that, a widget-only test left AsgWid* in Core/Widgets forever. Names ending in RANDOM / RANDOMId (schema-alter leftovers like AddNNDict…RANDOM) are a suffix of their own — the 8-hex matcher never sees them.

Cleanup discovery walks the workspace for suffix-named files, skipping only .git. Not every dot-folder: .generated/ holds one generated entity per object, suffix and all, and skipping it leaves those files behind for the sync to import straight back — the object then reappears after the test that made it is gone.

Running the suite in parallel#

The suite runs its classes concurrently: ~3.5 minutes where a serial run took ~14.

How it is wired. xUnit parallelises across collections and runs the classes inside one serially, so the collection count is the degree of parallelism. ApiFixture.cs defines nine: api-1…api-8 for the 75 suffix-isolated classes, and api for the 32 that touch state no suffix namespaces — security principals, layers, the workspace folder, the menu tree, global constants, generated-code inventories.

What makes it safe. Two things, both load-bearing:

1. Every test names its objects with a random 8-hex suffix (ApiFixture.Suffix), so concurrent tests cannot collide on names.
2. The two genuinely global server operations sit behind process-wide gates — SchemaSynchronizer.SyncGate and ModelCompileService.CompileGate. Without them, the 64 files calling /api/schema/sync race on DDL against shared tables.

What the api collection does NOT give you. It serialises its classes against each other; it still runs alongside api-1…api-8. xUnit v2 has no globally exclusive collection. A test asserting on a whole-stand property — an export's prune count, an inventory — needs the stand to itself, so it carries [Trait("Exclusive", "true")] and test_teststand.bat runs it in a second, single-threaded pass. WorkspaceOrphanPruneTests is the worked example: it exports the whole database into its own root, and beside parallel tests the third export prunes their deleted objects instead of nothing.

The remaining floor. /api/schema/sync is gated, so its cost is strictly serial however many threads run: ~0.5s × ~90 call sites. Per-test cleanup (~1.1s, dominated by the workspace walk across the container's bind mount) is not serial and divides by the thread count.

The Kind=unit tier — what CI actually runs#

Of the 133 *Tests.cs files, 116 drive a live Core on :5257 through ApiFixture. They need a running server, a database, and ZuloOne__CoreDevelopmentMode=true (the fixture constructor calls the dev-only POST /api/dev/test-artifacts/sweep). None of that exists on a build agent, and the full suite is only intermittently green even on a healthy stand — so gating releases on it would gate them on the weather.

The remaining 17 files carry [Trait("Kind", "unit")]. They compile against Core sources linked into the test project and touch no I/O, so they run offline in about a second:

dotnet test src/ZuloOne.Tests/ZuloOne.Tests.csproj --filter "Kind=unit"
# 234 passed (175 [Fact]/[Theory] declarations; theories expand)

This is the tier .github/workflows/ci.yml runs, with two guards against the one thing a trait cannot do by itself — be remembered:

• RunConfiguration.TreatNoTestsAsError=true, so a filter that matches nothing goes red instead of green.
• A guard step asserting the defining property: any *Tests.cs that does not mention ApiFixture is a unit test and must carry the trait. A new one without it fails the build.

A FullyQualifiedName~ list in YAML was rejected for this: ~ is a substring match, so a renamed class silently stops matching while its siblings still do — coverage drops with the build still green.

The API tier's replacement in CI is the container boot smoke test, which exercises migrations → schema sync → metadata compile against a throwaway Postgres. No unit test covers that path, and it is exactly what a deploy depends on.

Known limitations / v2 backlog#

• Run/Result history not persisted (transient only).
• Rollback is per-case full-scope, not nested savepoints.
• CreateSampleDocument doesn't yet resolve subtypes or auto-fill FK references.
• No jump-to-failed-line from a result back into the editor.
• No per-test permissions, no breakpoints, no generated-code preview.