--- url: /test-inventory/guide/why.md description: >- What the runners' lists leave out, and what test-inventory reads from the code without a run. --- # Why test-inventory Playwright and Vitest can list their tests: `playwright test --list` and `vitest list`. A list says which tests exist. It doesn't say which of them won't run, under what condition, or why. test-inventory reads the test files and answers those questions from the code, without running anything. ## What a runner's list leaves out This test from the [trading-web](/guide/examples#playwright-end-to-end-tests) example doesn't run on WebKit: ```ts // FIXME: TRD-412 the confirm dialog stays open on WebKit test( 'rejects a limit price far from the last trade', { annotation: { type: 'issue', description: 'TRD-412' } }, async ({ browserName, orderTicket }) => { test.fixme(browserName === 'webkit', 'confirm dialog stays open on WebKit'); await orderTicket.buy({ quantity: 1, limitPrice: 1 }); await expect(orderTicket.error).toHaveText('Limit price is more than 50% away from the last trade.'); }, ); ``` `playwright test --list` shows it as a test expected to pass, in the WebKit project too: the `fixme` call in its body only takes effect while the test runs. The reason and the comment with the ticket are not in the list. For a Vitest suite, `vitest list` leaves skipped and todo tests out altogether. test-inventory reports the test like this: ```json { "fullName": "orders > limit orders > rejects a limit price far from the last trade", "state": "fixme", "isConditional": true, "condition": "browserName === 'webkit'", "reason": "confirm dialog stays open on WebKit", "stateLine": 28, "annotations": [{ "type": "issue", "description": "TRD-412", "lineStart": 26 }], "comments": ["FIXME: TRD-412 the confirm dialog stays open on WebKit"] } ``` Each record also has the tags, an id that stays the same when the test moves to another line, and a hash of its body that changes when its code does. Playwright and Vitest suites get the same keys. [Compared with the runners](/guide/runners) has the full comparison. ## Questions it answers Each links to a recipe that runs on a scan: * [Tests that don't run, and why](/guide/recipes#tests-that-don-t-run-and-why): skipped, fixme, todo and expected-to-fail tests, with the reason given in the code. * [Tickets mentioned by tests that don't run](/guide/recipes#tickets-mentioned-by-tests-that-don-t-run): check that every disabled test has one, or list the tests a ticket blocks. * [Tests per tag](/guide/recipes#smoke-tests-per-file): the smoke tests in each file, or the tests with any other tag. * [Focused tests](/guide/ci#github-actions): fail a CI job when a test or a describe uses `.only`. * [Tests a pull request stopped running](/guide/ci#tests-that-stopped-running-in-a-pull-request): the scans of its base and its head, compared by id. * [The state of a test in an editor or a linter](/guide/library#scansource): `scanSource` reads code that isn't saved to disk. ## Why from the source * **Nothing runs.** A scan needs no browsers, no build, no environment variables and no installed runner. It takes seconds, so it fits any CI job that has Node.js, and it can scan a checkout of another branch, such as the base of a pull request. * **Conditions stay as written.** The runner evaluates a condition such as `browserName === 'webkit'` while it runs, for one browser on one machine at a time. The inventory keeps the condition, so one scan covers them all. * **The output is plain JSON.** Two scans of the same code give the same bytes, so an inventory can be kept as a build artifact, compared with an earlier one and read from any language. ## What it doesn't do test-inventory reads declarations, not results. It doesn't know whether a test passed or how long it took, and it doesn't evaluate a condition; the runner's report has those. Some things are only known at run time, such as the tests a config file's `grep` leaves out. [Known limits](/guide/limits) lists them; most show up in the output as a diagnostic. When all you need is the names of the tests that will run on one machine, the runner's own list is the right tool. --- --- url: /test-inventory/guide/getting-started.md description: >- Install test-inventory and list the tests of a Playwright or Vitest suite as JSON, without running them. --- # Getting started test-inventory reads JavaScript and TypeScript test files and lists the tests in them as JSON. It parses the code and never runs it, so it needs no browsers, no build and no configuration of the test runner. 0.1 supports [Playwright](https://playwright.dev) 1.42 and newer and [Vitest](https://vitest.dev) 3.0 and newer. It needs Node.js 22.12 or newer, and runs on Linux, macOS and Windows, on x64 and Arm. ## Install ```sh npm install --save-dev test-inventory ``` Or try it without adding it to the project. npx downloads the package for one run, after asking: ```sh npx test-inventory "tests/**/*.spec.ts" --framework playwright --output inventory.json ``` ## Scan a Playwright suite Take a spec from the UI tests of a trading app, `tests/orders/limit-orders.spec.ts`: ```ts import { test, expect } from '../../fixtures/index'; import { TAGS } from '../../test-data/tags'; test.describe('orders', { tag: '@orders' }, () => { test.describe('limit orders', () => { // ... // FIXME: TRD-412 the confirm dialog stays open on WebKit test( 'rejects a limit price far from the last trade', { annotation: { type: 'issue', description: 'TRD-412' } }, async ({ browserName, orderTicket }) => { test.fixme(browserName === 'webkit', 'confirm dialog stays open on WebKit'); await orderTicket.buy({ quantity: 1, limitPrice: 1 }); await expect(orderTicket.error).toHaveText('Limit price is more than 50% away from the last trade.'); }, ); }); }); ``` Run the scan from the project root: ```sh npx test-inventory "tests/**/*.spec.ts" --framework playwright --pretty --output inventory.json ``` The test's record in `inventory.json`, shortened: ```json { "id": "da2f5749ba9ab068", "relativeFilePath": "tests/orders/limit-orders.spec.ts", "lineStart": 24, "lineEnd": 32, "fullName": "orders > limit orders > rejects a limit price far from the last trade", "state": "fixme", "isFixme": true, "isConditional": true, "condition": "browserName === 'webkit'", "reason": "confirm dialog stays open on WebKit", "stateSource": "test", "stateLine": 28, "tags": ["@orders"], "annotations": [{ "type": "issue", "description": "TRD-412", "lineStart": 26 }], "comments": ["FIXME: TRD-412 the confirm dialog stays open on WebKit"] } ``` The test imports `test` from a local fixtures file. The scan follows that one import to check that it exports a Playwright test object created with `test.extend()`. ### Gherkin features with playwright-bdd [playwright-bdd](https://github.com/vitalets/playwright-bdd) turns `.feature` files into Playwright test files. Generate them first, then scan the generated folder: ```sh npx bddgen npx test-inventory ".features-gen/**/*.spec.js" --framework playwright --output inventory.json ``` The records point at the generated files, and each starts with a comment that names its feature file. ## Scan a Vitest suite ```sh npx test-inventory "tests/**/*.{test,test-d}.ts" "src/**/*.ts" --framework vitest --output inventory.json ``` Include the source files when they contain [in-source tests](https://vitest.dev/guide/in-source), and the `*.test-d.ts` files for [type tests](https://vitest.dev/guide/testing-types). ## Next * [Examples](/guide/examples) for end-to-end tests, a Node service, React, Vue, browser mode and a monorepo. * The [command line](/guide/cli) options. * [Query recipes](/guide/recipes) and [CI recipes](/guide/ci). * The [MCP server and other ways in](/guide/agents) for AI agents. * Every key in the [output](/reference/output). * The [diagnostics](/reference/diagnostics): what the scan could not read, and why. *** If test-inventory turns out to be useful to you, [sponsoring its development](https://github.com/sponsors/Huldoser) pays for the time that keeps it current with new Playwright and Vitest versions. --- --- url: /test-inventory/guide/examples.md description: >- Scans of a made-up trading app in common setups, from Playwright end-to-end tests to a monorepo. --- # Examples Each example below is a small project of a made-up trading app, in a common setup. The projects are in [`tests/fixtures`](https://github.com/Huldoser/test-inventory/tree/main/tests/fixtures), and the output shown is from scanning them; the test suite checks that both stay as shown here. ## Playwright end-to-end tests `trading-web` tests the trading app's UI with page objects and fixtures. Its specs import `test` from a fixtures file that extends Playwright's: ```ts // fixtures/index.ts export const test = base.extend({ signedInPage, loginPage: async ({ page }, use) => { await use(new LoginPage(page)); }, // ... }); export { expect } from '@playwright/test'; ``` The scan follows that import, sees `base.extend()` on Playwright's `test`, and reads the specs that use it like any other: ```ts // tests/orders/order-lifecycle.spec.ts import { test, expect } from '../../fixtures/index'; // One order moves through every status, so each step depends on the one before. test.describe('order lifecycle', { tag: '@orders' }, () => { test.describe.configure({ mode: 'serial' }); test('places a limit order for 100 NVDA', async ({ orderTicket, blotter }) => { // ... test('shows a partial fill', async ({ blotter }) => { await expect(blotter.status('NVDA')).toHaveText(/Partially filled \(\d+\/100\)/); }); ``` ```sh npx test-inventory "tests/**/*.spec.ts" --framework playwright --output inventory.json ``` The describe's record has the mode set by `describe.configure()`, and its tests inherit the mode and the tag: ```json { "id": "2b2853e11e8f8699", "relativeFilePath": "tests/orders/order-lifecycle.spec.ts", "lineStart": 4, "title": "order lifecycle", "isSerial": true, "tags": ["@orders"] } ``` ```json { "fullName": "order lifecycle > shows a partial fill", "suiteId": "2b2853e11e8f8699", "isSerial": true, "tags": ["@orders"] } ``` ## A Node.js service `order-service` is an HTTP service that sends orders to a broker, tested with Vitest. Its repository tests need a database and skip themselves without one: ```ts // tests/repository/orders-repository.test.ts describe.skipIf(!process.env.DATABASE_URL)('orders repository', { tags: ['db'] }, () => { // ... test('stores an order with its broker id', async () => { ``` ```sh npx test-inventory "tests/**/*.test.ts" --framework vitest --output inventory.json ``` Each test in the describe gets the describe's condition and tags. `stateSource` and `stateLine` say where the state comes from: ```json { "fullName": "orders repository > stores an order with its broker id", "state": "skip", "isConditional": true, "condition": "!process.env.DATABASE_URL", "stateSource": "suite", "stateLine": 5, "tags": ["db"] } ``` The project's Vitest config defines the `db` tag, and a tag in the config can skip tests or change how they run. The scan does not read the config, and says so: ```json { "code": "config-may-change-state", "level": "warning", "relativeFilePath": "tests/repository/orders-repository.test.ts", "lineStart": 5 } ``` A skip from the test context is read as well, with its condition and note: ```ts // tests/risk/daily-loss-limit.test.ts test('resets the loss at the start of the next session', ({ skip }) => { skip(process.env.TZ !== 'America/New_York', 'session boundaries are checked in New York time'); ``` ```json { "fullName": "daily loss limit > resets the loss at the start of the next session", "state": "skip", "condition": "process.env.TZ !== 'America/New_York'", "reason": "session boundaries are checked in New York time", "stateSource": "test", "stateLine": 14, "isParallel": true } ``` A known bug marked with `test.fails` is `expectedToFail`, and the comment above it is kept: ```ts // tests/broker/fill-events.test.ts // BUG: TRD-588 fractional partial fills are rounded down to whole shares test.fails('keeps the fractional shares of a partial fill', () => { ``` ```json { "fullName": "filledShares > keeps the fractional shares of a partial fill", "state": "expectedToFail", "comments": ["BUG: TRD-588 fractional partial fills are rounded down to whole shares"] } ``` ## React with Testing Library `trading-dashboard` keeps its tests next to its React components, in `*.test.tsx` files. JSX is read like the rest of the code: ```tsx // src/components/OrderTicket.test.tsx // TODO(TRD-731): the date picker for good-till-date orders is being rebuilt it.skip('submits a good-till-date order', async () => { const user = userEvent.setup(); render(); await user.click(screen.getByRole('button', { name: 'Good till date' })); }); it.todo('shows the estimated commission'); ``` ```sh npx test-inventory "src/**/*.test.tsx" --framework vitest --output inventory.json ``` ```json { "fullName": "OrderTicket > submits a good-till-date order", "relativeFilePath": "src/components/OrderTicket.test.tsx", "state": "skip", "comments": ["TODO(TRD-731): the date picker for good-till-date orders is being rebuilt"] } ``` ```json { "fullName": "OrderTicket > shows the estimated commission", "state": "todo", "bodyHash": null } ``` ## Vue with Vue Test Utils `watchlist-vue` turns on Vitest's globals, so its specs use `describe` and `it` without importing them: ```ts // tests/unit/Watchlist.spec.ts import { mount } from '@vue/test-utils'; import Watchlist from '../../src/components/Watchlist.vue'; // ... describe('when the feed is delayed', () => { it('shows how many minutes the quotes are delayed', () => { const wrapper = mount(Watchlist, { props: { quotes, delayedMinutes: 15 } }); expect(wrapper.get('[data-test="delay-banner"]').text()).toBe('Quotes are delayed by 15 minutes'); }); it.todo('greys out quotes older than the delay'); }); ``` ```sh npx test-inventory "tests/unit/**/*.spec.ts" --framework vitest --output inventory.json ``` A name that a file uses but never imports or declares is read as a Vitest global. The scan then looks for `globals: true` in a Vitest or Vite config, or `vitest/globals` in a tsconfig, in the file's folder and the folders above it, and warns with `globals-not-enabled` when there is none. Here `vitest.config.ts` turns them on: ```json { "fullName": "Watchlist > when the feed is delayed > greys out quotes older than the delay", "suitePath": ["Watchlist", "when the feed is delayed"], "state": "todo" } ``` ## Vitest browser mode `chart-browser` draws candlestick charts on a canvas and tests them in Chromium and WebKit with Vitest's browser mode. One test is skipped in WebKit: ```ts // src/candlestick-chart.test.ts // WebKit returns an empty blob for canvases larger than the device pixel ratio allows in headless mode. test.skipIf(server.browser === 'webkit')('exports the chart as a PNG', async () => { ``` ```sh npx test-inventory "src/**/*.test.ts" --framework vitest --output inventory.json ``` ```json { "fullName": "CandlestickChart > exports the chart as a PNG", "state": "skip", "isConditional": true, "condition": "server.browser === 'webkit'", "stateSource": "test" } ``` Browser instances come from the config, which is not read, so each test is one record. The runner runs it once per browser. ## In-source and type tests `strategy-engine` keeps some tests next to the code they test, inside `if (import.meta.vitest)`: ```ts // src/pnl/format.ts if (import.meta.vitest) { const { describe, it, expect } = import.meta.vitest; describe('formatPnl', () => { it('adds a plus sign to gains', () => { expect(formatPnl(1234.5)).toBe('+$1,234.50'); }); ``` It also has type tests in `*.test-d.ts` files: ```ts // tests/orders/order.test-d.ts describe('Order', () => { test('narrows to a limit order by its type', () => { const order = { symbol: 'AAPL', side: 'buy', type: 'limit', quantity: 10, limitPrice: 187.5 } as Order; if (order.type === 'limit') expectTypeOf(order).toEqualTypeOf(); }); ``` Scan the source files and the type test files along with the tests: ```sh npx test-inventory "tests/**/*.{test,test-d}.ts" "src/**/*.ts" --framework vitest --output inventory.json ``` The `import.meta.vitest` check is how in-source tests are written, so it does not count as a condition: ```json { "fullName": "formatPnl > adds a plus sign to gains", "relativeFilePath": "src/pnl/format.ts", "lineStart": 16, "state": "active", "isInCondition": false } ``` ```json { "fullName": "Order > narrows to a limit order by its type", "relativeFilePath": "tests/orders/order.test-d.ts", "isTypeTest": true } ``` ## A monorepo `trading-monorepo` has three packages, each with its own Vitest config, and a root config that lists them as projects. Scan them all from the root: ```sh npx test-inventory "packages/*/src/**/*.test.ts" --framework vitest --output inventory.json ``` Paths are relative to the root, so each record names its package: ```json { "fullName": "QuoteCache > returns the mid price of a fresh quote", "relativeFilePath": "packages/market-data/src/quote-cache.test.ts" } ``` Only `market-data` turns on globals, in `packages/market-data/vitest.config.ts`. The scan finds that config from the package's test files, so it gives no `globals-not-enabled` warning: ```json { "fileCount": 3, "suiteCount": 3, "testCount": 8, "activeCount": 7, "todoCount": 1, "diagnosticCount": 0 } ``` To scan one package, pass it as the root: `--root packages/risk`. The framework version is read from `node_modules` in the root or a folder above it, so a package of a monorepo gets the version installed at the top. --- --- url: /test-inventory/guide/cli.md description: >- The test-inventory command, its options and exit codes, and how it finds the framework version. --- # Command line ```sh test-inventory --framework [options] ``` The globs select the test files, relative to `--root`. Quote them so the shell passes them on unchanged. `node_modules` and `.git` folders are always left out. As in Playwright and Vitest, globs match folders and files whose names start with a dot, as in `.storybook/`. On Windows a `\` in a glob separates folders, so `tests\**\*.spec.ts` works; on Linux and macOS it escapes the next character, as in `tests/\[legacy\]/*.spec.ts`. A `/` separates folders on every system. | Option | Meaning | | ----------------------------- | --------------------------------------------------------------------------------------------------- | | `--framework ` | `playwright` or `vitest`. Required. One run reads one framework. | | `--root ` | The project root. Paths in the output are relative to it. Default: the current folder. | | `--ignore ` | Leave out matching files. Can be given more than once. | | `--framework-version ` | Apply the rules of this version instead of the one read from the project. | | `--comments ` | Which tests get the comment lines above them: `all`, `non-active` or `none`. Default: `non-active`. | | `--state ` | Keep tests in this state, such as `skip` or `fixme`. Can be given more than once. | | `--tag ` | Keep tests with this tag, as the output writes it, such as `@smoke`. Can be given more than once. | | `--grep ` | Keep tests whose `fullName` matches the regular expression, ignoring case. | | `--only` | Keep tests that use `.only`, or are inside a describe that does. | | `--without-reason` | Keep tests that don't run and give no reason. | | `--fields ` | Write only these keys of each test, separated by commas. `id` is always kept. | | `--format ` | `json`, or `jsonl` for one record per line. Default: `json`. | | `-o, --output ` | Write the JSON to a file instead of standard output. The file is replaced only once it is complete. | | `--pretty` | Indent the JSON with two spaces. Without it the JSON is on one line. | | `-h, --help` | Show the help. | | `-v, --version` | Show the version. | ## Filters The filters answer a question about a few tests without reading the whole inventory. A test is kept when it passes every filter given; a filter given more than once keeps a test that has any of its values. To list the tests in the web app that are skipped or marked fixme, with their reasons: ```sh npx test-inventory "tests/**/*.spec.ts" --framework playwright --state skip --state fixme --fields fullName,reason ``` Each test then has only `id` and the keys asked for: ```json { "id": "da2f5749ba9ab068", "fullName": "orders > limit orders > rejects a limit price far from the last trade", "reason": "confirm dialog stays open on WebKit" } ``` The rest of the output describes the tests that were kept: `files` has their files, `suites` the describes around them, `diagnostics` their problems and the ones that are not about a single test, and `summary` counts them. Files and diagnostics whose tests were all left out are dropped too, but a file without tests stays with its errors, since tests may be missing from the answer. `summary.fileCount` is still the number of files scanned, and each file's `suiteCount` and `testCount` still count the whole file. Output written with `--fields` leaves keys out of the test records, so it no longer matches the [JSON Schema](/reference/output#versioning), which requires every key. Without `--fields`, filtered output matches it. To keep only some files, give a narrower glob instead: the files left out are not read at all. ## JSON Lines `--format jsonl` writes the same inventory with one record on each line, which `grep`, `jq -c` and tools that read newline-delimited JSON, such as DuckDB, take one line at a time. Each line has a `type` key; the [output reference](/reference/output#json-lines) lists the line types. ## Exit codes | Code | When | | ---- | --------------------------------------------------------------------------------- | | 0 | The scan finished. Diagnostics are part of the output, not failures. | | 1 | Reserved for a future release. | | 2 | The arguments were invalid, the output could not be written, or the scan crashed. | These are the exit codes ESLint uses. A reader that stops early, as `| head` does, is not an error: the output ends there and the code is 0. Globs that match no files are not an error either: the output is an empty inventory with a `no-files-matched` warning, and one line on standard error says so. There are no options to fail a build on skipped tests or on diagnostics. The output is data; a check that needs a rule can read it, for example with Node.js: ```sh npx test-inventory "tests/**/*.spec.ts" --framework playwright --output inventory.json node -e "if (require('./inventory.json').summary.onlyCount > 0) process.exit(1)" ``` ## Framework version Some syntax depends on the framework version: `test.fail.only` needs Playwright 1.49, the `lock` option 1.63, and Vitest 5 removed `describe.sequential`. The version is read from the project, in this order: 1. `node_modules/@playwright/test/package.json` or `node_modules/vitest/package.json`. 2. `package-lock.json`, `pnpm-lock.yaml`, `yarn.lock` or `bun.lock`. 3. The version in `package.json`. Each is looked for in the root, then in each folder above it, as a package in a monorepo may be installed or locked at the top. `--framework-version` overrides all of them. The version ends up in `frameworkVersion` in the output. When `package.json` gives only a range, such as `^4.0.0`, `frameworkVersion` is the lowest version in it, the rules of the newest release in that major apply, and an `approximate-framework-version` warning names the range and the file. When nothing is found, `frameworkVersion` is `null`, the rules of the newest supported version apply, and an `unknown-framework-version` warning says so. ## Large suites Files are read one at a time and their syntax trees are dropped before the next file, and the JSON is written one record at a time. Memory grows with the number of records, not with the size of the files. --- --- url: /test-inventory/guide/library.md description: >- The scan and scanSource functions, filterInventory and pickTestFields, the inventory they return and the JSON Schema of the output. --- # Library The package also exports the functions the command line uses. It is ESM only. ```ts import { filterInventory, pickTestFields, scan, scanSource, type Inventory } from 'test-inventory'; ``` ## scan Scans the files that match the patterns and returns the inventory. The options are the same as on the [command line](/guide/cli), in camel case. ```ts /** * Scans the test files that match `patterns` under `root`, one file at a time. * * Throws a `TypeError` for invalid options. Problems with files and code are reported as diagnostics. */ export function scan(options: ScanOptions): Promise; export interface ScanOptions { /** * Glob patterns for the test files, relative to `root`. As in Playwright and Vitest, they match names that start with * a dot too. */ patterns: string[]; framework: Framework; /** The project root. Paths in the output are relative to it. Defaults to the current directory. */ root?: string; /** Glob patterns to leave out, in addition to `node_modules` and `.git`. */ ignore?: string[]; /** The framework version whose rules apply, instead of the one read from the project. */ frameworkVersion?: string; /** Which tests get the comments above them. Defaults to `non-active`. */ comments?: CommentsMode; } export type Framework = 'playwright' | 'vitest'; export type CommentsMode = 'all' | 'non-active' | 'none'; ``` To count the tests that are always skipped in the web app: ```ts const inventory = await scan({ patterns: ['tests/**/*.spec.ts'], framework: 'playwright', root: 'apps/trading-web', ignore: ['tests/legacy/**'], }); const skipped = inventory.tests.filter((test) => test.isSkipped && !test.isConditional); console.log(`${skipped.length} tests are always skipped`); ``` ## scanSource Scans the code of one file without reading anything from disk. Use it in editors, linters or tests. ```ts /** * Scans the code of one test file, without reading anything from disk. * * Throws a `TypeError` for invalid options. Problems in the code itself are reported as diagnostics. * A custom `test` object imported from a local file can't be followed here; use `scan()` for that. */ export function scanSource(options: ScanSourceOptions): Inventory; export interface ScanSourceOptions { code: string; /** Path of the file from the project root, used in the output and in ids. */ relativeFilePath: string; framework: Framework; /** The framework version whose rules apply. The newest rules apply when it is missing. */ frameworkVersion?: string | null; /** Which tests get the comments above them. Defaults to `non-active`. */ comments?: CommentsMode; } ``` To read the state of a test from code that is not on disk: ```ts const inventory = scanSource({ code: "import { test } from 'vitest';\ntest.todo('charges the daily borrow fee');", relativeFilePath: 'tests/orders/short-selling.test.ts', framework: 'vitest', frameworkVersion: '5.0.3', }); inventory.tests[0].state; // 'todo' ``` Because it reads no files, `scanSource` cannot follow a custom `test` object imported from a fixtures file. Tests that use one are reported with an `unresolved-test-import` diagnostic; `scan` follows the import. ## filterInventory Keeps the tests that pass a filter, as the [filter options](/guide/cli#filters) on the command line do. The result is an inventory of those tests, which still matches the JSON Schema. ```ts /** * Keeps the tests that pass every condition of the filter; a list keeps a test that has any of its values. The * result describes only those tests: their files, the describes around them, their diagnostics, and a summary * counted over them. Files without any tests stay, with their diagnostics, since tests may be missing from them. * `summary.fileCount` stays the number of files scanned. * * Throws a `TypeError` for an invalid filter. */ export function filterInventory(inventory: Inventory, filter: InventoryFilter): Inventory; export interface InventoryFilter { /** Keeps tests in one of these states. */ states?: TestState[]; /** Keeps tests with one of these tags, as the output writes them. */ tags?: string[]; /** Keeps tests whose `fullName` matches this regular expression, ignoring case, as Playwright's `--grep` does. */ grep?: string; /** Keeps tests that use `.only`, or are inside a describe that does. */ only?: boolean; /** Keeps tests that don't run and give no reason. Tests in a file the runner refuses are left out. */ withoutReason?: boolean; } ``` To list the smoke tests that are marked fixme, with their files: ```ts const inventory = await scan({ patterns: ['tests/**/*.spec.ts'], framework: 'playwright' }); const fixme = filterInventory(inventory, { states: ['fixme'], tags: ['@smoke'] }); for (const test of fixme.tests) console.log(`${test.relativeFilePath}:${test.lineStart} ${test.fullName}`); ``` ## pickTestFields Keeps only `id` and the given keys on each test, as `--fields` does, to make the output smaller. TypeScript knows which keys are left. ```ts /** * Leaves only `id` and the given keys on each test record, in that order, to make the output smaller. Suites, * diagnostics and the summary are unchanged. The result no longer matches the JSON Schema, which requires every key. * * Throws a `TypeError` for a key that test records don't have. */ export function pickTestFields( inventory: Inventory, fields: readonly K[], ): PickedInventory; /** An inventory whose test records have only `id` and the keys picked with `pickTestFields()`. */ export type PickedInventory = Omit & { tests: Pick[]; }; ``` ```ts const short = pickTestFields(fixme, ['fullName', 'reason']); short.tests[0].reason; // string | null ``` ## Inventory Both scan functions return an inventory, the object the command line writes as JSON. The [output reference](/reference/output) describes each of its records key by key. ```ts /** The output of one scan. */ export interface Inventory { /** Version of this format. It changes only when a key is removed, renamed or changes meaning. */ schemaVersion: 1; tool: Tool; /** The framework the scan was run for. */ framework: Framework; /** Read from the project, or given with `--framework-version`; null when unknown. */ frameworkVersion: string | null; summary: Summary; /** One per scanned file, in the order the files were scanned. */ files: FileRecord[]; suites: SuiteRecord[]; tests: TestRecord[]; diagnostics: Diagnostic[]; } export interface Tool { name: 'test-inventory'; /** Version of test-inventory that produced the output. */ version: string; } ``` ## Errors Every function throws a `TypeError` that names the option when an option is invalid. Anything about the scanned code or files becomes a [diagnostic](/reference/diagnostics) in the result. ## JSON Schema The output's JSON Schema ships with the package, under the `schemaVersion` it describes, so a reader can pin the version of the format it understands: ```ts import schema from 'test-inventory/schema/v1.json' with { type: 'json' }; ``` It is also published at [huldoser.github.io/test-inventory/schema/v1.json](https://huldoser.github.io/test-inventory/schema/v1.json), the URL in its `$id`. --- --- url: /test-inventory/guide/recipes.md description: >- Node.js scripts that answer questions about a scan, such as which tests don't run and why. --- # Query recipes The output is plain JSON, so any language can answer questions about a test suite. Each recipe below is a Node.js script that answers one question. It reads `inventory.json` from the current folder: save it as a `.mjs` file and run it with `node`. The output under each recipe is what it prints for the [trading-web](/guide/examples#playwright-end-to-end-tests) example project; the test suite runs the recipes on that project's scan, so they stay correct. ## Tests that don't run, and why Skipped, fixme, todo and expected-to-fail tests, with the reason given in the code. ```js import { readFileSync } from 'node:fs'; const { tests } = JSON.parse(readFileSync('inventory.json', 'utf8')); for (const test of tests.filter((test) => test.state !== 'active')) { const reason = test.reason ? ` (${test.reason})` : ''; console.log(`${test.state} ${test.relativeFilePath}:${test.lineStart} ${test.fullName}${reason}`); } ``` ```text skip tests/alerts.spec.ts:11 price alerts > notifies when the price crosses above the alert (Alerts only trigger while the market is open) skip tests/alerts.spec.ts:16 price alerts > does not notify for a price that was never reached (Alerts only trigger while the market is open) skip tests/alerts.spec.ts:21 price alerts > deletes an alert (Alerts only trigger while the market is open) expectedToFail tests/backtest.spec.ts:20 backtests > exports trades to CSV fixme tests/login.spec.ts:25 login > asks for a one-time code on a new device fixme tests/orders/limit-orders.spec.ts:24 orders > limit orders > rejects a limit price far from the last trade (confirm dialog stays open on WebKit) fixme tests/price-stream.spec.ts:8 live price stream > updates prices without a reload (The WebSocket reconnects in a loop on WebKit (TRD-390)) fixme tests/price-stream.spec.ts:14 live price stream > shows a stale badge when the stream disconnects (The WebSocket reconnects in a loop on WebKit (TRD-390)) ``` ## Tickets mentioned by tests that don't run Ticket numbers such as `TRD-412` in the comments above a test or in its reason. Use it to check that every disabled test has a ticket, or to list the tests a ticket blocks. ```js import { readFileSync } from 'node:fs'; const { tests } = JSON.parse(readFileSync('inventory.json', 'utf8')); for (const test of tests.filter((test) => test.state !== 'active')) { const text = [...test.comments, test.reason ?? ''].join(' '); const tickets = [...new Set(text.match(/[A-Z]+-[0-9]+/g))].sort(); for (const ticket of tickets) console.log(`${ticket} ${test.relativeFilePath}:${test.lineStart} ${test.fullName}`); } ``` ```text TRD-301 tests/login.spec.ts:25 login > asks for a one-time code on a new device TRD-412 tests/orders/limit-orders.spec.ts:24 orders > limit orders > rejects a limit price far from the last trade TRD-390 tests/price-stream.spec.ts:8 live price stream > updates prices without a reload TRD-390 tests/price-stream.spec.ts:14 live price stream > shows a stale badge when the stream disconnects ``` ## Smoke tests per file Tests tagged `@smoke`, counted per file. Tags include those inherited from describes. ```js import { readFileSync } from 'node:fs'; const { tests } = JSON.parse(readFileSync('inventory.json', 'utf8')); const perFile = Map.groupBy( tests.filter((test) => test.tags.includes('@smoke')), (test) => test.relativeFilePath, ); for (const [file, smoke] of [...perFile].sort(([a], [b]) => (a < b ? -1 : 1))) { console.log(`${smoke.length} ${file}`); } ``` ```text 4 tests/login.spec.ts 1 tests/orders/limit-orders.spec.ts 3 tests/orders/market-orders.spec.ts 1 tests/portfolio.spec.ts 1 tests/watchlist.spec.ts ``` *** If these recipes save you time, consider [sponsoring test-inventory](https://github.com/sponsors/Huldoser). A query you miss here is welcome as an [issue](https://github.com/Huldoser/test-inventory/issues). --- --- url: /test-inventory/guide/ci.md description: >- Keep the inventory as a CI artifact, block focused tests and find the tests a pull request stopped. --- # CI recipes A scan takes seconds and needs no browsers and no build, so it fits in any job that has Node.js. These recipes keep the inventory as a build artifact, list the tests that don't run in the job summary, block focused tests, and find tests that stopped running in a pull request. The examples scan a Playwright suite. For Vitest, change the globs and `--framework vitest`. ## GitHub Actions ```yaml # .github/workflows/test-inventory.yml name: Test inventory on: pull_request: push: branches: [main] permissions: contents: read jobs: inventory: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 - uses: actions/setup-node@v7 with: node-version: 24 cache: npm - run: npm ci - run: npx test-inventory "tests/**/*.spec.ts" --framework playwright --output inventory.json - name: Fail on focused tests run: node -e "if (require('./inventory.json').summary.onlyCount > 0) process.exit(1)" - name: List tests that don't run run: node scripts/not-running.mjs >> "$GITHUB_STEP_SUMMARY" - uses: actions/upload-artifact@v7 with: name: test-inventory path: inventory.json ``` The focused-test check exits with 1 when a test or a describe uses `.only`, which fails the step. The summary step runs a script kept in the repository, so it can be read, linted and run locally like any other code: ```js // scripts/not-running.mjs import { readFileSync } from 'node:fs'; const { tests } = JSON.parse(readFileSync('inventory.json', 'utf8')); console.log('### Tests that do not run'); for (const test of tests.filter((test) => test.state !== 'active')) { console.log(`- \`${test.state}\` ${test.fullName} (${test.relativeFilePath}:${test.lineStart})`); } ``` For the [trading-web](/guide/examples#playwright-end-to-end-tests) example, it writes: ```text ### Tests that do not run - `skip` price alerts > notifies when the price crosses above the alert (tests/alerts.spec.ts:11) - `skip` price alerts > does not notify for a price that was never reached (tests/alerts.spec.ts:16) - `skip` price alerts > deletes an alert (tests/alerts.spec.ts:21) - `expectedToFail` backtests > exports trades to CSV (tests/backtest.spec.ts:20) - `fixme` login > asks for a one-time code on a new device (tests/login.spec.ts:25) - `fixme` orders > limit orders > rejects a limit price far from the last trade (tests/orders/limit-orders.spec.ts:24) - `fixme` live price stream > updates prices without a reload (tests/price-stream.spec.ts:8) - `fixme` live price stream > shows a stale badge when the stream disconnects (tests/price-stream.spec.ts:14) ``` ## Tests that stopped running in a pull request A test's `id` does not change when the test moves to another line, so two scans can be compared by id. This job scans the pull request's base in a second worktree and lists the tests that ran there and don't run any more: ```yaml # Another job in the workflow above, under jobs: stopped-running: if: github.event_name == 'pull_request' runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 with: fetch-depth: 0 - uses: actions/setup-node@v7 with: node-version: 24 cache: npm - run: npm ci - run: git worktree add ../base "origin/${{ github.base_ref }}" - run: npx test-inventory "tests/**/*.spec.ts" --framework playwright --root ../base --output base.json - run: npx test-inventory "tests/**/*.spec.ts" --framework playwright --output inventory.json - name: List tests that stopped running run: node scripts/stopped-running.mjs base.json inventory.json ``` ```js // scripts/stopped-running.mjs import { readFileSync } from 'node:fs'; const [base, head] = process.argv.slice(2).map((file) => JSON.parse(readFileSync(file, 'utf8'))); const active = new Set(base.tests.filter((test) => test.isActive).map((test) => test.id)); for (const test of head.tests.filter((test) => !test.isActive && active.has(test.id))) { console.log(`${test.state} ${test.fullName} (${test.relativeFilePath}:${test.lineStart})`); } ``` A renamed test gets a new id, so it shows up as removed and added rather than as stopped. ## GitLab CI ```yaml # .gitlab-ci.yml test-inventory: stage: test image: node:24 script: - npm ci - npx test-inventory "tests/**/*.spec.ts" --framework playwright --output inventory.json - node -e "if (require('./inventory.json').summary.onlyCount > 0) process.exit(1)" artifacts: paths: - inventory.json expire_in: 30 days ``` ## Azure Pipelines ```yaml - script: npx test-inventory "tests/**/*.spec.ts" --framework playwright --output $(Build.ArtifactStagingDirectory)/inventory.json displayName: Scan the tests - publish: $(Build.ArtifactStagingDirectory)/inventory.json artifact: test-inventory ``` ## Jenkins ```groovy stage('Test inventory') { steps { sh 'npx test-inventory "tests/**/*.spec.ts" --framework playwright --output inventory.json' sh '''node -e "if (require('./inventory.json').summary.onlyCount > 0) process.exit(1)"''' archiveArtifacts artifacts: 'inventory.json' } } ``` *** If test-inventory runs in your pipeline, [sponsoring it](https://github.com/sponsors/Huldoser) helps keep it working with each new Playwright and Vitest release. --- --- url: /test-inventory/guide/agents.md description: >- Give AI coding agents a project's tests with the test-inventory MCP server, the command line or the docs as plain text. --- # AI agents Coding agents ask the questions about tests that people do: which tests cover limit orders, which are skipped on WebKit and why, whether a change left a `.only` behind. test-inventory answers them from the source code, so an agent gets the answer without running the suite. ## MCP server [test-inventory-mcp](https://github.com/Huldoser/test-inventory-mcp) is a [Model Context Protocol](https://modelcontextprotocol.io) server built on test-inventory. Agents in Claude Code, Cursor, VS Code and other MCP clients call its tools instead of running commands. It reads the test files, runs no code from the project, writes nothing and opens no network connections. In Claude Code, from the project folder: ```sh claude mcp add --scope project test-inventory -- npx -y test-inventory-mcp@0.2 "tests/**/*.spec.ts" --framework playwright ``` `--scope project` saves the server in `.mcp.json`, so everyone who works on the project gets it. The globs and `--framework` are the ones you would give the [command line](/guide/cli). The setup for Cursor and VS Code, the options and the security notes are in the server's [README](https://github.com/Huldoser/test-inventory-mcp#setup). | Tool | Answers | | ------------------ | --------------------------------------------------------------------------------------------- | | `test_summary` | Counts by state, tag and annotation, for the whole suite or for each file or tag. | | `find_tests` | The tests that match states, tags, title text, `.only` or a missing reason, with chosen keys. | | `get_tests` | Every key of up to 20 tests by id, with the diagnostics about them. | | `scan_diagnostics` | What the scan could not read, such as a file that failed to parse. | The tests have the keys of the [output](/reference/output). Each call scans the files again, so answers follow the agent's edits. The server has its own versions. Before 1.0, a change that breaks its tools raises the minor version, so `test-inventory-mcp@0.2` gets fixes only. ## Command line An agent with a shell can run the command line itself. The [filters](/guide/cli#filters) and `--fields` keep the output to what the question needs: ```sh npx test-inventory "tests/**/*.spec.ts" --framework playwright --state skip --state fixme --fields fullName,reason ``` A line in the project's `AGENTS.md`, or `CLAUDE.md` for Claude Code, tells agents when to use it: ```md To find tests and their skip reasons without running them, use `npx test-inventory "tests/**/*.spec.ts" --framework playwright` with `--state`, `--tag`, `--grep` and `--fields`. Options: https://huldoser.github.io/test-inventory/guide/cli ``` ## Docs as plain text The docs are also published for agents to read: [`llms.txt`](https://huldoser.github.io/test-inventory/llms.txt) lists the pages, [`llms-full.txt`](https://huldoser.github.io/test-inventory/llms-full.txt) has all of them in one file, and each page is available as Markdown at its address with `.md` added, such as [`/guide/cli.md`](https://huldoser.github.io/test-inventory/guide/cli.md). --- --- url: /test-inventory/guide/limits.md description: >- What a scan can't know without running the code, such as config files and imported values. --- # Known limits test-inventory reads the code as text and builds a syntax tree. It does not run the code, so a few things the runner knows at run time are out of its reach. Most limits below show up in the output, as a diagnostic or a key such as `isInLoop`; each section says when one does not. ## Config files are not read `playwright.config.*` and `vitest.config.*` can change which tests run and how: `grep`, `testIgnore`, `forbidOnly`, `fullyParallel`, projects, and Vitest tags with options. The scan reads only the test files. A test that a config `grep` leaves out is still listed, and `isParallel` only reflects what is written in the file. Nothing in the output says so. When a Vitest file uses tags, a `config-may-change-state` warning points out that a tag defined in the config can skip a test or change how it runs. The same goes for run settings: the config's `timeout`, `retries` and `repeatEach`, and Vitest's `testTimeout`, `retry` and tags with options. A `timeout`, `retries` or `repeats` of `null` means the file sets none, so the config decides. Calls that change settings while tests run are not read either: `testInfo.setTimeout()` and `testInfo.slow()`, Vitest's `vi.setConfig()`, and calls in a function passed to a hook by name, as in `test.beforeEach(slowOnCI)`. `expectCount`, `steps` and `fixtures` read the test function only. Assertions and steps in page objects and helpers, and fixtures that other fixtures or automatic fixtures set up, are not in them. A test function passed by name is read when the file declares it, not when it is imported. Vitest's `annotate()` in a hook is not read. Vitest's `benchmark.include` in the config can name other benchmark files than `*.bench.ts` and `*.benchmark.ts`. ## Imported values are not resolved Values written in the same file are worked out: string and number literals, constants, enums, arrays and objects built from them, spreads, string concatenation and template strings. A value imported from another file is not. When a title, tag, annotation or table comes from an import, the record keeps what it can and a diagnostic names the module the value came from: ```ts import { TAGS } from '../../test-data/tags'; test('cancels an open limit order', { tag: TAGS.risk }, async ({ orderTicket }) => { // listed without the tag, with an unresolved-tag warning }); ``` Two kinds of import are followed: a custom `test` object, and a function that declares tests, as described under [tests in helper functions](#tests-in-helper-functions). They are found through: * a relative path, as in `../fixtures`; * a path alias from `compilerOptions.paths` or `baseUrl` in the tsconfig nearest the importing file, with every file it `extends`, packages such as `@trading/tsconfig` included. Without a `tsconfig.json`, the nearest `tsconfig.base.json` is used, as in an Nx workspace; a solution-style tsconfig takes them from the projects it `references`; * an `imports` entry in the nearest `package.json`, as in `#fixtures`; * a workspace package whose `exports`, `types` or `main` point to TypeScript source, or to build output with the source next to it, as `dist/esm/index.js` is to `src/index.ts`: through its link in `node_modules`, or before an install, from the `workspaces` of `package.json` and from `pnpm-workspace.yaml`. Patterns and links that lead out of the workspace folder are not followed. A custom `test` object is followed through as many files as it takes: a fixtures file may re-export it, as in `export * from './base'`, build it with `extend` on a test object it imports from another file, or build it with a function, written there or imported, as in `export const test = defineDeskTest('equities')`. A fixtures file that builds its test object with a call the scan can't follow gets the error described below, also when it doesn't import the framework itself. Installed packages are not followed. When a call shaped like a test, outside any test or hook body, uses a name from an import that can't be followed, an `unresolved-test-import` error names the import and says where the chain of files ends: a file that is missing or can't be parsed, an installed package, or a file that doesn't export a test object. The tests that use it are missing. The error needs a name that reads like a test function, as in `brokerTest('routes to NASDAQ', ...)`, or a member such as `.describe` after it. With such a name, a call with a title and a body counts through any member, as in `it.effect('fills an order', ...)` on a test object that a package extends, and with a timeout after the body. Helpers shaped the same way, such as `waitForFill('AAPL', (order) => ...)`, are not reported. `scanSource()` reads no files, so it follows no imports. ## Test objects made by helpers A function that returns a test object or a describe function is followed when it is declared in the file or imported from a local file, through the imports above: ```ts function onNode(major: number) { return NODE_MAJOR < major ? it.skip : it; } onNode(22)('streams quotes over HTTP/3', async () => {}); ``` What it returns may be called in place, as here, or kept in a variable, as in `const quoteTest = onNode(22)`. When the function returns a different one on different runs, only the modifiers they all have apply, as for a variable chosen by a condition such as `isCI ? it.skip : it`: this test is listed as `active`, not as a conditional skip. A function that returns what another such function does is followed too, as in `const fastTest = () => onNode(22)`. A test object made by a function the scan cannot follow, as from an installed package, is read on trust, with one `local-test-object` warning per variable or function: ```ts const test = createBrokerTest({ venue: 'NASDAQ' }); test('fills a market order', async ({ page }) => {}); liveBrokerTest('NYSE')('sends a live order', async ({ page }) => {}); ``` This applies only to a variable or function named like a test object (`test`, `it`, `describe`, `suite` or a name ending in `Test`) whose test object is called with a title and a function, or through a member such as `test.skip`. Other calls, such as a helper named `expectFill('AAPL', 100, 'filled', check)`, are not tests. ## States set in helper functions In Playwright, a `test.skip()`, `test.fixme()`, `test.fail()` or `test.describe.configure()` call inside a plain function applies to the tests, hooks and describes that call that function by name, also through helpers that call other helpers. A function called from the file or a describe may also be imported from another file: ```ts function skipWithoutMarketData() { test.skip(!process.env.MARKET_DATA_URL, 'needs the market data feed'); } test('streams AAPL quotes', async ({ page }) => { skipWithoutMarketData(); // state: skip, isConditional: true }); ``` When the file exports the function, or nothing in the file calls it by name, the scan can't tell which tests it affects, and a `state-in-helper` warning says the call was not applied. That includes a function passed as a value, as in `test.beforeEach(skipWithoutMarketData)`. ## Tests in helper functions The runner registers a test when the `test()` call runs, in the describe that is being collected at that moment. For a test declared in a function, that is the describe around each call of the function, not the place where the test is written. The scan does the same: it reads the function at each call from the file or a describe, with that call's arguments, so titles built from them are worked out: ```ts function validatesQuantity(side: string) { test(`${side} rejects a zero quantity`, async ({ orderTicket }) => {}); } test.describe('buy orders', () => { validatesQuantity('buy'); }); test.describe.skip('short sells', () => { validatesQuantity('short'); }); ``` Playwright lists two tests here, and so does the scan: `buy orders > buy rejects a zero quantity`, and `short sells > short rejects a zero quantity`, which is skipped. Both have `isInFunction: true` and the line of the `test()` call. A function imported from another file is followed the same way, through the imports above. Its records give the call in the scanned file as their location, since that is the line that registers the tests, and `declaredAt` gives the file and line where the helper writes the test. Helpers are followed up to 16 calls deep, and at up to 10,000 calls in a file, counting the calls inside helpers; real suites make far fewer. Past 10,000, a `test-in-function` warning marks the call where the scan stopped. A `test-in-function` warning remains for tests in a function the scan can't place: one that nothing in the file calls by name, one the file only exports, and one called only from tests or hooks, where the runner doesn't collect tests. Such a test is listed where it is written. Tests in a callback passed to another function, as in `withPaperAccount(() => { test(...) })`, are listed where they are written, with no warning. ## Loops and tables are one record A test declared in a loop, `forEach` or `map` is listed once, with `isInLoop: true` and a `test-in-loop` warning. In Playwright, a test in a loop whose title path is the same on each pass is registered twice with the same title, so Playwright refuses the file: it gets a `duplicate-title` error, and its tests are `notLoaded`. A `test.each` or `test.for` table is listed once, with `isParameterized: true` and `caseCount` set when the table is written in the file. The runner creates one test per iteration or per case; expanding them is planned for a later release. ## Conditions are not evaluated `test.skip(browserName === 'webkit', 'reason')` gives `state: 'skip'` with `isConditional: true` and the condition's source text. Whether the test is skipped on a given run depends on the browser, the environment or the clock. A condition written as a constant, such as `test.skip(true)` or `test.skipIf(false)`, is worked out. A skip inside `if`, `?:`, `&&`, `||`, `??`, `switch` or a `catch` block runs only when that code does, so it is conditional too, with the source text of what decides it as its condition. For a `catch` block that is the clause itself, `catch` or `catch (error)`, since the block runs only when the `try` block throws. ## What a scan never sees Tests created inside a function from an installed package, or from a file the scan can't read, as in `registerOrderTests(orderTicket)`, are not listed, and nothing in the output says so: the call does not look like a test. When a call does look like a test and goes through an import that can't be followed, an `unresolved-test-import` error says tests are missing. --- --- url: /test-inventory/guide/runners.md description: >- How the inventory differs from playwright test --list and vitest list, and how CI compares them. --- # Compared with the runners Playwright and Vitest can list their tests too: `playwright test --list --reporter=json` and `vitest list --json`. Both load the test files, and so run the code at the top level of every file. test-inventory reads the files without running them. | | test-inventory | `playwright test --list` | `vitest list` | | ------------------------------ | ------------------------------------ | ---------------------------- | ------------------------------- | | Runs code from the test files | No | Yes, top level and describes | Yes, or parses them in Vitest 5 | | Needs the runner installed | No | Yes | Yes | | Loops and `.each` tables | One record, with the number of cases | One test per iteration | One test per case | | Skipped and todo tests | Listed, with the state | Listed | Left out | | Skip conditions | Kept as source text | Evaluated at load time | Not shown | | Tags | As written, `@smoke` | Without the `@`, `smoke` | Not shown | | Stable id and body hash | Yes | No | No | | One schema for both frameworks | Yes | No | No | ## Matching records with the runners' lists To match a record with a runner's entry, use the file and the line range: the entry's line falls between `lineStart` and `lineEnd`. Columns differ: Playwright reports the column of the call's last name, `fail` in `test.fail(`, or of its opening parenthesis, while test-inventory and Vitest report the column where the call starts. ## Vitest 5's static list `vitest list` in Vitest 5 parses the files instead of running them, unless `--no-static-parse` is given. Its results differ from what the runner does: it treats any `.only` as applying to the whole run, lets the last modifier in a chain win, and treats `runIf(true)` and `skipIf(false)` as skipped. It also keeps escapes such as `\"` in names as written, drops a test or describe chained after a call, as in `describe.runIf(hasBroker).skip(...)`, and lists a call such as `testRouting('NYSE')` as a test because its name starts with `test`. test-inventory follows what the runner does. ## Checked in CI On every pull request, CI scans the fixture projects in the repository and compares the inventory with Playwright 1.42, 1.63 and 1.64 and Vitest 3.0 and 5.0. Playwright lists the tests with `--list`; Vitest runs them with its JSON reporter, which, unlike `vitest list`, includes skipped and todo tests. Each test is matched by file and line, and its full title, its state and its tags are compared. A state set by a condition is left out, since the runner decides it on the machine it runs on. Any difference fails the check. The [real projects](/guide/real-projects) that use Vitest are compared too, with Vitest's static parser, which lists their tests without an install. The scan applies the rules of the same Vitest version. Each difference must have a known cause, such as one of the parser's differences above; one without fails the check, which doesn't block a merge, since it clones the projects over the network. --- --- url: /test-inventory/guide/real-projects.md description: The open-source test suites CI scans at pinned commits on every pull request. --- # Real projects CI scans the test suites of well-known open-source projects at pinned commits on every pull request, and a few of them were checked against the runners' own lists. `npm run corpus` repeats the scans; the list of projects is in [`tests/compare/corpus.json`](https://github.com/Huldoser/test-inventory/blob/main/tests/compare/corpus.json), with the number of files and tests each scan found. A scan fails when it finds fewer, when a file gets a `parse-error` or `internal-error`, or when a file calls `test`, `it` or `describe` but gets neither a test record nor an error that says why. The projects that use Vitest are also listed with Vitest's static parser and compared with the scan, as described in [Compared with the runners](/guide/runners#checked-in-ci). ## Scans Folders named `fixtures` and `__fixtures__` are left out: they hold test inputs, some with syntax errors on purpose, that the projects' runners never load. | Project | Framework | Files | Tests | | --------------------------------------------------------------------------------------------------- | ---------- | ----: | -----: | | [freeCodeCamp/freeCodeCamp](https://github.com/freeCodeCamp/freeCodeCamp) `34b1acd`, end to end | Playwright | 84 | 273 | | [vuejs/core](https://github.com/vuejs/core) `4ab865a` | Vitest | 196 | 3,729 | | [sveltejs/kit](https://github.com/sveltejs/kit) `3bc9234`, test apps | Playwright | 10 | 404 | | [sveltejs/kit](https://github.com/sveltejs/kit) `3bc9234`, unit | Vitest | 78 | 707 | | [microsoft/playwright](https://github.com/microsoft/playwright) `4357c23` | Playwright | 538 | 5,984 | | [supabase/supabase](https://github.com/supabase/supabase) `18080d8`, end to end | Playwright | 32 | 263 | | [supabase/supabase](https://github.com/supabase/supabase) `18080d8`, unit | Vitest | 815 | 8,305 | | [storybookjs/storybook](https://github.com/storybookjs/storybook) `e57cf30` | Vitest | 827 | 9,464 | | [TanStack/router](https://github.com/TanStack/router) `663282b`, end to end | Playwright | 309 | 1,487 | | [TanStack/router](https://github.com/TanStack/router) `663282b`, unit | Vitest | 450 | 4,801 | | [vitejs/vite](https://github.com/vitejs/vite) `b6c20f6` | Vitest | 180 | 1,919 | | [vitest-dev/vitest](https://github.com/vitest-dev/vitest) `3e3624c` | Vitest | 509 | 3,438 | | [nuxt/nuxt](https://github.com/nuxt/nuxt) `0296ca4` | Vitest | 266 | 2,603 | | [excalidraw/excalidraw](https://github.com/excalidraw/excalidraw) `4c00f31` | Vitest | 142 | 2,216 | | [getsentry/sentry-javascript](https://github.com/getsentry/sentry-javascript) `dffd6b1`, end to end | Playwright | 714 | 2,188 | | [getsentry/sentry-javascript](https://github.com/getsentry/sentry-javascript) `dffd6b1`, unit | Vitest | 853 | 10,124 | No scan had a `parse-error` or `internal-error`. Most diagnostics are warnings about values the scan can't work out without running the code: titles built in loops, `.each` tables built by functions, and tests declared in loops. The errors and the larger warning counts have these causes: * **Playwright's own suite, 118 `unresolved-test-import` errors.** Its tests import `test` from fixtures files such as `pageTest.ts` and `browserTest.ts`, which build it in several layers from other local files; those are followed. 98 imports end at the copy of Playwright that `tests/playwright-test/stable-test-runner` pins in its own `node_modules`, which a checkout without an install doesn't have. The other 20 are in `tests/installation`, whose `npmTest.ts` builds its test object on `_baseTest`, an internal export of Playwright, not on `test`. Its 4 `local-test-object` warnings are variables such as `const test = baseTest.extend(...)` over that pinned copy, read on trust. Its one `state-in-helper` warning is a `test.skip()` inside a fixture, which skips the tests that use the fixture. The framework version is unknown because the repository is Playwright itself. * **Nuxt, 24 `unresolved-test-import` errors.** Playwright tests that the Vitest pattern also matches import `test` from a helper that builds it from `@nuxt/test-utils/playwright`. * **Sentry unit, 24 `unresolved-test-import` errors.** 7 files are Bun tests that import `test`, `it` and `describe` from `bun:test`, and 5 import them from `@effect/vitest`, an installed package that wraps Vitest. Each name is reported once per file. * **Vue core, 217 tests in helper functions.** `render.spec.ts` declares its tests in `testRender()`, which it calls three times, once for each way to render, and `Teleport.spec.ts` calls `runSharedTests()` in an eager and a defer describe. The scan records those tests at each call, in the describe around it, as Vitest registers them. Other projects do the same in places, such as freeCodeCamp's `donatePageTests()`, which runs in a signed-in and a signed-out describe. * **Vite, 11 `local-test-object` warnings.** `const it = await createModuleRunnerTester(...)` builds a test object from a helper; its calls are read as tests. * **Vitest's own suite.** 10 `nested-test` warnings come from tests written inside test bodies, which Vitest's own tests serialize and run in a separate Vitest. The one `globals-not-enabled` warning is from the same code, which runs there with `globals: true`. The framework version is unknown because the repository is Vitest itself. * **Sentry end to end.** One `framework-mismatch`: a Vitest file inside a Playwright test app. One `nested-test`: a `test.skip` written inside another test's body, which Playwright never collects. ## Compared with the runners Three of the projects were installed and listed with their own runner, and the lists were matched with the scan by file and line, as described in [Compared with the runners](/guide/runners). This was done with test-inventory 0.1.1, before tests in helper functions were recorded at each call. | Project and runner | Records | Listed by the runner | Differences | | ---------------------------------------------------- | ------: | -------------------: | ----------: | | supabase `e2e/studio`, Playwright 1.59.1 | 253 | 256 | 0 | | Vite `packages`, Vitest 5.0.3 | 878 | 1,036 | 4 | | TanStack Router `packages/router-core`, Vitest 4.1.4 | 1,486 | 3,643 | 7 | The runners list more tests because they repeat a test for each loop iteration and each `.each` case, which the scan keeps as one record with `isInLoop` or `caseCount`. Every difference has a known cause: * Vite: three tests inside `if (isWindows)` are not collected on Linux; the scan marks them `isInCondition`. One `test.skip` test is missing from `vitest list`, which leaves out skipped and todo tests. * TanStack Router: two `test.skip` tests are missing from `vitest list` for the same reason. Three skipped tests are in a `*.perf.test.ts` file that the package's Vitest config excludes, while the scan's pattern included it. Two type tests share a title, and Vitest lists type tests without a line, so each matches both entries. --- --- url: /test-inventory/reference/output.md description: >- Every key of the JSON output for tests, suites, the summary and diagnostics, and how it is versioned. --- # Output A scan returns one JSON object. Its [JSON Schema](https://huldoser.github.io/test-inventory/schema/v1.json) is published with each release. ```json { "schemaVersion": 1, "tool": { "name": "test-inventory", "version": "0.1.0" }, "framework": "playwright", "frameworkVersion": "1.63.0", "summary": { "fileCount": 11, "testCount": 31 }, "suites": [], "tests": [], "diagnostics": [] } ``` ## Rules for every key * Every key is always present, in every record, for both frameworks. Nothing is `undefined` or left out. The one exception is asked for: `--fields` keeps only the named keys of each test. * A boolean that cannot be true in a framework is `false`: `isFixme` in Vitest, `isShuffled` in Playwright. * A list with nothing in it is `[]`. * A value that does not exist or could not be worked out is `null`. * Two scans of the same code give the same output, byte for byte. There is no timestamp. * Paths are relative to the project root and use `/` on every system. * Lines and columns start at 1. A line ends at `\n`, `\r\n` or `\r`, and columns count UTF-16 code units, as editors do. In the TypeScript declarations on this page, ids and hashes are a `Hash`, lines and columns a `Position`, and counts a `Count`: ::: details Hash, Position and Count ```ts /** @pattern ^[0-9a-f]{16}$ */ type Hash = string; /** * @asType integer * @minimum 1 */ type Position = number; /** * @asType integer * @minimum 0 */ type Count = number; ``` ::: ## Top level | Key | Type | Meaning | | ------------------ | -------------- | ------------------------------------------------------------------------------------------------------------------ | | `schemaVersion` | `1` | The version of this format. See [versioning](#versioning). | | `tool` | object | `name` is `test-inventory`; `version` is the version that produced the output. | | `framework` | string | `playwright` or `vitest`, as given to the scan. | | `frameworkVersion` | string or null | Read from the project, or given with `--framework-version`. See [framework version](/guide/cli#framework-version). | | `summary` | object | Counts over the whole scan. | | `files` | array | One record per scanned file. | | `suites` | array | One record per describe. | | `tests` | array | One record per test declaration. | | `diagnostics` | array | What the scan could not read, or read only in part. | ::: details TypeScript ```ts /** The output of one scan. */ export interface Inventory { /** Version of this format. It changes only when a key is removed, renamed or changes meaning. */ schemaVersion: 1; tool: Tool; /** The framework the scan was run for. */ framework: Framework; /** Read from the project, or given with `--framework-version`; null when unknown. */ frameworkVersion: string | null; summary: Summary; /** One per scanned file, in the order the files were scanned. */ files: FileRecord[]; suites: SuiteRecord[]; tests: TestRecord[]; diagnostics: Diagnostic[]; } export interface Tool { name: 'test-inventory'; /** Version of test-inventory that produced the output. */ version: string; } export type Framework = 'playwright' | 'vitest'; ``` ::: ## Tests A test record holds the values the runner applies to the test, including those it inherits from the describes and the file around it. | Key | Type | Meaning | | ------------------------------------------------------------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | string | 16 hex characters from the file path, the describe titles and title without tags, and the position among tests with the same title. It stays the same when the test moves to another line. | | `bodyHash` | string or null | 16 hex characters from the test function's syntax tree, so formatting, comments, quote style, semicolons, trailing commas and line endings don't change it. For a function passed by name, the hash of that function when it is declared in the file. `null` without a function. Trivial bodies such as `async () => {}` share a hash. | | `suiteId` | string or null | The nearest describe. `null` outside any describe. | | `suitePath` | string\[] | Titles of the describes around the test, outermost first. | | `relativeFilePath` | string | The file, from the project root. | | `fileName` | string | The file name without folders. | | `lineStart`, `lineEnd` | number | First and last line of the test call. | | `columnStart`, `columnEnd` | number | Where the test call starts, and the column just after its last character. | | `declaredAt` | object or null | `{ relativeFilePath, lineStart, columnStart }` of the test in a helper imported from another file, when that helper declares it. The record's own location is then the call in this file that registers the test. `null` for tests written in the file. | | `title` | string | The title as the runner shows it, tags included. | | `titleWithoutTags` | string | Playwright: the title without its `@tag` words. Vitest: the title. | | `hasDynamicTitle` | boolean | The title could not be worked out; `title` holds its source text. | | `fullName` | string | `suitePath` and `title` joined with `>`. | | `isTypeTest` | boolean | Declared in a Vitest type test file, `*.test-d.ts` or `*.spec-d.ts`. | | `state` | string | `active`, `skip`, `fixme`, `todo`, `expectedToFail` or `notLoaded`. | | `isActive`, `isSkipped`, `isFixme`, `isTodo`, `isExpectedToFail`, `isNotLoaded` | boolean | Exactly one is true, matching `state`. | | `isOnly` | boolean | The test or a describe around it uses `.only`. | | `isConditional` | boolean | The state applies only when a condition holds. | | `condition` | string or null | The condition's source text, on one line and cut at 200 characters. | | `reason` | string or null | The reason given with the state. | | `hasDynamicReason` | boolean | The reason could not be worked out to a string; `reason` holds its source text. | | `stateSource` | string or null | Where the state is set: `test`, `suite` or `file`. `null` for active tests. | | `stateLine` | number or null | The line that sets the state. `null` for active tests. | | `isParallel`, `isSerial` | boolean | The mode set in the file. Both `false` means the framework's default. | | `tags` | string\[] | Tags of the test and every describe around it, outermost first, without duplicates. | | `annotations` | object\[] | `{ type, description, lineStart }` from Playwright's `annotation` option and `test.info().annotations.push()`. | | `comments` | string\[] | The comment lines directly above the test, without lint, compiler, formatter and coverage directives. By default only for tests that are not active. | | `locks` | string\[] | Playwright `lock` names on the test and its describes (1.63), and from `test.describe.configure()` in the file and those describes (1.64). | | `meta` | object or null | Vitest `meta`, merged from the describes and the test, literal values only. | | `hooks` | object | Every hook that runs around the test: those at the top of the file and in each describe around it, by kind. See [files](#files). | | `timeout` | number or null | In milliseconds. See [run settings](#run-settings). `null` when the file sets none, so the config's timeout applies. | | `isSlow`, `slowCondition` | boolean, string or null | Playwright `test.slow()` applies, and its condition's source text; `null` when it always applies. | | `retries` | number or null | How many times the runner retries the test when it fails. `null` when the file sets none. | | `retryDelay`, `retryCondition` | number or null, string or null | Vitest's `retry` object (4.1): the milliseconds between retries, and the source text of its `condition`. | | `repeats` | number or null | Vitest `repeats`: how many times the runner runs the test again after the first run. | | `fixtures` | string\[] | The names the test function takes from its context, then those of the `beforeEach`, `afterEach` and `aroundEach` hooks around it, once each. Vitest's own context keys, such as `expect`, `task` and `annotate`, are left out. See [what the body does](#what-the-body-does). | | `expectCount` | number | The `expect()`, `expect.soft()` and `expect.poll()` calls written in the body. | | `expectedAssertions`, `expectsAssertions` | number or null, boolean | Vitest's `expect.assertions(n)` and `expect.hasAssertions()` in the body. | | `hasEmptyBody` | boolean | The test function has no statements, comments aside. | | `steps` | object\[] | `{ title, hasDynamicTitle, lineStart, depth, isSkipped }` for each Playwright `test.step()` and `test.step.skip()` in the body, in source order; `depth` is 0 for a step directly in the test. | | `isBenchmark` | boolean | A Vitest 5 benchmark: in a file such as `*.bench.ts`, or using the `bench` fixture. Plain `vitest` doesn't run it; `vitest bench` does. | | `isParameterized` | boolean | Declared with `.each` or `.for`, or inside a `describe.each` or `describe.for`, so the runner repeats it. | | `caseCount` | number or null | How many times the runner repeats the test: its own cases times those of each describe table around it. `null` when a table is not written out in the file, or the test is not parameterized. | | `isInLoop` | boolean | Declared inside a loop, `forEach` or `map`. | | `isInCondition` | boolean | Declared inside `if`, `?:`, `&&`, `\|\|`, `??`, `switch` or a `catch` block. | | `isInFunction` | boolean | Declared inside a plain function. A function called from the file or a describe has its tests recorded at each call; one the scan can't place gets a `test-in-function` warning. | ::: details TypeScript ```ts /** A test, with the effective values the runner applies, including those inherited from its describes. */ export interface TestRecord { /** Derived from the file path, the describe titles, the title and its position among tests with the same title. */ id: Hash; /** Hash of the test body, ignoring whitespace and comments; null when the test has no body. */ bodyHash: Hash | null; /** The nearest describe; null outside any describe. */ suiteId: Hash | null; /** Titles of the describes around the test, outermost first. */ suitePath: string[]; /** Path from the project root, with `/` separators. */ relativeFilePath: string; fileName: string; lineStart: Position; lineEnd: Position; /** In UTF-16 code units. */ columnStart: Position; /** In UTF-16 code units. */ columnEnd: Position; /** * Where a helper imported from another file writes the test; the record's location is then the * call that registers it. Null for code written in the scanned file. */ declaredAt: DeclaredAt | null; /** As the runner shows it, tags included. */ title: string; /** `title` with `@tag` words removed and spaces tidied. */ titleWithoutTags: string; /** The title could not be resolved to a fixed string, so `title` holds its source text. */ hasDynamicTitle: boolean; /** `suitePath` and `title` joined with " > ". */ fullName: string; /** Declared in a Vitest type test file (`*.test-d.ts`, `*.spec-d.ts`). */ isTypeTest: boolean; state: TestState; isActive: boolean; isSkipped: boolean; isFixme: boolean; isTodo: boolean; isExpectedToFail: boolean; /** The runner refuses to load the file; a diagnostic at `stateLine` says why. */ isNotLoaded: boolean; /** The test or a describe around it uses `.only`. */ isOnly: boolean; /** The state applies only when a condition holds, as with `test.skip(condition)` or `skipIf`. */ isConditional: boolean; /** Source text of the condition, shortened when long; null when unconditional. */ condition: string | null; reason: string | null; /** The reason could not be worked out to a string, so `reason` holds its source text. */ hasDynamicReason: boolean; /** null for active tests. */ stateSource: StateSource | null; /** The line that sets the state; null for active tests. */ stateLine: Position | null; /** Both `isParallel` and `isSerial` false means the framework's default mode. */ isParallel: boolean; isSerial: boolean; /** From the test and every describe around it, outermost first, without duplicates. */ tags: string[]; annotations: Annotation[]; /** Comment lines directly above the test. By default only filled for tests that are not active. */ comments: string[]; /** From the test, its describes and the `test.describe.configure()` calls around it. Playwright only. */ locks: string[]; /** Merged from the describes and the test. Vitest only; literal values only. */ meta: Record | null; /** Every hook that runs around the test: those of the file and of each describe around it. */ hooks: HookCounts; /** * In milliseconds: the last `test.setTimeout()` in the body or a `beforeEach`, otherwise the nearest describe or file * that sets one. Null when the file sets none, so the config's timeout applies. */ timeout: Count | null; /** Playwright `test.slow()` applies: the runner triples the timeout. */ isSlow: boolean; /** Source text of the condition of the `test.slow()` that applies; null when unconditional or not slow. */ slowCondition: string | null; /** How many times the runner retries the test when it fails; null when the file sets none. */ retries: Count | null; /** Vitest only: milliseconds between retries, from the `retry` object (4.1). */ retryDelay: Count | null; /** Vitest only: source text of the `retry` object's `condition`, a RegExp or a function. */ retryCondition: string | null; /** Vitest only: how many times the runner runs the test again after the first run. */ repeats: Count | null; /** * Names the test function takes from its context, then those of the `beforeEach`, `afterEach` and `aroundEach` hooks * around it, once each. Vitest's own context keys, such as `expect` and `task`, are left out. */ fixtures: string[]; /** Calls of `expect()`, `expect.soft()` and `expect.poll()` written in the body, each counted once. */ expectCount: Count; /** Vitest `expect.assertions(n)` in the body. */ expectedAssertions: Count | null; /** Vitest `expect.hasAssertions()` in the body. */ expectsAssertions: boolean; /** The test function has no statements, comments aside. */ hasEmptyBody: boolean; /** Playwright `test.step()` and `test.step.skip()` calls in the body, in source order. */ steps: Step[]; /** Vitest 5 benchmark: in a benchmark file, or using the `bench` fixture. Plain `vitest` doesn't run it. */ isBenchmark: boolean; /** Declared with `.each` or `.for`, or inside a describe declared with them, so the runner repeats it. */ isParameterized: boolean; /** * How many times the runner repeats the test: the cases of its own table times those of each describe table around * it. Null when a table is not written out in the file, or the test is not parameterized. */ caseCount: Count | null; /** Declared inside a loop, `forEach` or `map`. */ isInLoop: boolean; /** Declared inside `if`, `?:`, `&&`, `switch` or a `catch` block. */ isInCondition: boolean; /** Declared inside a plain function. */ isInFunction: boolean; } /** * The state the runner applies to a test, after describes and file-level calls are taken into account. `notLoaded`: * the runner refuses to load the file, so none of its tests run. */ export type TestState = 'active' | 'skip' | 'fixme' | 'todo' | 'expectedToFail' | 'notLoaded'; /** Where a test's state is set: on the test, on a describe around it, or at file level. */ export type StateSource = 'test' | 'suite' | 'file'; export interface Annotation { type: string; description: string | null; lineStart: Position; } export interface Step { /** The title, or its source text when it can't be worked out. */ title: string; hasDynamicTitle: boolean; lineStart: Position; /** 0 for a step directly in the test, 1 for a step inside it, and so on. */ depth: Count; /** Declared with `test.step.skip()`. */ isSkipped: boolean; } export type MetaValue = string | number | boolean | null | MetaValue[] | { [key: string]: MetaValue }; ``` ::: ### How the state is chosen The state is the one the runner applies. The test's own skip, fixme or todo comes first: `test.fixme()` in a skipped describe stays `fixme`. Then the describes around it from the innermost out, then the file. A state that always applies wins over one that only applies under a condition, and an unconditional skip, fixme or todo around a test wins over the test's own expected failure, as neither runner runs that test. For the same reason it wins over an unconditional skip in a test or hook body: in a fixme or todo describe, the test keeps that state. `stateSource` says where the chosen state was set. In Vitest, `.only` wins over `.skip`, which wins over `.todo`, within one declaration: `test.only.skip` is active. A test's options win over its chain, so `test.skip('title', { skip: false }, fn)` runs, while a describe's chain wins over its options, and so does a test's for its mode. A describe passes its `skip` option down to the tests and describes inside, where it wins over todo: `test.todo()` in `describe('title', { skip: true }, fn)` is skipped. It does so also when the describe's own `.only` keeps the describe from being skipped. A `test.skip()`, `test.fixme()`, `test.fail()` or `test.describe.configure()` call inside a plain function in the file applies where the function is called: to a test that calls it in its body, to the tests a hook covers when a hook calls it, or to a describe when it is called directly in the describe's body. The conditions around the call and inside the function are joined, and `stateLine` is the line of the call inside the function. Calls through other functions in the file are followed too. A function that is exported, or that nothing in the file calls, changes no record and gets a `state-in-helper` warning. When a file has a problem that makes the runner refuse to load it, such as a Playwright test without a body or a Vitest call to an API that the version removed, every test in it is `notLoaded`: `stateSource` is `file`, `stateLine` is the line of the first such problem, and `condition` and `reason` are `null`. The [diagnostic](/reference/diagnostics#files-the-runner-refuses-to-load) says why. Everything else about the tests stays as written, so fixing the file brings their states back. A refused call that declares a test or describe is listed without the part that fails, so its tests are `notLoaded` too. ### Run settings The run settings are those written in the file. Each key comes from the nearest place that sets it: 1. Calls in the test body, the last one winning: Playwright's `test.setTimeout()` and `test.slow()`. 2. The same calls in a `beforeEach` around the test, which runs before the body. In `beforeAll` and `afterAll` they set the hook's own timeout, and an `afterEach` runs after the body, so those are left out. 3. The test's own options: Vitest's `timeout`, `retry` and `repeats`, or a timeout after the body, as in `test(title, fn, 30_000)`. 4. Each describe around the test, from the innermost out, and then the file: Playwright's `test.describe.configure({ timeout, retries })`, `test.setTimeout()` and `test.slow()` written in the describe, and the options Vitest passes down from `describe()`. A Vitest `retry` replaces the whole `retry` around it, so a test with `retry: 1` in a describe with a `delay` has no `retryDelay`. Vitest reads the `retry` object from 4.1 and ignores it before, and so does the scan. Playwright triples the timeout of a slow test once, when `test.slow()` runs; a `test.setTimeout()` after it replaces the tripled value. The record keeps `timeout` as written and `isSlow` beside it. `test.slow()` under a condition, or inside `if`, gives `slowCondition`. A timeout set only under a condition, or one the scan can't work out, such as a constant imported from another file, is left out with an `unresolved-option` warning, and the value around it applies. In the web app, the live price stream retries its tests, since the demo feed drops connections now and then: ```ts test.describe('live price stream', () => { // The demo feed drops connections now and then. test.describe.configure({ retries: 2 }); // ... }); ``` ```json { "fullName": "live price stream > updates prices without a reload", "timeout": null, "isSlow": false, "retries": 2, "retryDelay": null, "repeats": null } ``` ### What the body does These keys come from the test function itself, not from the functions it calls: an `expect()` or a `test.step()` in a page object or a helper isn't counted. Each call is counted once as written, also inside a loop. Vitest's `annotate(message, type?)` from the test context (3.2) adds an annotation, of type `notice` when no type is given. In the web app, the order lifecycle enters and confirms the order in two steps: ```ts test('places a limit order for 100 NVDA', async ({ orderTicket, blotter }) => { await test.step('enter the order', async () => { await orderTicket.open('NVDA'); await orderTicket.buy({ quantity: 100, limitPrice: 118 }); }); await test.step('confirm it', async () => { await orderTicket.confirm(); }); await expect(blotter.status('NVDA')).toHaveText('Working'); }); ``` ```json { "fullName": "order lifecycle > places a limit order for 100 NVDA", "fixtures": ["orderTicket", "blotter"], "expectCount": 1, "hasEmptyBody": false, "steps": [ { "title": "enter the order", "hasDynamicTitle": false, "lineStart": 8, "depth": 0, "isSkipped": false }, { "title": "confirm it", "hasDynamicTitle": false, "lineStart": 12, "depth": 0, "isSkipped": false } ] } ``` ## Suites A suite record describes what is written on the describe itself. Values a test inherits from it are in the test record. | Key | Type | Meaning | | ------------------------------------------------------------------------------------------------ | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `id`, `parentId` | string, string or null | The describe and the one around it. | | `bodyHash` | string or null | The describe's own code: hooks, `test.use()`, `describe.configure()` and helpers, without the tests and describes inside it, hashed as for tests. `null` when nothing is left. | | `relativeFilePath`, `fileName`, `lineStart`, `lineEnd`, `columnStart`, `columnEnd`, `declaredAt` | | As for tests. | | `title`, `titleWithoutTags`, `hasDynamicTitle` | | As for tests. An anonymous Playwright describe has an empty title and is left out of paths. | | `path` | string\[] | The titles of this describe and those around it. | | `isSkipped`, `isFixme`, `isTodo`, `isExpectedToFail`, `isOnly` | boolean | Set on this describe. `isExpectedToFail` is a Playwright `test.fail()` in the describe's body. | | `isConditional`, `condition`, `reason`, `hasDynamicReason` | | As for tests. | | `isParallel`, `isSerial`, `isShuffled` | boolean | The mode set on this describe. | | `isParameterized` | boolean | `describe.each` or `describe.for`. | | `tags`, `annotations`, `locks`, `meta` | | Written on this describe. | | `hooks` | object | The hooks registered directly in this describe, by kind. | | `timeout`, `isSlow`, `slowCondition`, `retries`, `retryDelay`, `retryCondition`, `repeats` | | Set on this describe: by its options, or by `describe.configure()`, `test.setTimeout()` and `test.slow()` in its body. | ::: details TypeScript ```ts /** A describe block, with what is written on the describe itself. Inherited values are on the tests. */ export interface SuiteRecord { id: Hash; /** The describe around this one; null at the top level. */ parentId: Hash | null; /** * Hash of the describe's own code (hooks, `test.use()`, `describe.configure()`, helpers), ignoring whitespace, * comments, and nested tests and describes; null when nothing is left. */ bodyHash: Hash | null; /** Path from the project root, with `/` separators. */ relativeFilePath: string; fileName: string; lineStart: Position; lineEnd: Position; /** In UTF-16 code units. */ columnStart: Position; /** In UTF-16 code units. */ columnEnd: Position; /** * Where a helper imported from another file writes the describe; the record's location is then the * call that registers it. Null for code written in the scanned file. */ declaredAt: DeclaredAt | null; /** As the runner shows it, tags included. */ title: string; /** `title` with `@tag` words removed and spaces tidied. */ titleWithoutTags: string; /** The title could not be resolved to a fixed string, so `title` holds its source text. */ hasDynamicTitle: boolean; /** Titles of the describes around this one and of this one, outermost first. */ path: string[]; isSkipped: boolean; /** Playwright only. */ isFixme: boolean; /** Vitest only. */ isTodo: boolean; /** Tests in the describe are expected to fail, as with `test.fail()` in its body. Playwright only. */ isExpectedToFail: boolean; isOnly: boolean; /** The state applies only when a condition holds. */ isConditional: boolean; /** Source text of the condition, shortened when long; null when unconditional. */ condition: string | null; reason: string | null; /** The reason could not be worked out to a string, so `reason` holds its source text. */ hasDynamicReason: boolean; isParallel: boolean; /** Playwright only. */ isSerial: boolean; /** Vitest only. */ isShuffled: boolean; /** Declared with `.each` or `.for`. */ isParameterized: boolean; tags: string[]; annotations: Annotation[]; /** Playwright only. */ locks: string[]; /** Vitest only; literal values only. */ meta: Record | null; /** Hooks registered directly in the describe. */ hooks: HookCounts; /** In milliseconds; null when nothing here sets it. */ timeout: Count | null; /** Playwright `test.slow()` written in the describe, which triples the timeout of its tests. */ isSlow: boolean; /** Source text of the `test.slow()` condition; null when unconditional or not slow. */ slowCondition: string | null; retries: Count | null; /** Vitest only: milliseconds between retries, from the `retry` object (4.1). */ retryDelay: Count | null; /** Vitest only: source text of the `retry` object's `condition`, a RegExp or a function. */ retryCondition: string | null; /** Vitest only: how many times the runner runs each test again after the first run. */ repeats: Count | null; } ``` ::: ## Files A file record is written for every scanned file, also one that has no tests, could not be read or failed to parse. | Key | Type | Meaning | | ------------------------------ | ------ | ------------------------------------------------------------------------------------ | | `relativeFilePath`, `fileName` | string | As for tests. | | `hooks` | object | The hooks at the top level of the file, which run around every test in it. | | `suiteCount`, `testCount` | number | The describes and tests in the file, all of them also when a filter keeps only some. | `hooks` counts each kind: `beforeAll`, `beforeEach`, `afterEach` and `afterAll`, and Vitest's `aroundEach` and `aroundAll` (4.1). A hook in a helper function with a name counts where the helper is called, and not at all when it is never called; one in a function without a name, such as a callback, counts where it is written. Vitest's `onTestFailed` and `onTestFinished` run inside a test, so they are not counted. A test's `hooks` adds up those of its file and of each describe around it. In the web app, the price alerts open their page in a `beforeEach`: ```json { "id": "006bcf2289191aef", "fullName": "price alerts > notifies when the price crosses above the alert", "hooks": { "beforeAll": 0, "beforeEach": 1, "afterEach": 0, "afterAll": 0, "aroundEach": 0, "aroundAll": 0 } } ``` ::: details TypeScript ```ts /** A scanned file, also one that has no tests or could not be read. */ export interface FileRecord { /** Path from the project root, with `/` separators. */ relativeFilePath: string; fileName: string; /** Hooks at the top level of the file, which run around every test in it. */ hooks: HookCounts; suiteCount: Count; testCount: Count; } /** How many hooks of each kind are registered, including by helper functions called there. */ export interface HookCounts { beforeAll: Count; beforeEach: Count; afterEach: Count; afterAll: Count; /** Vitest only. */ aroundEach: Count; /** Vitest only. */ aroundAll: Count; } ``` ::: ## Summary Each count uses the same definition as the matching key: `skipCount` counts tests with `isSkipped: true`. `fileCount`, `suiteCount`, `testCount`, `activeCount`, `skipCount`, `fixmeCount`, `todoCount`, `expectedToFailCount`, `notLoadedCount`, `onlyCount`, `conditionalCount`, `parallelCount`, `serialCount`, `parameterizedCount`, `inLoopCount`, `inConditionCount`, `inFunctionCount`, `typeTestCount`, `benchmarkCount`, `diagnosticCount`. `tagCounts` maps each tag to the number of tests with it. `annotationCounts` maps each annotation type to the number of tests with at least one annotation of that type. Both are sorted by name. ::: details TypeScript ```ts /** Counts over the whole scan. Each test count uses the same definition as the matching test key. */ export interface Summary { /** Test files scanned. */ fileCount: Count; suiteCount: Count; testCount: Count; activeCount: Count; skipCount: Count; fixmeCount: Count; todoCount: Count; expectedToFailCount: Count; notLoadedCount: Count; onlyCount: Count; conditionalCount: Count; parallelCount: Count; serialCount: Count; parameterizedCount: Count; inLoopCount: Count; inConditionCount: Count; inFunctionCount: Count; typeTestCount: Count; benchmarkCount: Count; diagnosticCount: Count; /** Tests per tag. */ tagCounts: Record; /** Tests per annotation type. */ annotationCounts: Record; } ``` ::: ## Diagnostics | Key | Type | Meaning | | -------------------------- | -------------- | ------------------------------------------------------------------------------------ | | `code` | string | One of the [diagnostic codes](/reference/diagnostics). | | `level` | string | `error` when tests may be missing; `warning` when a value is missing or approximate. | | `message` | string | One sentence about the problem. | | `relativeFilePath` | string | The file. | | `lineStart`, `columnStart` | number or null | Where the problem is. `null` for the whole file. | | `testId` | string or null | The test it concerns. | | `source` | string or null | The code involved, cut at 200 characters. | ::: details TypeScript ```ts export interface Diagnostic { code: DiagnosticCode; level: DiagnosticLevel; /** One sentence describing the problem. */ message: string; relativeFilePath: string; /** null for problems with the whole file. */ lineStart: Position | null; /** null for problems with the whole file. */ columnStart: Position | null; /** The affected test; null when none. */ testId: Hash | null; /** Source text involved, shortened when long; null when none. */ source: string | null; } /** `error`: tests may be missing from the output. `warning`: a value is missing or approximate. */ export type DiagnosticLevel = 'error' | 'warning'; export type DiagnosticCode = | 'unresolved-test-import' | 'parse-error' | 'read-error' | 'framework-mismatch' | 'unresolved-tag' | 'unresolved-annotation' | 'unresolved-cases' | 'test-in-loop' | 'dynamic-title' | 'duplicate-title' | 'duplicate-id' | 'internal-error' | 'invalid-chain' | 'skip-without-body' | 'test-without-body' | 'describe-not-called' | 'async-describe' | 'nested-test' | 'tag-without-at' | 'removed-api' | 'only-in-skipped-describe' | 'config-may-change-state' | 'symlink-ignored' | 'invalid-import' | 'globals-not-enabled' | 'unknown-framework-version' | 'unsupported-framework-version' | 'local-test-object' | 'state-in-helper' | 'unresolved-option' | 'dynamic-reason' | 'no-files-matched' | 'unsupported-file' | 'approximate-framework-version' | 'test-in-function' | 'empty-title' | 'bench-outside-benchmark-file'; ``` ::: ## JSON Lines With `--format jsonl` the same records are written one per line, each with a `type` key added in front: first an `inventory` line with the keys of the top level, then a `file`, `suite`, `test` or `diagnostic` line for each record, in the order of the JSON output, and a `summary` line last. Without its `type`, a line has exactly the keys of the record described above, and filters and `--fields` work the same way. ```ts /** * One line of the JSON Lines output (`--format jsonl`): the inventory with each record on its own line, marked by * `type`. The `inventory` line comes first, then the files, suites, tests and diagnostics in the order of the JSON * output, and the `summary` line last. */ export type InventoryLine = | ({ type: 'inventory' } & Pick) | ({ type: 'file' } & FileRecord) | ({ type: 'suite' } & SuiteRecord) | ({ type: 'test' } & TestRecord) | ({ type: 'diagnostic' } & Diagnostic) | ({ type: 'summary' } & Summary); ``` ## Versioning * The package follows semantic versioning. Before 1.0, a breaking change comes in a minor release. * Adding a key, a diagnostic code or a value to an open set keeps `schemaVersion`. Read the output in a way that ignores keys you don't know; the published schema allows them. * These sets are open, so a reader should expect values it doesn't know yet: `framework`, `state`, `stateSource` and the diagnostic `code`. The published schema lists the known values and accepts any string. * Removing or renaming a key, or changing its type or meaning, raises `schemaVersion` by one and is a breaking release. * Each `schemaVersion` has its own schema file, shipped in the package as `test-inventory/schema/v1.json` and published on this site. * A key that is going away is marked deprecated in these pages and in the schema for at least one minor release first. --- --- url: /test-inventory/reference/syntax.md description: The Playwright and Vitest imports, calls and modifiers a scan reads. --- # Supported syntax ## Playwright Playwright 1.42 and newer. ### Where test comes from | Form | Example | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Named import, also renamed | `import { test } from '@playwright/test'`, `import { test as base } from '@playwright/test'` | | Default import | `import pwTest from '@playwright/test'` | | Namespace import | `import * as pw from '@playwright/test'` then `pw.test(...)` | | CommonJS | `const { test } = require('@playwright/test')`, `require('@playwright/test').test(...)` | | Other packages | `playwright/test`, `@playwright/experimental-ct-react` and the other component testing packages, and `playwright-bdd`, whose generated files import `test` from it | | Derived | `base.extend({...})`, chained `.extend().extend()`, `mergeTests(a, b)` | | Local fixtures file | `import { test } from '../fixtures'`, or a path alias, `#` import or workspace package, followed through re-exports, `extend` chains and functions that build it, across files | | Returned by a function | `brokerTest('live')('sends an order', ...)`, also kept in a variable, where `brokerTest` is declared in the file or a local file and returns a test object or describe function | ### Calls | Call | Read as | | --------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | `test(title, body)`, `test(title, details, body)` | A test. The body may be a function passed by name. | | `test.only`, `test.skip`, `test.fixme`, `test.fail` with a title and body | A test with that state. | | `test.fail.only` | Expected to fail, with `isOnly`. Playwright 1.49 and newer. | | `test.skip()`, `test.skip(condition, description)`, `test.skip(({ browserName }) => condition, description)`; the same for `fixme` and `fail` | At file or describe level, a state for every test in the scope. In a test body or hook, a state for that test or the tests the hook runs for. | | `test.describe(title, callback)`, with details, or anonymous `test.describe(callback)` | A describe. | | `test.describe.only`, `.skip`, `.fixme`, `.serial`, `.serial.only`, `.parallel`, `.parallel.only` | A describe with that state or mode. | | `test.describe.configure({ mode, lock })` | The mode of the describe or file. `lock`, from Playwright 1.64, adds locks to every test in it. | | `test.info().annotations.push({ type, description })` | An annotation of the test. | | `test.use`, `test.slow`, `test.setTimeout`, `test.step`, `test.step.skip` (1.50), `test.expect`, `test.info`, `test.abort` (1.60), hooks | Known; not tests. | | Any other chain, such as `test.only.skip` | An `invalid-chain` error: Playwright fails to load the file and runs no tests at all. | ### Details `{ tag, annotation, lock }`: `tag` is a string or an array of strings starting with `@`; `annotation` an object or an array of `{ type, description }`; `lock` a string or an array of strings, from Playwright 1.63. Tags are also read from `@words` in titles. ## Vitest Vitest 3.0 and newer. ### Where test comes from | Form | Example | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Named import, also renamed | `import { test, it, describe, suite } from 'vitest'` | | Namespace import | `import * as vt from 'vitest'` then `vt.test(...)` | | Globals | `describe`, `it`, `test` and `suite` used without an import or declaration. A warning says when no config turns globals on. | | In-source tests | `const { it, describe } = import.meta.vitest` | | Describe callback | `describe('fees', (test) => { test(...) })`: Vitest passes its `test` to the callback | | Derived | `test.extend({...})`, `test.extend('name', value)`, chained; `test.override()` and `test.scoped()` are known | | Local fixtures file | `import { test } from './context'`, or a path alias, `#` import or workspace package, followed through re-exports, `extend` chains and functions that build it, across files | | Returned by a function | `onNode(22)('reads the order book', ...)`, also kept in a variable, where `onNode` is declared in the file or a local file and returns a test or describe function, as in `it.skip` | ### Calls | Call | Read as | | ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | `test(title, fn)`, `test(title, options, fn)`, `test(title, fn, timeout)` | A test. Without a function, a todo. | | `test(title, fn, options)` | Accepted before Vitest 4; a `removed-api` error from Vitest 4. | | `.concurrent`, `.only`, `.skip`, `.todo`, `.fails`, in any order and combination | `.only` wins over `.skip`, which wins over `.todo`. | | `.each(table)`, `.for(table)`, also as tagged templates | One parameterized record, with `caseCount` when the table is in the file. | | `.skipIf(condition)`, `.runIf(condition)` | A conditional skip. `runIf(x)` is stored as the condition `!(x)`. | | `describe` with `.concurrent`, `.shuffle`, `.skip`, `.only`, `.todo`, `.each`, `.for`, `.skipIf`, `.runIf` | A describe. `describe(title)` without a function is a todo. | | `describe(fn)` without a title | A `describe-not-called` error: Vitest never calls the function. | | `context.skip()`, `skip(note)`, `skip(condition, note)` in a test, `beforeEach` or `aroundEach` | A state for that test, or for the tests in the describe. | | Options `skip`, `only`, `todo`, `fails`, `concurrent`, `sequential`, `shuffle`, `tags` (4.1), `meta` | Read when written in the file; a test's own states win over its chain. | | `// @module-tag name` | A tag on every test in the file, from Vitest 4.1. | | `.sequential` | The default mode before Vitest 5; a `removed-api` error from Vitest 5. | | `bench(...)`, with `bench` imported from `vitest` | A `removed-api` error from Vitest 5. Benchmarks are never tests. | | `vi.mock`, `vi.unmock`, `vi.hoisted` below the top level of the file | A `removed-api` error from Vitest 5, except in files with in-source tests. | | Any other chain, such as `test.fixme` | An `invalid-chain` error. | ## In both * Tests in loops, conditions and plain functions are listed with `isInLoop`, `isInCondition` and `isInFunction`. * A test or describe declared inside a test body or a hook is not listed, with a `nested-test` warning. * TypeScript wrappers such as `test!.skip(...)` and `(test as any).only(...)` are read through. * `.js`, `.mjs` and `.cjs` files may contain JSX. --- --- url: /test-inventory/reference/diagnostics.md description: Every diagnostic a scan reports, its level and when it is given. --- # Diagnostics A diagnostic is part of the output, not a failure: the scan always finishes and exits with 0. Its `level` is `error` when tests may be missing from the output, and `warning` when a value is missing or approximate. ## Files and imports | Code | Level | When | | ------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `read-error` | error | A test file or a folder could not be read, or a test file is a link to a file that does not exist. | | `parse-error` | error | The file has a syntax error. When the parser recovers, the tests it read are kept. | | `internal-error` | error | Reading the file failed unexpectedly. Please report it with the file if you can. The scan goes on with the next file. | | `framework-mismatch` | error | The file imports another framework, such as Vitest in a Playwright scan or `@jest/globals` in a Vitest scan. | | `unresolved-test-import` | error | A call shaped like a test, outside any test or hook body, uses a name imported from a module that could not be followed. Files are followed through re-exports and `extend` chains; the message says where the chain ends: a missing or broken file, an installed package, or a file that doesn't export a test object. The name reads like a test function, as `test` or `checkoutTest` do, or is followed by a member such as `.describe`. | | `invalid-import` | error | Vitest imported in a way it does not support: a default import, or `require('vitest')`. | | `no-files-matched` | warning | No test file matches the patterns. When a pattern names a folder, the message suggests a pattern for the test files in it. The command line also says so on standard error. | | `unsupported-file` | warning | A file that matches the patterns is not JavaScript or TypeScript, so it is not read. | | `symlink-ignored` | warning | A Playwright test file is reached through a symbolic link, which Playwright does not follow. | | `local-test-object` | warning | A variable or function named like a test object makes one in a way the scan cannot follow, as in `const it = await createTester()` or `liveBrokerTest(options)('title', fn)`. Calls on it shaped like tests are read as tests. | | `globals-not-enabled` | warning | A Vitest file uses `describe`, `it` or `test` without importing them, and no config with `globals: true` was found. | | `unknown-framework-version` | warning | The framework version could not be read from the project; the newest rules apply. | | `approximate-framework-version` | warning | Only a version range was found, such as `^4.0.0` in `package.json`; the rules of the newest release in that major apply. | | `unsupported-framework-version` | warning | The project uses a version older than Playwright 1.42 or Vitest 3.0. | ## Declarations | Code | Level | When | | -------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `invalid-chain` | error | A chain the framework does not have, such as `test.only.skip` in Playwright, or one that needs a newer version. | | `test-without-body` | error | Playwright `test('title')` without a function, which makes Playwright fail to load the file and run no tests at all. | | `describe-not-called` | error | Vitest `describe(fn)` without a title. Vitest never calls the function. | | `async-describe` | error | A Playwright describe callback that awaits. Playwright does not wait for it: tests declared after the first `await` end up in the describe around it, or are not collected at all. | | `tag-without-at` | error | A Playwright tag that does not start with `@`, which makes Playwright fail to load the file and run no tests at all. | | `removed-api` | error | An API the framework version removed: Vitest 5 `sequential` and `bench`, `vi.mock`, `vi.unmock` or `vi.hoisted` below the top level of the file from Vitest 5, or options as the third argument from Vitest 4. Vitest fails the file. | | `duplicate-id` | error | Two records in a file have the same id. Ids include a position among equal titles, so only a hash collision causes this. | | `duplicate-title` | warning, error in Playwright | Playwright: two tests in the file have the same title path, without anonymous describes, or a test in a loop has the same title path on each pass, so Playwright fails to load the file and runs no tests at all. Vitest: two tests in the same describe have the same title. | | `state-in-helper` | warning | A `test.skip()`, `test.fixme()`, `test.fail()` or `describe.configure()` is in a function that the file exports or never calls, so it is not applied to any record. Also given when a call in a test or hook reaches helpers more than 10,000 times, counting the calls inside them; the states set past that are not applied. | | `skip-without-body` | warning | Playwright `test.skip('title')` without a function: Playwright reads the title as a condition and skips every test in the scope. | | `nested-test` | warning | A test or describe declared inside a test body or hook. The runner does not collect it, so it is not listed. | | `only-in-skipped-describe` | warning | A test with `.only` inside a skipped describe. It is still skipped. | ## Values | Code | Level | When | | ------------------------------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `dynamic-title` | warning | A title could not be worked out, for example a template with a loop variable. The record uses its source text. | | `dynamic-reason` | warning | The reason of a skip, fixme or fail could not be worked out to a string. The record uses its source text, with `hasDynamicReason: true`. | | `unresolved-tag` | warning | A tag comes from a value that could not be worked out, usually an import. The tag is missing from the record. | | `unresolved-annotation` | warning | The same for an annotation. | | `unresolved-option` | warning | An options object, a spread in one, a lock, a Vitest flag such as `concurrent` or `only`, a `meta` value, a timeout, retry or repeat count, or the count of `expect.assertions()` could not be worked out, or a timeout is set only under a condition. What it would set is missing from the record. | | `unresolved-cases` | warning | The cases of `.each` or `.for` could not be counted; `caseCount` is `null`. | | `test-in-loop` | warning | The test is declared in a loop. It is listed once, while the runner creates one test per iteration. | | `test-in-function` | warning | The test is declared in a function the scan can't place: nothing in the file calls it by name, the file only exports it, or it is called only from tests or hooks. The record stays where the test is written. Functions called from the file or a describe have their tests recorded at each call, up to 10,000 calls in a file, where this warning marks the call the scan stopped at; see [limits](/guide/limits#tests-in-helper-functions). | | `empty-title` | warning | A test or describe is titled with an empty string. Both runners allow it, but reports and filters can't tell it apart. | | `bench-outside-benchmark-file` | warning | A test uses Vitest 5's `bench` fixture outside a benchmark file such as `*.bench.ts`, where Vitest fails the test. It still has `isBenchmark`. | | `config-may-change-state` | warning | A Vitest file uses tags, and tags defined in the Vitest config can skip tests or change how they run. | ## Files the runner refuses to load Some problems make the runner fail while it loads a file: * Playwright: `test-without-body`, `tag-without-at`, `invalid-chain` and `duplicate-title`. Playwright then stops before it runs any test, in this file or any other. * Vitest: `invalid-chain` and `removed-api`. Vitest reports the file as failed and runs none of its tests; other files still run. Every test in such a file gets `state: "notLoaded"`, with `stateSource: "file"` and `stateLine` at the first of these problems. These tests are counted in `notLoadedCount`, not in `activeCount`. Everything else about them is kept, so the records show what the file declares once the problem is fixed. A refused call written like a declaration, such as `test.only.skip('title', fn)` or `describe.sequential('title', fn)` in Vitest 5, is listed too, as a plain test or describe without the part that fails, so its tests are `notLoaded` rather than missing. Tests in other files keep their states, even though Playwright runs none of them: the records describe the code, and the diagnostic points at the file to fix. Only code that runs while the file loads counts. An `invalid-chain` inside a test, hook or helper body, such as `test.step.skip()` before Playwright 1.50 or `test.slow()` in Vitest, runs later and fails only the test that reaches it, so the file loads. ## Example ```json { "code": "unresolved-tag", "level": "warning", "message": "Tag value TAGS.risk comes from '../../test-data/tags' and could not be worked out; the tag is missing.", "relativeFilePath": "tests/orders/limit-orders.spec.ts", "lineStart": 16, "columnStart": 48, "testId": "74099e2f6d05b9ee", "source": "TAGS.risk" } ``` --- --- url: /test-inventory/api/interfaces/InventoryFilter.md --- # Interface: InventoryFilter **`Inline`** ## Properties ### states? > `optional` **states?**: (`"active"` | `"skip"` | `"fixme"` | `"todo"` | `"expectedToFail"` | `"notLoaded"`)\[] Keeps tests in one of these states. *** ### tags? > `optional` **tags?**: `string`\[] Keeps tests with one of these tags, as the output writes them. *** ### grep? > `optional` **grep?**: `string` Keeps tests whose `fullName` matches this regular expression, ignoring case, as Playwright's `--grep` does. *** ### only? > `optional` **only?**: `boolean` Keeps tests that use `.only`, or are inside a describe that does. *** ### withoutReason? > `optional` **withoutReason?**: `boolean` Keeps tests that don't run and give no reason. Tests in a file the runner refuses are left out. --- --- url: /test-inventory/api/interfaces/ScanOptions.md --- # Interface: ScanOptions **`Inline`** ## Properties ### patterns > **patterns**: `string`\[] Glob patterns for the test files, relative to `root`. As in Playwright and Vitest, they match names that start with a dot too. *** ### framework > **framework**: `"playwright"` | `"vitest"` *** ### root? > `optional` **root?**: `string` The project root. Paths in the output are relative to it. Defaults to the current directory. *** ### ignore? > `optional` **ignore?**: `string`\[] Glob patterns to leave out, in addition to `node_modules` and `.git`. *** ### frameworkVersion? > `optional` **frameworkVersion?**: `string` The framework version whose rules apply, instead of the one read from the project. *** ### comments? > `optional` **comments?**: `"all"` | `"non-active"` | `"none"` Which tests get the comments above them. Defaults to `non-active`. --- --- url: /test-inventory/api/interfaces/ScanSourceOptions.md --- # Interface: ScanSourceOptions **`Inline`** ## Properties ### code > **code**: `string` *** ### relativeFilePath > **relativeFilePath**: `string` Path of the file from the project root, used in the output and in ids. *** ### framework > **framework**: `"playwright"` | `"vitest"` *** ### frameworkVersion? > `optional` **frameworkVersion?**: `string` | `null` The framework version whose rules apply. The newest rules apply when it is missing. *** ### comments? > `optional` **comments?**: `"all"` | `"non-active"` | `"none"` Which tests get the comments above them. Defaults to `non-active`. --- --- url: /test-inventory/api/interfaces/Inventory.md --- # Interface: Inventory The output of one scan. ## Properties ### schemaVersion > **schemaVersion**: `1` Version of this format. It changes only when a key is removed, renamed or changes meaning. *** ### tool > **tool**: `object` #### name > **name**: `"test-inventory"` #### version > **version**: `string` Version of test-inventory that produced the output. *** ### framework > **framework**: `"playwright"` | `"vitest"` The framework the scan was run for. *** ### frameworkVersion > **frameworkVersion**: `string` | `null` Read from the project, or given with `--framework-version`; null when unknown. *** ### summary > **summary**: [`Summary`](Summary.md) *** ### files > **files**: [`FileRecord`](FileRecord.md)\[] One per scanned file, in the order the files were scanned. *** ### suites > **suites**: [`SuiteRecord`](SuiteRecord.md)\[] *** ### tests > **tests**: [`TestRecord`](TestRecord.md)\[] *** ### diagnostics > **diagnostics**: [`Diagnostic`](Diagnostic.md)\[] --- --- url: /test-inventory/api/interfaces/Tool.md --- # Interface: Tool **`Inline`** ## Properties ### name > **name**: `"test-inventory"` *** ### version > **version**: `string` Version of test-inventory that produced the output. --- --- url: /test-inventory/api/interfaces/Summary.md --- # Interface: Summary Counts over the whole scan. Each test count uses the same definition as the matching test key. ## Properties ### fileCount > **fileCount**: `number` Test files scanned. *** ### suiteCount > **suiteCount**: `number` *** ### testCount > **testCount**: `number` *** ### activeCount > **activeCount**: `number` *** ### skipCount > **skipCount**: `number` *** ### fixmeCount > **fixmeCount**: `number` *** ### todoCount > **todoCount**: `number` *** ### expectedToFailCount > **expectedToFailCount**: `number` *** ### notLoadedCount > **notLoadedCount**: `number` *** ### onlyCount > **onlyCount**: `number` *** ### conditionalCount > **conditionalCount**: `number` *** ### parallelCount > **parallelCount**: `number` *** ### serialCount > **serialCount**: `number` *** ### parameterizedCount > **parameterizedCount**: `number` *** ### inLoopCount > **inLoopCount**: `number` *** ### inConditionCount > **inConditionCount**: `number` *** ### inFunctionCount > **inFunctionCount**: `number` *** ### typeTestCount > **typeTestCount**: `number` *** ### benchmarkCount > **benchmarkCount**: `number` *** ### diagnosticCount > **diagnosticCount**: `number` *** ### tagCounts > **tagCounts**: `Record`<`string`, `Count`> Tests per tag. *** ### annotationCounts > **annotationCounts**: `Record`<`string`, `Count`> Tests per annotation type. --- --- url: /test-inventory/api/interfaces/FileRecord.md --- # Interface: FileRecord A scanned file, also one that has no tests or could not be read. ## Properties ### relativeFilePath > **relativeFilePath**: `string` Path from the project root, with `/` separators. *** ### fileName > **fileName**: `string` *** ### hooks > **hooks**: `object` Hooks at the top level of the file, which run around every test in it. #### beforeAll > **beforeAll**: `number` #### beforeEach > **beforeEach**: `number` #### afterEach > **afterEach**: `number` #### afterAll > **afterAll**: `number` #### aroundEach > **aroundEach**: `number` Vitest only. #### aroundAll > **aroundAll**: `number` Vitest only. *** ### suiteCount > **suiteCount**: `number` *** ### testCount > **testCount**: `number` --- --- url: /test-inventory/api/interfaces/HookCounts.md --- # Interface: HookCounts **`Inline`** How many hooks of each kind are registered, including by helper functions called there. ## Properties ### beforeAll > **beforeAll**: `number` *** ### beforeEach > **beforeEach**: `number` *** ### afterEach > **afterEach**: `number` *** ### afterAll > **afterAll**: `number` *** ### aroundEach > **aroundEach**: `number` Vitest only. *** ### aroundAll > **aroundAll**: `number` Vitest only. --- --- url: /test-inventory/api/interfaces/SuiteRecord.md --- # Interface: SuiteRecord A describe block, with what is written on the describe itself. Inherited values are on the tests. ## Properties ### id > **id**: `string` *** ### parentId > **parentId**: `string` | `null` The describe around this one; null at the top level. *** ### bodyHash > **bodyHash**: `string` | `null` Hash of the describe's own code (hooks, `test.use()`, `describe.configure()`, helpers), ignoring whitespace, comments, and nested tests and describes; null when nothing is left. *** ### relativeFilePath > **relativeFilePath**: `string` Path from the project root, with `/` separators. *** ### fileName > **fileName**: `string` *** ### lineStart > **lineStart**: `number` *** ### lineEnd > **lineEnd**: `number` *** ### columnStart > **columnStart**: `number` In UTF-16 code units. *** ### columnEnd > **columnEnd**: `number` In UTF-16 code units. *** ### declaredAt > **declaredAt**: { `relativeFilePath`: `string`; `lineStart`: `number`; `columnStart`: `number`; } | `null` Where a helper imported from another file writes the describe; the record's location is then the call that registers it. Null for code written in the scanned file. #### Union Members ##### Type Literal { `relativeFilePath`: `string`; `lineStart`: `number`; `columnStart`: `number`; } ##### relativeFilePath > **relativeFilePath**: `string` The helper's file, from the project root. ##### lineStart > **lineStart**: `number` ##### columnStart > **columnStart**: `number` In UTF-16 code units. *** `null` *** ### title > **title**: `string` As the runner shows it, tags included. *** ### titleWithoutTags > **titleWithoutTags**: `string` `title` with `@tag` words removed and spaces tidied. *** ### hasDynamicTitle > **hasDynamicTitle**: `boolean` The title could not be resolved to a fixed string, so `title` holds its source text. *** ### path > **path**: `string`\[] Titles of the describes around this one and of this one, outermost first. *** ### isSkipped > **isSkipped**: `boolean` *** ### isFixme > **isFixme**: `boolean` Playwright only. *** ### isTodo > **isTodo**: `boolean` Vitest only. *** ### isExpectedToFail > **isExpectedToFail**: `boolean` Tests in the describe are expected to fail, as with `test.fail()` in its body. Playwright only. *** ### isOnly > **isOnly**: `boolean` *** ### isConditional > **isConditional**: `boolean` The state applies only when a condition holds. *** ### condition > **condition**: `string` | `null` Source text of the condition, shortened when long; null when unconditional. *** ### reason > **reason**: `string` | `null` *** ### hasDynamicReason > **hasDynamicReason**: `boolean` The reason could not be worked out to a string, so `reason` holds its source text. *** ### isParallel > **isParallel**: `boolean` *** ### isSerial > **isSerial**: `boolean` Playwright only. *** ### isShuffled > **isShuffled**: `boolean` Vitest only. *** ### isParameterized > **isParameterized**: `boolean` Declared with `.each` or `.for`. *** ### tags > **tags**: `string`\[] *** ### annotations > **annotations**: `object`\[] #### type > **type**: `string` #### description > **description**: `string` | `null` #### lineStart > **lineStart**: `number` *** ### locks > **locks**: `string`\[] Playwright only. *** ### meta > **meta**: `Record`<`string`, [`MetaValue`](../type-aliases/MetaValue.md)> | `null` Vitest only; literal values only. *** ### hooks > **hooks**: `object` Hooks registered directly in the describe. #### beforeAll > **beforeAll**: `number` #### beforeEach > **beforeEach**: `number` #### afterEach > **afterEach**: `number` #### afterAll > **afterAll**: `number` #### aroundEach > **aroundEach**: `number` Vitest only. #### aroundAll > **aroundAll**: `number` Vitest only. *** ### timeout > **timeout**: `number` | `null` In milliseconds; null when nothing here sets it. *** ### isSlow > **isSlow**: `boolean` Playwright `test.slow()` written in the describe, which triples the timeout of its tests. *** ### slowCondition > **slowCondition**: `string` | `null` Source text of the `test.slow()` condition; null when unconditional or not slow. *** ### retries > **retries**: `number` | `null` *** ### retryDelay > **retryDelay**: `number` | `null` Vitest only: milliseconds between retries, from the `retry` object (4.1). *** ### retryCondition > **retryCondition**: `string` | `null` Vitest only: source text of the `retry` object's `condition`, a RegExp or a function. *** ### repeats > **repeats**: `number` | `null` Vitest only: how many times the runner runs each test again after the first run. --- --- url: /test-inventory/api/interfaces/TestRecord.md --- # Interface: TestRecord A test, with the effective values the runner applies, including those inherited from its describes. ## Properties ### id > **id**: `string` Derived from the file path, the describe titles, the title and its position among tests with the same title. *** ### bodyHash > **bodyHash**: `string` | `null` Hash of the test body, ignoring whitespace and comments; null when the test has no body. *** ### suiteId > **suiteId**: `string` | `null` The nearest describe; null outside any describe. *** ### suitePath > **suitePath**: `string`\[] Titles of the describes around the test, outermost first. *** ### relativeFilePath > **relativeFilePath**: `string` Path from the project root, with `/` separators. *** ### fileName > **fileName**: `string` *** ### lineStart > **lineStart**: `number` *** ### lineEnd > **lineEnd**: `number` *** ### columnStart > **columnStart**: `number` In UTF-16 code units. *** ### columnEnd > **columnEnd**: `number` In UTF-16 code units. *** ### declaredAt > **declaredAt**: { `relativeFilePath`: `string`; `lineStart`: `number`; `columnStart`: `number`; } | `null` Where a helper imported from another file writes the test; the record's location is then the call that registers it. Null for code written in the scanned file. #### Union Members ##### Type Literal { `relativeFilePath`: `string`; `lineStart`: `number`; `columnStart`: `number`; } ##### relativeFilePath > **relativeFilePath**: `string` The helper's file, from the project root. ##### lineStart > **lineStart**: `number` ##### columnStart > **columnStart**: `number` In UTF-16 code units. *** `null` *** ### title > **title**: `string` As the runner shows it, tags included. *** ### titleWithoutTags > **titleWithoutTags**: `string` `title` with `@tag` words removed and spaces tidied. *** ### hasDynamicTitle > **hasDynamicTitle**: `boolean` The title could not be resolved to a fixed string, so `title` holds its source text. *** ### fullName > **fullName**: `string` `suitePath` and `title` joined with " > ". *** ### isTypeTest > **isTypeTest**: `boolean` Declared in a Vitest type test file (`*.test-d.ts`, `*.spec-d.ts`). *** ### state > **state**: `"active"` | `"skip"` | `"fixme"` | `"todo"` | `"expectedToFail"` | `"notLoaded"` *** ### isActive > **isActive**: `boolean` *** ### isSkipped > **isSkipped**: `boolean` *** ### isFixme > **isFixme**: `boolean` *** ### isTodo > **isTodo**: `boolean` *** ### isExpectedToFail > **isExpectedToFail**: `boolean` *** ### isNotLoaded > **isNotLoaded**: `boolean` The runner refuses to load the file; a diagnostic at `stateLine` says why. *** ### isOnly > **isOnly**: `boolean` The test or a describe around it uses `.only`. *** ### isConditional > **isConditional**: `boolean` The state applies only when a condition holds, as with `test.skip(condition)` or `skipIf`. *** ### condition > **condition**: `string` | `null` Source text of the condition, shortened when long; null when unconditional. *** ### reason > **reason**: `string` | `null` *** ### hasDynamicReason > **hasDynamicReason**: `boolean` The reason could not be worked out to a string, so `reason` holds its source text. *** ### stateSource > **stateSource**: `"file"` | `"suite"` | `"test"` | `null` null for active tests. *** ### stateLine > **stateLine**: `number` | `null` The line that sets the state; null for active tests. *** ### isParallel > **isParallel**: `boolean` Both `isParallel` and `isSerial` false means the framework's default mode. *** ### isSerial > **isSerial**: `boolean` *** ### tags > **tags**: `string`\[] From the test and every describe around it, outermost first, without duplicates. *** ### annotations > **annotations**: `object`\[] #### type > **type**: `string` #### description > **description**: `string` | `null` #### lineStart > **lineStart**: `number` *** ### comments > **comments**: `string`\[] Comment lines directly above the test. By default only filled for tests that are not active. *** ### locks > **locks**: `string`\[] From the test, its describes and the `test.describe.configure()` calls around it. Playwright only. *** ### meta > **meta**: `Record`<`string`, [`MetaValue`](../type-aliases/MetaValue.md)> | `null` Merged from the describes and the test. Vitest only; literal values only. *** ### hooks > **hooks**: `object` Every hook that runs around the test: those of the file and of each describe around it. #### beforeAll > **beforeAll**: `number` #### beforeEach > **beforeEach**: `number` #### afterEach > **afterEach**: `number` #### afterAll > **afterAll**: `number` #### aroundEach > **aroundEach**: `number` Vitest only. #### aroundAll > **aroundAll**: `number` Vitest only. *** ### timeout > **timeout**: `number` | `null` In milliseconds: the last `test.setTimeout()` in the body or a `beforeEach`, otherwise the nearest describe or file that sets one. Null when the file sets none, so the config's timeout applies. *** ### isSlow > **isSlow**: `boolean` Playwright `test.slow()` applies: the runner triples the timeout. *** ### slowCondition > **slowCondition**: `string` | `null` Source text of the condition of the `test.slow()` that applies; null when unconditional or not slow. *** ### retries > **retries**: `number` | `null` How many times the runner retries the test when it fails; null when the file sets none. *** ### retryDelay > **retryDelay**: `number` | `null` Vitest only: milliseconds between retries, from the `retry` object (4.1). *** ### retryCondition > **retryCondition**: `string` | `null` Vitest only: source text of the `retry` object's `condition`, a RegExp or a function. *** ### repeats > **repeats**: `number` | `null` Vitest only: how many times the runner runs the test again after the first run. *** ### fixtures > **fixtures**: `string`\[] Names the test function takes from its context, then those of the `beforeEach`, `afterEach` and `aroundEach` hooks around it, once each. Vitest's own context keys, such as `expect` and `task`, are left out. *** ### expectCount > **expectCount**: `number` Calls of `expect()`, `expect.soft()` and `expect.poll()` written in the body, each counted once. *** ### expectedAssertions > **expectedAssertions**: `number` | `null` Vitest `expect.assertions(n)` in the body. *** ### expectsAssertions > **expectsAssertions**: `boolean` Vitest `expect.hasAssertions()` in the body. *** ### hasEmptyBody > **hasEmptyBody**: `boolean` The test function has no statements, comments aside. *** ### steps > **steps**: `object`\[] Playwright `test.step()` and `test.step.skip()` calls in the body, in source order. #### title > **title**: `string` The title, or its source text when it can't be worked out. #### hasDynamicTitle > **hasDynamicTitle**: `boolean` #### lineStart > **lineStart**: `number` #### depth > **depth**: `number` 0 for a step directly in the test, 1 for a step inside it, and so on. #### isSkipped > **isSkipped**: `boolean` Declared with `test.step.skip()`. *** ### isBenchmark > **isBenchmark**: `boolean` Vitest 5 benchmark: in a benchmark file, or using the `bench` fixture. Plain `vitest` doesn't run it. *** ### isParameterized > **isParameterized**: `boolean` Declared with `.each` or `.for`, or inside a describe declared with them, so the runner repeats it. *** ### caseCount > **caseCount**: `number` | `null` How many times the runner repeats the test: the cases of its own table times those of each describe table around it. Null when a table is not written out in the file, or the test is not parameterized. *** ### isInLoop > **isInLoop**: `boolean` Declared inside a loop, `forEach` or `map`. *** ### isInCondition > **isInCondition**: `boolean` Declared inside `if`, `?:`, `&&`, `switch` or a `catch` block. *** ### isInFunction > **isInFunction**: `boolean` Declared inside a plain function. --- --- url: /test-inventory/api/interfaces/DeclaredAt.md --- # Interface: DeclaredAt **`Inline`** ## Properties ### relativeFilePath > **relativeFilePath**: `string` The helper's file, from the project root. *** ### lineStart > **lineStart**: `number` *** ### columnStart > **columnStart**: `number` In UTF-16 code units. --- --- url: /test-inventory/api/interfaces/Annotation.md --- # Interface: Annotation **`Inline`** ## Properties ### type > **type**: `string` *** ### description > **description**: `string` | `null` *** ### lineStart > **lineStart**: `number` --- --- url: /test-inventory/api/interfaces/Diagnostic.md --- # Interface: Diagnostic ## Properties ### code > **code**: [`DiagnosticCode`](../type-aliases/DiagnosticCode.md) *** ### level > **level**: `"error"` | `"warning"` *** ### message > **message**: `string` One sentence describing the problem. *** ### relativeFilePath > **relativeFilePath**: `string` *** ### lineStart > **lineStart**: `number` | `null` null for problems with the whole file. *** ### columnStart > **columnStart**: `number` | `null` null for problems with the whole file. *** ### testId > **testId**: `string` | `null` The affected test; null when none. *** ### source > **source**: `string` | `null` Source text involved, shortened when long; null when none. --- --- url: /test-inventory/api/type-aliases/PickedInventory.md --- # Type Alias: PickedInventory\ > **PickedInventory**<`K`> = `Omit`<[`Inventory`](../interfaces/Inventory.md), `"tests"`> & `object` An inventory whose test records have only `id` and the keys picked with `pickTestFields()`. ## Type Declaration ### tests > **tests**: `Pick`<[`TestRecord`](../interfaces/TestRecord.md), `"id"` | `K`>\[] ## Type Parameters ### K `K` *extends* keyof [`TestRecord`](../interfaces/TestRecord.md) --- --- url: /test-inventory/api/type-aliases/CommentsMode.md --- # Type Alias: CommentsMode > **CommentsMode** = `"all"` | `"non-active"` | `"none"` **`Inline`** --- --- url: /test-inventory/api/type-aliases/Framework.md --- # Type Alias: Framework > **Framework** = `"playwright"` | `"vitest"` **`Inline`** --- --- url: /test-inventory/api/type-aliases/TestState.md --- # Type Alias: TestState > **TestState** = `"active"` | `"skip"` | `"fixme"` | `"todo"` | `"expectedToFail"` | `"notLoaded"` **`Inline`** The state the runner applies to a test, after describes and file-level calls are taken into account. `notLoaded`: the runner refuses to load the file, so none of its tests run. --- --- url: /test-inventory/api/type-aliases/StateSource.md --- # Type Alias: StateSource > **StateSource** = `"test"` | `"suite"` | `"file"` **`Inline`** Where a test's state is set: on the test, on a describe around it, or at file level. --- --- url: /test-inventory/api/type-aliases/DiagnosticLevel.md --- # Type Alias: DiagnosticLevel > **DiagnosticLevel** = `"error"` | `"warning"` **`Inline`** `error`: tests may be missing from the output. `warning`: a value is missing or approximate. --- --- url: /test-inventory/api/type-aliases/DiagnosticCode.md --- # Type Alias: DiagnosticCode > **DiagnosticCode** = `"unresolved-test-import"` | `"parse-error"` | `"read-error"` | `"framework-mismatch"` | `"unresolved-tag"` | `"unresolved-annotation"` | `"unresolved-cases"` | `"test-in-loop"` | `"dynamic-title"` | `"duplicate-title"` | `"duplicate-id"` | `"internal-error"` | `"invalid-chain"` | `"skip-without-body"` | `"test-without-body"` | `"describe-not-called"` | `"async-describe"` | `"nested-test"` | `"tag-without-at"` | `"removed-api"` | `"only-in-skipped-describe"` | `"config-may-change-state"` | `"symlink-ignored"` | `"invalid-import"` | `"globals-not-enabled"` | `"unknown-framework-version"` | `"unsupported-framework-version"` | `"local-test-object"` | `"state-in-helper"` | `"unresolved-option"` | `"dynamic-reason"` | `"no-files-matched"` | `"unsupported-file"` | `"approximate-framework-version"` | `"test-in-function"` | `"empty-title"` | `"bench-outside-benchmark-file"` --- --- url: /test-inventory/api/type-aliases/MetaValue.md --- # Type Alias: MetaValue > **MetaValue** = `string` | `number` | `boolean` | `null` | `MetaValue`\[] | {\[`key`: `string`]: `MetaValue`; } --- --- url: /test-inventory/api/type-aliases/InventoryLine.md --- # Type Alias: InventoryLine > **InventoryLine** = `object` & `Pick`<[`Inventory`](../interfaces/Inventory.md), `"schemaVersion"` | `"tool"` | `"framework"` | `"frameworkVersion"`> | `object` & [`FileRecord`](../interfaces/FileRecord.md) | `object` & [`SuiteRecord`](../interfaces/SuiteRecord.md) | `object` & [`TestRecord`](../interfaces/TestRecord.md) | `object` & [`Diagnostic`](../interfaces/Diagnostic.md) | `object` & [`Summary`](../interfaces/Summary.md) One line of the JSON Lines output (`--format jsonl`): the inventory with each record on its own line, marked by `type`. The `inventory` line comes first, then the files, suites, tests and diagnostics in the order of the JSON output, and the `summary` line last. --- --- url: /test-inventory/api/functions/filterInventory.md --- # Function: filterInventory() > **filterInventory**(`inventory`, `filter`): [`Inventory`](../interfaces/Inventory.md) Keeps the tests that pass every condition of the filter; a list keeps a test that has any of its values. The result describes only those tests: their files, the describes around them, their diagnostics, and a summary counted over them. Files without any tests stay, with their diagnostics, since tests may be missing from them. `summary.fileCount` stays the number of files scanned. Throws a `TypeError` for an invalid filter. ## Parameters ### inventory [`Inventory`](../interfaces/Inventory.md) ### filter #### states? (`"active"` | `"skip"` | `"fixme"` | `"todo"` | `"expectedToFail"` | `"notLoaded"`)\[] Keeps tests in one of these states. #### tags? `string`\[] Keeps tests with one of these tags, as the output writes them. #### grep? `string` Keeps tests whose `fullName` matches this regular expression, ignoring case, as Playwright's `--grep` does. #### only? `boolean` Keeps tests that use `.only`, or are inside a describe that does. #### withoutReason? `boolean` Keeps tests that don't run and give no reason. Tests in a file the runner refuses are left out. ## Returns [`Inventory`](../interfaces/Inventory.md) --- --- url: /test-inventory/api/functions/pickTestFields.md --- # Function: pickTestFields() > **pickTestFields**<`K`>(`inventory`, `fields`): [`PickedInventory`](../type-aliases/PickedInventory.md)<`K`> Leaves only `id` and the given keys on each test record, in that order, to make the output smaller. Suites, diagnostics and the summary are unchanged. The result no longer matches the JSON Schema, which requires every key. Throws a `TypeError` for a key that test records don't have. ## Type Parameters ### K `K` *extends* keyof [`TestRecord`](../interfaces/TestRecord.md) ## Parameters ### inventory [`Inventory`](../interfaces/Inventory.md) ### fields readonly `K`\[] ## Returns [`PickedInventory`](../type-aliases/PickedInventory.md)<`K`> --- --- url: /test-inventory/api/functions/scanSource.md --- # Function: scanSource() > **scanSource**(`options`): [`Inventory`](../interfaces/Inventory.md) Scans the code of one test file, without reading anything from disk. Throws a `TypeError` for invalid options. Problems in the code itself are reported as diagnostics. A custom `test` object imported from a local file can't be followed here; use `scan()` for that. ## Parameters ### options #### code `string` #### relativeFilePath `string` Path of the file from the project root, used in the output and in ids. #### framework `"playwright"` | `"vitest"` #### frameworkVersion? `string` | `null` The framework version whose rules apply. The newest rules apply when it is missing. #### comments? `"all"` | `"non-active"` | `"none"` Which tests get the comments above them. Defaults to `non-active`. ## Returns [`Inventory`](../interfaces/Inventory.md) --- --- url: /test-inventory/api/functions/scan.md --- # Function: scan() > **scan**(`options`): `Promise`<[`Inventory`](../interfaces/Inventory.md)> Scans the test files that match `patterns` under `root`, one file at a time. Throws a `TypeError` for invalid options. Problems with files and code are reported as diagnostics. ## Parameters ### options #### patterns `string`\[] Glob patterns for the test files, relative to `root`. As in Playwright and Vitest, they match names that start with a dot too. #### framework `"playwright"` | `"vitest"` #### root? `string` The project root. Paths in the output are relative to it. Defaults to the current directory. #### ignore? `string`\[] Glob patterns to leave out, in addition to `node_modules` and `.git`. #### frameworkVersion? `string` The framework version whose rules apply, instead of the one read from the project. #### comments? `"all"` | `"non-active"` | `"none"` Which tests get the comments above them. Defaults to `non-active`. ## Returns `Promise`<[`Inventory`](../interfaces/Inventory.md)> --- --- url: /test-inventory/about.md description: >- Who maintains test-inventory, how to support it and where the examples in the docs come from. --- # About test-inventory is written and maintained by [Andrey Rychkov](https://github.com/Huldoser). It is open source under the [MIT License](https://github.com/Huldoser/test-inventory/blob/main/LICENSE). ## Support the project * [Star it on GitHub](https://github.com/Huldoser/test-inventory). Stars help other teams find it. * [Sponsor it on GitHub](https://github.com/sponsors/Huldoser). Sponsorship pays for the time that goes into new Playwright and Vitest versions, fixes and docs. * [Report a problem](https://github.com/Huldoser/test-inventory/issues/new?template=scan-reads-test-wrong.yml) with a test file the scan reads wrong. A small file that shows it is the most useful part of a report. ## Contributing [CONTRIBUTING.md](https://github.com/Huldoser/test-inventory/blob/main/CONTRIBUTING.md) explains how to set up the project, run the checks and add a changeset. ## The examples Every example in these docs comes from a made-up trading app: a web app with Playwright tests, and a strategy engine, an order service, a React dashboard, a Vue watchlist, a chart tested in the browser and a monorepo, all with Vitest. They live in [`tests/fixtures`](https://github.com/Huldoser/test-inventory/tree/main/tests/fixtures), the [examples](/guide/examples) page shows each one, and every output in these docs is produced by scanning them, which the test suite checks. --- --- url: /test-inventory/changelog.md description: What changed in each release of test-inventory. --- # test-inventory ## 0.4.0 ### Minor Changes * [#42](https://github.com/Huldoser/test-inventory/pull/42) [`6849f55`](https://github.com/Huldoser/test-inventory/commit/6849f555d8b214ebbc8fdb9aa9cbac8cfe0913db) - Tests say what their body does: `fixtures` (the names the test and the `beforeEach`, `afterEach` and `aroundEach` hooks around it take from the context, without Vitest's own keys), `expectCount` (`expect()`, `expect.soft()` and `expect.poll()` calls in the body), Vitest's `expect.assertions(n)` and `expect.hasAssertions()` as `expectedAssertions` and `expectsAssertions`, `hasEmptyBody`, and Playwright's `test.step()` titles in `steps`. Vitest's `annotate()` from the test context adds annotations. Vitest 5 benchmarks, in `*.bench.ts` files or using the `bench` fixture, have `isBenchmark` and a `benchmarkCount` in the summary. New warnings: `empty-title` for a test or describe titled `''`, and `bench-outside-benchmark-file`. * [#40](https://github.com/Huldoser/test-inventory/pull/40) [`f462ee0`](https://github.com/Huldoser/test-inventory/commit/f462ee015fea8ee802f567a26dbaa4db94ada1d5) - The output has a record for each scanned file in `files`, and counts hooks: `hooks` on a file holds the hooks at its top level, on a describe the hooks registered in it, and on a test every hook that runs around it, by kind (`beforeAll`, `beforeEach`, `afterEach`, `afterAll`, and Vitest's `aroundEach` and `aroundAll`). Hooks a helper function registers count where it is called. JSON Lines output has a `file` line for each file, and a filtered inventory keeps the files of the matched tests and the files without tests. * [#41](https://github.com/Huldoser/test-inventory/pull/41) [`1ab9cb4`](https://github.com/Huldoser/test-inventory/commit/1ab9cb4002dd971ddb62570dc55ac564504e3a83) - Tests and describes have their run settings: `timeout`, `isSlow` and `slowCondition`, `retries`, `retryDelay`, `retryCondition` and `repeats`. Playwright's come from `test.describe.configure({ timeout, retries })`, `test.setTimeout()` and `test.slow()` in a describe, a `beforeEach` or the test body; Vitest's from the `timeout`, `retry` and `repeats` options, the timeout after the body, and the options a describe passes down. A test gets the value from the nearest place that sets it, and `null` when the file sets none, so the config decides. A timeout that can't be worked out, or that is set only under a condition, gets an `unresolved-option` warning. ## 0.3.0 ### Minor Changes * [#37](https://github.com/Huldoser/test-inventory/pull/37) [`5aeb139`](https://github.com/Huldoser/test-inventory/commit/5aeb13959daaeb0fdf4488765944fe42ef7754bc) - `--fields` keeps only the named keys of each test, and `id`, as in `--fields fullName,reason`, so a short question gets a short answer. Output written with it leaves keys out, so it no longer matches the JSON Schema. The library exports the same as `pickTestFields()`. * [#37](https://github.com/Huldoser/test-inventory/pull/37) [`5aeb139`](https://github.com/Huldoser/test-inventory/commit/5aeb13959daaeb0fdf4488765944fe42ef7754bc) - Filters keep only the tests a question is about: `--state`, `--tag`, `--grep` (a regular expression on the full name, ignoring case), `--only` and `--without-reason`, as in `--state skip --state fixme` for the tests that don't run. The output then describes only those tests: the describes around them, their diagnostics and a summary counted over them. The library exports the same as `filterInventory()`. * [#37](https://github.com/Huldoser/test-inventory/pull/37) [`5aeb139`](https://github.com/Huldoser/test-inventory/commit/5aeb13959daaeb0fdf4488765944fe42ef7754bc) - `--format jsonl` writes the inventory as JSON Lines: an `inventory` line, one line per suite, test and diagnostic, and a `summary` line, each marked with a `type` key. Tools that read one record per line, such as `grep` or DuckDB, no longer need the whole document in memory. The `InventoryLine` type describes the lines. ### Patch Changes * [#34](https://github.com/Huldoser/test-inventory/pull/34) [`0f96258`](https://github.com/Huldoser/test-inventory/commit/0f96258e65085fa1b7c667b5fa7219a8b19230df) - A fixtures file that builds its test object with a function, written there or imported from another file, as in `export const test = defineDeskTest('equities')`, is followed: MSW's test files import such a `test`. Their tests were missing before, with no diagnostic when the fixtures file didn't import the framework itself. A test object built with a call the scan can't follow now gets an `unresolved-test-import` error that names the call, as in "builds it with withWarmup(), which is not followed". A function that returns what another function returns, as `() => onFeed('NYSE')`, is followed too. * [#34](https://github.com/Huldoser/test-inventory/pull/34) [`0f96258`](https://github.com/Huldoser/test-inventory/commit/0f96258e65085fa1b7c667b5fa7219a8b19230df) - A call the runner refuses that is written like a declaration, as `describe.sequential('title', fn)` in Vitest 5, `test('title', fn, { retry: 2 })` from Vitest 4 or `test.only.skip('title', fn)`, is listed as a plain test or describe without the part that fails. Its tests were missing before; now they are `notLoaded`, like every other test in the file. * [#34](https://github.com/Huldoser/test-inventory/pull/34) [`0f96258`](https://github.com/Huldoser/test-inventory/commit/0f96258e65085fa1b7c667b5fa7219a8b19230df) - States follow the runners' precedence more closely, as checked against Vitest 5.0.4 running generated files. A skip in a test or hook body loses to a fixme, skip or todo set while the file loads, as on a describe around the test, since the body never runs. In Vitest, a test's own options win over its chain, so `test.skip('title', { skip: false }, fn)` is active, but the chain still sets its mode: `test.concurrent('title', { concurrent: false }, fn)` runs in parallel. A describe passes its `skip` option down to the tests and describes inside, where it wins over todo, also when the describe's own `.only` keeps the describe itself from being skipped. ## 0.2.1 ### Patch Changes * [#28](https://github.com/Huldoser/test-inventory/pull/28) [`04ff6d2`](https://github.com/Huldoser/test-inventory/commit/04ff6d2dd5a7ecc8fabffbf16b259fc9f2a68110) - A skip in a `catch` block, such as `context.skip()` when a test server can't start, is conditional, with the `catch` clause as its condition, as in `catch (error)`. It was recorded as an unconditional skip before. A test declared in a `catch` block gets `isInCondition: true`. * [#32](https://github.com/Huldoser/test-inventory/pull/32) [`3fcc933`](https://github.com/Huldoser/test-inventory/commit/3fcc93368dc9a4d810ff52b8d7d5fe4f99ca63ee) - Globs match folders and files whose names start with a dot, as in `.storybook/` or `.github/scripts/`, as they do in Playwright and Vitest; test files there were left out before. `.git` folders are always left out, like `node_modules`. * [#31](https://github.com/Huldoser/test-inventory/pull/31) [`631cc6b`](https://github.com/Huldoser/test-inventory/commit/631cc6b208aca5b28076da6eba33d8e6426ab778) - Workspace packages are found from both `pnpm-workspace.yaml` and the `workspaces` of `package.json`. When a repository had both, as after a move to pnpm that kept the old list, only `package.json` was read, and a package listed only in `pnpm-workspace.yaml` got an `unresolved-test-import` error. * [#30](https://github.com/Huldoser/test-inventory/pull/30) [`968e2bc`](https://github.com/Huldoser/test-inventory/commit/968e2bcd15c2157faae8fe91604634b1c1f68175) - A call through an imported name that reads like a test function is followed, and reported with `unresolved-test-import` when it can't be, also when it goes through a member the scan doesn't know, as in `it.effect('title', fn)` from `@effect/vitest`, or passes a timeout after the body, as in `test('title', fn, 60_000)`. Such tests were missing before, with no diagnostic. A module's own export of a name now wins over the same name passed on by `export *`, as in JavaScript: `export const it = ...` next to `export * from 'vitest'` is no longer read as Vitest's `it`. * [#28](https://github.com/Huldoser/test-inventory/pull/28) [`04ff6d2`](https://github.com/Huldoser/test-inventory/commit/04ff6d2dd5a7ecc8fabffbf16b259fc9f2a68110) - Tests on a test object that a function returns are found, as in `conditionalTest({ min: 22 })('title', fn)` or `const quoteTest = onNode(22)`, when the function is declared in the file or imported from a local file. They were missing before, with no diagnostic. When the function returns a different test function on different runs, as `it.skip` or `it`, only the modifiers they all have apply, as for a variable chosen by a condition. When the function can't be followed and its name reads like a test object, as for one from an installed package, the calls on what it returns are read as tests on trust, with a `local-test-object` warning. ## 0.2.0 ### Minor Changes * [#18](https://github.com/Huldoser/test-inventory/pull/18) [`4698945`](https://github.com/Huldoser/test-inventory/commit/46989455dda81842d0a947bafb8d1270654be250) - A custom `test` object is followed through as many files as it takes: a fixtures file may re-export it, as in `export * from './base'`, or build it with `extend` on one imported from another file. When it can't be followed, the `unresolved-test-import` message says where the chain of files ends. A test file that builds its own test object from an imported one, as in `const test = base.extend({...})`, no longer gets a `local-test-object` warning when the import is followed. * [#18](https://github.com/Huldoser/test-inventory/pull/18) [`4698945`](https://github.com/Huldoser/test-inventory/commit/46989455dda81842d0a947bafb8d1270654be250) - Tests and describes declared in a helper function are recorded at each call of the helper from the file or a describe, in the describe around the call, as the runner registers them; before, they were listed once, where written. Titles built from the helper's parameters are worked out from each call's arguments. Helpers imported from other files are followed too: their records give the call in the scanned file as their location, and a new `declaredAt` key on test and suite records gives the file and line where the helper writes the test, `null` for code in the scanned file. Records of tests in helpers get new ids, as their describes change. The `test-in-function` warning now only marks tests in functions the scan can't place, and the call where the scan stops following helpers, after 10,000 calls in a file. * [#18](https://github.com/Huldoser/test-inventory/pull/18) [`4698945`](https://github.com/Huldoser/test-inventory/commit/46989455dda81842d0a947bafb8d1270654be250) - Files that playwright-bdd generates from Gherkin features are read: they import Playwright's `test` from `playwright-bdd`. * [#18](https://github.com/Huldoser/test-inventory/pull/18) [`4698945`](https://github.com/Huldoser/test-inventory/commit/46989455dda81842d0a947bafb8d1270654be250) - In Playwright, a test in a loop whose title path is the same on each pass gets a `duplicate-title` error, and the tests in its file are `notLoaded`, as Playwright refuses the file. Vitest runs duplicate titles, so nothing changes there. * [#18](https://github.com/Huldoser/test-inventory/pull/18) [`4698945`](https://github.com/Huldoser/test-inventory/commit/46989455dda81842d0a947bafb8d1270654be250) - Path aliases come from the tsconfig nearest the importing file, as TypeScript finds it, with every file it `extends`, packages included, and `${configDir}` filled in. Without a `tsconfig.json`, the nearest `tsconfig.base.json` is used, as in an Nx workspace, and a solution-style tsconfig takes its aliases from the projects it `references`. * [#18](https://github.com/Huldoser/test-inventory/pull/18) [`4698945`](https://github.com/Huldoser/test-inventory/commit/46989455dda81842d0a947bafb8d1270654be250) - Workspace packages are followed before an install, from the `workspaces` of `package.json` or from `pnpm-workspace.yaml`. When a package's `exports`, `types` or `main` point to build output that doesn't exist yet, as in `dist/esm/index.js`, the source next to it in `src/` is used. Patterns and links that lead out of the workspace folder are not followed. ### Patch Changes * [#19](https://github.com/Huldoser/test-inventory/pull/19) [`bc7a0c3`](https://github.com/Huldoser/test-inventory/commit/bc7a0c36b364b338e1aef0c0a7622b22590a8305) - A test or hook that calls helper functions which call each other in many places no longer makes the scan hang. Each helper's states are applied once for each set of conditions it is reached under, and past 10,000 helper calls from one call, a `state-in-helper` warning says the rest are not applied. * [#20](https://github.com/Huldoser/test-inventory/pull/20) [`7f32dd4`](https://github.com/Huldoser/test-inventory/commit/7f32dd4605e93fa0db47200b917dfafb91ac1da5) - oxc-parser is updated from 0.153.0 to 0.154.0. The scan output doesn't change. ## 0.1.2 ### Patch Changes * [#13](https://github.com/Huldoser/test-inventory/pull/13) [`89a029f`](https://github.com/Huldoser/test-inventory/commit/89a029f9e45c2aaa57b5f907717bb8f00829e335) - Locks set with `test.describe.configure({ lock })`, new in Playwright 1.64, are added to every test in the file or describe where the call is made. They were dropped before. * [#13](https://github.com/Huldoser/test-inventory/pull/13) [`89a029f`](https://github.com/Huldoser/test-inventory/commit/89a029f9e45c2aaa57b5f907717bb8f00829e335) - From Vitest 5, a file that calls `vi.mock`, `vi.unmock` or `vi.hoisted` anywhere but its top level gets a `removed-api` error at the call, and its tests are `notLoaded`, as Vitest refuses the file. Files with in-source tests are not checked, as in Vitest. ## 0.1.1 ### Patch Changes * [#5](https://github.com/Huldoser/test-inventory/pull/5) [`c812450`](https://github.com/Huldoser/test-inventory/commit/c812450a0652339ccc713ce305a14882154dd6e6) - A custom test object imported from `'.'` or `'..'` is followed like other relative imports. * [#5](https://github.com/Huldoser/test-inventory/pull/5) [`c812450`](https://github.com/Huldoser/test-inventory/commit/c812450a0652339ccc713ce305a14882154dd6e6) - When a pattern names a folder, the `no-files-matched` message suggests a pattern for the test files in it, and the command line prints that message. * [#5](https://github.com/Huldoser/test-inventory/pull/5) [`c812450`](https://github.com/Huldoser/test-inventory/commit/c812450a0652339ccc713ce305a14882154dd6e6) - A pattern whose folder doesn't exist now gives only the `no-files-matched` warning, not a `read-error`. * [#5](https://github.com/Huldoser/test-inventory/pull/5) [`c812450`](https://github.com/Huldoser/test-inventory/commit/c812450a0652339ccc713ce305a14882154dd6e6) - Messages for problems that make Playwright refuse a file now say that Playwright runs no tests at all, in any file. * [#5](https://github.com/Huldoser/test-inventory/pull/5) [`c812450`](https://github.com/Huldoser/test-inventory/commit/c812450a0652339ccc713ce305a14882154dd6e6) - A new `test-in-function` warning marks a test declared in a function that is called more than once, from another describe, never, or from other files. The runner registers such a test in the describe around each call, while the record stays where the test is written. * [#5](https://github.com/Huldoser/test-inventory/pull/5) [`c812450`](https://github.com/Huldoser/test-inventory/commit/c812450a0652339ccc713ce305a14882154dd6e6) - A version range that starts below the oldest supported version, such as `^1.40.0`, now gets the `approximate-framework-version` warning, since the newest rules of its major apply. * [#5](https://github.com/Huldoser/test-inventory/pull/5) [`c812450`](https://github.com/Huldoser/test-inventory/commit/c812450a0652339ccc713ce305a14882154dd6e6) - Every test in a Vitest file that fails to load is now `notLoaded`, as Vitest runs none of them: a chain Vitest doesn't have, `sequential` or `bench` from Vitest 5, or options as the third argument from Vitest 4. A call to `bench` is reported where it is made, not where it is imported. ## 0.1.0 ### Minor Changes * [#1](https://github.com/Huldoser/test-inventory/pull/1) [`5073b07`](https://github.com/Huldoser/test-inventory/commit/5073b078db34185f1069d71a869a6c8671621977) - First release. Scans Playwright and Vitest test files and lists every test declaration as JSON, with its location, describes, tags, annotations and state, without running the tests. --- --- url: /test-inventory/api.md --- # test-inventory ## Interfaces * [InventoryFilter](interfaces/InventoryFilter.md) * [ScanOptions](interfaces/ScanOptions.md) * [ScanSourceOptions](interfaces/ScanSourceOptions.md) * [Inventory](interfaces/Inventory.md) * [Tool](interfaces/Tool.md) * [Summary](interfaces/Summary.md) * [FileRecord](interfaces/FileRecord.md) * [HookCounts](interfaces/HookCounts.md) * [SuiteRecord](interfaces/SuiteRecord.md) * [TestRecord](interfaces/TestRecord.md) * [DeclaredAt](interfaces/DeclaredAt.md) * [Annotation](interfaces/Annotation.md) * [Diagnostic](interfaces/Diagnostic.md) ## Type Aliases * [PickedInventory](type-aliases/PickedInventory.md) * [CommentsMode](type-aliases/CommentsMode.md) * [Framework](type-aliases/Framework.md) * [TestState](type-aliases/TestState.md) * [StateSource](type-aliases/StateSource.md) * [DiagnosticLevel](type-aliases/DiagnosticLevel.md) * [DiagnosticCode](type-aliases/DiagnosticCode.md) * [MetaValue](type-aliases/MetaValue.md) * [InventoryLine](type-aliases/InventoryLine.md) ## Functions * [filterInventory](functions/filterInventory.md) * [pickTestFields](functions/pickTestFields.md) * [scanSource](functions/scanSource.md) * [scan](functions/scan.md)