Testing Strategy

Correctness is a core value for the Ganzua project. To this end, Ganzua has an exhaustive test suite. Large parts of this test suite are part of the documentation: the specification is the test. This doesn't just ensure exhaustive documentation for humans, but also makes it easier to keep docs in sync with the actual behavior.

Guiding values

If it can be tested automatically, do test it automatically. Human attention to detail doesn't scale. Thus, there are lots of automated checks within these docs. All internal links are checked, so there cannot be broken references. Nearly all command line examples are checked automatically.
- Corollary: if it can be generated automatically, do generate it automatically. For example, the CLI reference documentation is auto-generated by doing a bit of reflection on the code that defines the CLI. There is a single source of truth.
If it cannot be tested, it doesn't exist. Ganzua configures Pytest to require 100% code coverage. This encourages thorough tests, and discourages implementing unnecessary functionality. Coverage metrics aren't perfect, though. For example, testing different combinations of CLI options is difficult to guide via code coverage.
Write end-to-end tests. The tests double as documentation, so should be meaningful for users. This is strongly inspired by Behavior-Driven Development (BDD) practices. Instead of unit-testing internal functionality, as much as possible should be tested via the same command line interface that's accessible to end users.

How it works

Ganzua uses the Pytest testing framework, but uses a custom doctest-like approach for running examples embedded in the documentation.

For each documentation Markdown file, a custom tool parses the Markdown to extract examples, and rewrites it to include up-to-date outputs. Instead of writing the new version back directly (which might override intentional changes!), the test suite uses the Inline-Snapshot plugin to diff the file contents, and optionally update the file on disk.

Here's the entire driver for that test:

@pytest.mark.parametrize(
    "path",
    resources.DOCS.glob("**/*.md"),
    ids=lambda p: str(p.relative_to(resources.DOCS)),
)
def test_docs(path: pathlib.Path) -> None:
    markdown = Runner.run(path)
    assert markdown == external_file(str(path), format=".txt")

For example, we might have an outdated example of running Ganzua:

```console
$ touch $EXAMPLE/pyproject.toml
$ ganzua constraints inspect $EXAMPLE
{"outdated": "shape"}
```

Inline-Snapshot will show a diff with the actual output, as part of the Pytest session:

═══════════════════════ inline-snapshot ═══════════════════════
╭────────────────────── docs/testing.md ──────────────────────╮
│ @@ -75,5 +75,7 @@                                           │
│                                                             │
│  ```console                                                 │
│  $ touch $EXAMPLE/pyproject.toml                            │
│  $ ganzua constraints inspect $EXAMPLE                      │
│ -{"outdated": "shape"}                                      │
│ +{                                                          │
│ +  "requirements": []                                       │
│ +}                                                          │
│  ```                                                        │
╰─────────────────────────────────────────────────────────────╯
Do you want to fix these snapshots? [y/n] (n):

Resulting in a replaced Markdown fragment:

```console
$ touch $EXAMPLE/pyproject.toml
$ ganzua constraints inspect $EXAMPLE
{
  "requirements": []
}
```

Recognized Markdown patterns

This section documents the available Markdown patterns that can be used for running tests and automatically updating documentation files.

Each kind of pattern triggers on a marker line that matches a given regex, and can then transform that line and the following lines.

If no marker pattern matches, content is passed through verbatim.

no transformations by default

foo
bar
baz

foo
bar
baz

unknown <-- doctest: directives raise an error

<!-- doctest: foo bar -->

Error: unknown directive
note: at example:1
note: around here: <!-- doctest: foo bar -->

Doctest example

Used for self-documenting the supported Markdown directives.

Consists of a <div> or <details> tag with class doctest-example, and contains a Markdown code block with info-string md,doctest-example. A second Markdown code block (autogenerated) with info-string md,doctest-output will contain the output. If there's an error during processing, the autogenerated output block will have the info-string doctest-error.

<div class="doctest-example">

```md,doctest-example
<!-- command output: echo foo bar -->
<!-- command output end -->
```

</div>

<div class="doctest-example">

```md,doctest-example
<!-- command output: echo foo bar -->
<!-- command output end -->
```
```md,doctest-output
<!-- command output: echo foo bar -->

foo bar

<!-- command output end -->
```

</div>

input code block must be closed

```md,doctest-example
not closed

Error: must have matching closing fence
note: at example:1
note: around here: ```md,doctest-example

Command Output

Replace the content between to marker comments with the output from the given command. This is useful for auto-generating Markdown sections.

<!-- command output: echo foo bar -->
<!-- command output end -->

<!-- command output: echo foo bar -->

foo bar

<!-- command output end -->

output region must be closed

<!-- command output: echo foo bar -->
old content

Error: must have matching `<!-- command output end -->`
note: at example:1
note: around here: <!-- command output: echo foo bar -->

Console code block

Given a fenced code block of type console which contains $ command lines, execute the commands (sandboxed!) and insert their output into the code block.

```console
$ echo print this
$ echo another command
```

```console
$ echo print this
print this
$ echo another command
another command
```

console block must contain commands

```console
not a $ command
```

Error: must have at least one `$ ...` command line
note: at example:1
note: around here: ```console

must have closing fence

````console
$ echo some command
```

Error: must have matching closing fence
note: at example:1
note: around here: ````console

Count executed commands

Consistency check to count how many shell commands were executed by the doctest runner so far. Only counts commands inside console code blocks.

```console
$ echo 1
$ echo 2
$ echo 3
```

<!-- doctest: ran ??? commands -->

```console
$ echo 1
1
$ echo 2
2
$ echo 3
3
```

<!-- doctest: ran 3 commands -->

Collapsible output blocks

When the full output would be distracting, a <details> element can be used to hide it. The <summary> must contain a <code> element with the command.

<details><summary><code>$ echo foo bar</code></summary>
</details>

<details><summary><code>$ echo foo bar</code></summary>

```
foo bar
```

</details>

If the collapsible block should be open by default, the <details open> attribute may be used.

May have the open attribute

<details open><summary><code>$ echo foo bar</code></summary>
</details>

<details open><summary><code>$ echo foo bar</code></summary>

```
foo bar
```

</details>

Check Ganzua diff notes

A special utility for testing the notes feature of ganzua diff. Given a marker comment that's followed by a diff table, old and new lockfiles are constructed from the data in the table, and the table is recreated from ganzua diff output. This may change the notes column.

<!-- doctest: check ganzua diff notes -->

| package | old | new | notes |
|--|--|--|--|
| foo | 0.1.2 | 3.4.5 | |

<!-- doctest: check ganzua diff notes -->

| package | old   | new   | notes |
|---------|-------|-------|-------|
| foo     | 0.1.2 | 3.4.5 | (M)   |

Check Ganzua constraints bump

A special utility for checking how constraints are bumped. Consists of a marker comment and a table that will be updated. Columns:

locked: contains <package> <version> to define a single-package lockfile content.
old: contains a constraint/requirement in one of the following forms:
- <package> <constraint> for PEP-508 requirements
- <package> = <constraint> for Poetry requirements (without extra TOML-style quoting)
new (autogenerated): contains a constraint/requirement after bumping the old constraint in the context of the given lockfile.

Each column value may be quoted as inline code.

<!-- doctest: check ganzua constraints bump -->

| locked | old | new |
| --- | --- | --- |
| `foo 1.2.3` | `foo>=0.1` | |
| `bar 4.5.6` | `bar = ^3.2` | |

<!-- doctest: check ganzua constraints bump -->

| locked      | old          | new          |
|-------------|--------------|--------------|
| `foo 1.2.3` | `foo>=0.1`   | `foo>=1.2`   |
| `bar 4.5.6` | `bar = ^3.2` | `bar = ^4.5` |

Compare output

Given a list of multiple commands, check whether they all produce the same output. Consists of a marker comment and a list of commands, and will generate a details block with outputs if appropriate.

Example when the output is the same for all commands:

<!-- doctest: compare output -->
* `$ echo foo bar`
* `$ echo 'foo bar'`

<!-- doctest: compare output -->
* `$ echo foo bar`
* `$ echo 'foo bar'`

<details><summary>output for the above commands</summary>

```
foo bar
```

</details>

Example when some commands produce different output

<!-- doctest: compare output -->
* `$ echo foo bar`
* `$ echo different output`
* `$ echo 'foo bar'`

<!-- doctest: compare output -->
* `$ echo foo bar`
* `$ echo different output`
* `$ echo 'foo bar'`

<details><summary>output for the above commands</summary>

Output for:

* `$ echo foo bar`
* `$ echo 'foo bar'`

```
foo bar
```

Output for:

* `$ echo different output`

```
different output
```

</details>

must contain at least one command

<!-- doctest: compare output -->
not a command list

Error: must be followed by at least one command list item
note: at example:1
note: around here: <!-- doctest: compare output -->

Clean example

Reset the $EXAMPLE directory to a clean state. It's a good idea to invoke this at the beginning of each section of a specification.

```console
$ touch $EXAMPLE/a
$ touch $EXAMPLE/b
$ ls $EXAMPLE
```
<!-- doctest: clean example -->
```console
$ touch $EXAMPLE/c
$ ls $EXAMPLE
```

```console
$ touch $EXAMPLE/a
$ touch $EXAMPLE/b
$ ls $EXAMPLE
a
b
```
<!-- doctest: clean example -->
```console
$ touch $EXAMPLE/c
$ ls $EXAMPLE
c
```

Create lockfile

Create a lockfile from an example table. Triggered by a marker comment that must be followed by a table.

Syntax for the marker comment:

where:

format is either uv or poetry
path is the name of the file to be created, which MUST start with $EXAMPLE/

The table defines the locked dependency versions. Available columns (all optional):

name (default: example) is the package name
version (default: 0.1.0) is the locked version
source_toml is a TOML fragment describing the source of the locked package. Expected formats and default values are format-specific.

example creating an uv lockfile

<!-- doctest: create uv lockfile $EXAMPLE/uv.lock -->

| name | version |
|------|---------|
| foo  | 1.2.3   |

```console
$ cat $EXAMPLE/uv.lock
```

<!-- doctest: create uv lockfile $EXAMPLE/uv.lock -->

| name | version |
|------|---------|
| foo  | 1.2.3   |

```console
$ cat $EXAMPLE/uv.lock
version = 1
revision = 3
requires-python = ">=3.12"

[[package]]
name = "foo"
version = "1.2.3"
source = { registry = "https://pypi.org/simple" }
```

example creating a Poetry lockfile

<!-- doctest: create poetry lockfile $EXAMPLE/poetry.lock -->

| name | version |
|------|---------|
| foo  | 1.2.3   |

```console
$ cat $EXAMPLE/poetry.lock
```

<!-- doctest: create poetry lockfile $EXAMPLE/poetry.lock -->

| name | version |
|------|---------|
| foo  | 1.2.3   |

```console
$ cat $EXAMPLE/poetry.lock
[[package]]
name = "foo"
version = "1.2.3"

[metadata]
lock-version = "2.1"
python-versions = ">=3.12"
content-hash = "0000000000000000000000000000000000000000000000000000000000000000"
```

Supported shell commands

For security reasons, supported shell commands are tightly allowlisted.

Each shell command is processed as follows:

word splitting is performed per the usual Posix rules
variables are expanded – also within single quotes, unlike Posix rules
resulting word lists are matched against the command allowlist and emulated

Other shell features like redirects, pipelines, and substitutions other than the given variables are not supported.

Supported variables (may use either $name or ${name} syntax):

$EXAMPLE contains a temporary directory in which files may be edited
$CORPUS points to the corpus of example projects

A path is writable if it is below the $EXAMPLE directory.

ganzua

ganzua ARGS...

Can run arbitrary Ganzua commands.

Error handling: In most cases, errors will be printed out as part of the output. A line like [command exited with status 2] may be appended to the output.

Output post-processing: Parts of the output that match known variables will be un-substituted. The type of the output (JSON or plaintext) will be sniffed.

Example of an error:

$ ganzua inspect $EXAMPLE

Usage: ganzua inspect [OPTIONS] [LOCKFILE]
Try 'ganzua inspect --help' for help.

Error: Could not infer `LOCKFILE` for `${EXAMPLE}`.
[command exited with status 2]

After we create an example lockfile, we can observe that the output format is detected as JSON and properly syntax-highlighted:

name	version
foo	1.2.3

$ ganzua inspect $EXAMPLE

{
  "packages": [
    {
      "name": "foo",
      "version": "1.2.3",
      "source": "pypi"
    }
  ]
}

echo

echo ARGS...

Just print out the arguments, separated by spaces. Mostly useful for testing the doctest system.

cat

cat FILE

Print the contents of the given file.

If the filename looks like a TOML file, then syntax highlighting will be added. This is particularly useful for looking at pyproject.toml files.

$ cat $CORPUS/old-uv-project/pyproject.toml

[project]
name = "example"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.13"
dependencies = [
    "typing-extensions>=3,<4",
]

cp

cp SOURCE DEST

Copy the SOURCE file to DEST. Both arguments must be filenames. The DEST path must be writable.

$ ls $EXAMPLE
$ cp $CORPUS/new-uv-project/pyproject.toml $EXAMPLE/pyproject.toml
$ ls $EXAMPLE
pyproject.toml

touch

touch DEST

Touch the DEST file, i.e. create an empty file under that path if it doesn't exist. The DEST path must be writable.

ls

ls DIRNAME

List the contents of a directory.

env

env -C DEST ganzua ARGS...

Change the current working directory to DEST before running Ganzua.

For example, these two commands end up inspecting the same project:

$ ganzua inspect $CORPUS/new-uv-project
$ env -C $CORPUS/new-uv-project ganzua inspect

output for the above commands

{
  "packages": [
    {
      "name": "annotated-types",
      "version": "0.7.0",
      "source": "pypi"
    },
    {
      "name": "example",
      "version": "0.1.0",
      "source": {
        "direct": "."
      }
    },
    {
      "name": "typing-extensions",
      "version": "4.14.1",
      "source": "pypi"
    }
  ]
}

Keyboard shortcuts

Ganzua Manual