The PACT for test-modules

Tip

This version of the package was tagged as v.0.0.1 in the project repository, and can be examined there if desired.

I summarized the goal that the Python package that this series of articles is concerned with in the first article in the series as:

Goal

Every code-element in a project’s source tree should have a corresponding test-suite element.

My first step in pursuing this goal, after some thinking on where I wanted to start, and how I wanted to proceed, was to start with the code-elements that could only be described as file-system entities: modules (files) and packages (directories). The corresponding test-elements rule above, with that focus, could be implemented as some test or set of tests that simply asserted that for every given file-system object in a project’s source-tree, there was a corresponding file-system object in the relevant test-suite of the project. For example, given a source module at my_package/module.py, there should be a test_my_package/test_my_module.py in the related test-suite, and if that test-module did not exist, that was grounds for a failure of the test in question. For the purposes of the testing implemented in this version, the goal above could be restated as

Goal

Every source module and directory should have a corresponding test module and directory.

I did not expect the basic mechanics of that test-process to be difficult to implement. Even though I added some layers of functionality for various interim steps in the processes needed to get to that point, that expectation panned out about as I expected. The most significant initial challenge, as it turned out, was figuring out how to identify a project’s root directory. To understand why that was significant, I’ll summarize where the final version ended up.

The projects.py module (goblinfish.testing.pact.projects) provides a single class, ExaminesProjectModules, that provides a “baked in” test-method (test_source_modules_have_test_modules) that performs the actual test. That method relies on a handful of properties that are built in to the class itself:

Everything in the process that eventually get executed by test_source_modules_have_test_modules starts with determining the project_root. There are several common project-structure variations, often depending on how project package dependencies and the project’s Python Virtual Environment (PVE) are managed.

When pipenv manages packages, there is no set project structure, but it is a pretty safe bet that the Pipfile and Pipfile.lock files will live at the project root, and the project_root property uses the presence of either of those as an indicator to determine the value for the property. A fairly detailed but typical project structure under pipenv will likely look much like this:

project-name/
├── Pipfile
├── Pipfile.lock
├── pyproject.toml             # Build system configuration (if applicable)
├── src/                       # Source code directory
│   └── project_name/          # Main package directory
│       └── ...                # Package modules and subdirectories
├── tests/                     # Test suites directory
│   ├── unit                   # Unit tests
│   │   └── test_project_name  # Test suite for the project starts here
│   └── ...                    # Other test-types, like integration, E2E, etc.
├── scripts/                   # Scripts directory (optional)
│   └── ...
├── data/                      # Data directory (optional)
│   └── ...
├── docs/                      # Documentation directory (optional)
│   └── ...
├── README.md                  # Project description
├── LICENSE                    # License file
└── .gitignore                 # Git ignore file

The poetry package- and dependency-manager tool, as of this writing, shows a different project structure created using its tools:

project-name
├── pyproject.toml
├── README.md
├── project_name               # This is the SOURCE_DIR
│   └── ...
└── tests                      # Test suites directory
    └── ...

The pyproject.toml file in this structure is expected to live at the project root, so the project_root property will also look for that file as an indicator.

The uv package- and dependency-manager tool has several project-creation options, but all of them generate a pyproject.toml file, several will create a src directory, and none appear to generate a tests directory, so the defaults noted above will work for determining the source- and test-directory properties without modification, though the creation of a tests/unit directory in the project will be necessary.

Between those tools’ varied project-structure expectations and the src-layout and flat-layout structures that have been categorized with respect to their use and expectations for packaging with the setuptools package, it seems safe to assume that my preferred structure, the one shown above in the discussion of pipenv, is a reasonable default to expect, though I planned to allow configuration options to handle other structures. In any event, so long as there is a Pipfile, Pipfile.lock or pyproject.toml file in the project, that directory can be assumed to be the root directory of the project, and the src and tests/unit directories can be found easily enough from there. The basic implementation rules for this version’s test, then, boil down to:

• The root directory for project source code, modules and directories alike, will start at a directory named src at the project root.
• The root directory for project tests in general will start in a tests directory adjacent to the src directory (also in the project root), with unit tests living in a unit directory under that.

In that context, the test-method doesn’t really need to do much more than getting a collection of the source files, generating a collection of expected test files, then asserting that every test-file in that expected list exists. The final shape of the ExaminesProjectModules class can be diagrammed like this:

Tying the process of the test_source_modules_have_test_modules test-method to the class members in that diagram:

• test_source_modules_have_test_modules iterates over the members of the instance’s expected_test_entities, generating an expected-path Path value for each of those members under the instance’s test_dir, and asserting that the expected path exists.
• The expected_test_entities builds a path-string, with each directory- and module-name in the instance’s source_entities prefixed with the TEST_PREFIX attribute-value. For example, if there is a module at src/package/module.py, the expected_test_entities value will include a test_package/test_module.py.
• The source_entities property simply returns a set of path-strings for each module under the project’s source-directory, which is named in the SOURCE_DIR class-attribute, resolved in the instance’s source_dir property, and lives under the instance’s `project_root.
• The project_root property finds the project’s root directory Path by walking up the file-system from the location of the module that the test-case class lives in, and looking for any of the files noted earlier that indicate the root directory of a project at each iteration until it either finds one, or reaches the root of the file-system, raising an error if no viable root directory could be identified.
• The test_dir property resolves a Path under the project_root that is named in the TEST_DIR class attribute.
• The last remaining property, test_entities is actually not used in this process, and I toyed with the idea of removing it, but eventually decided against doing so because I’m planning to eventually make use of it. Functionally, it behaves in much the same way as the source_dir property, building a Path that resolves from the project_root to a subdirectory named in the TEST_DIR class-attribute.

The SOURCE_DIR, TEST_DIR and TEST_PREFIX class attributes are where project-specific configuration can be implemented for projects that use some other project-structure. A test-module in a project that doesn’t need to change those could be as simple as this:

import unittest

from goblinfish.testing.pact.projects import \
    ExaminesProjectModules

class test_ProjectTestModulesExist(
	unittest.TestCase, ExaminesProjectModules
):
    pass

if __name__ == '__main__':

    # Run tests locally using unittest to facilitate detailed
    # views of failures in subTest contexts.
    unittest.main()

If a project used a code directory for its source-code, and a unit-tests directory for its unit tests, and used test as the prefix for test-elements (instead of test_), handling those changes would only require the addition of three lines of class-attribute code:

import unittest

from goblinfish.testing.pact.projects import \
    ExaminesProjectModules

class test_ProjectTestModulesExist(
	unittest.TestCase, ExaminesProjectModules
):
    SOURCE_DIR = Path('code')
    TEST_DIR = Path('unit-tests')
    TEST_PREFIX = 'test'

if __name__ == '__main__':

    # Run tests locally using unittest to facilitate detailed
    # views of failures in subTest contexts.
    unittest.main()

That’s really all that would be needed in a project’s test-code in order to actively test that source modules have corresponding test-modules. When the test-suite containing this module is executed, it will generate separate failures for each test module that is expected but does not exist. For example, given a project with module.py and module2.py modules in it that do not have unit-test modules defined, the test would report both of those as missing if the test-suite was executed with unittest like so:

==============================================================
FAIL: test_source_modules_have_test_modules 
  (...test_source_modules_have_test_modules)
  [Expecting test_my_package/test_module.py test-module]
Test that source modules have corresponding test-modules
--------------------------------------------------------------
Traceback (most recent call last):

  File "/.../goblinfish/testing/pact/projects.py",
  	line 187, in test_source_modules_have_test_modules

AssertionError: False is not true :
    The expected test-module at test_my_package/test_module.py
    does not exist

...

AssertionError: False is not true :
    The expected test-module at test_my_package/test_module2.py
    does not exist
---------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (failures=2)

The same test-suite, run with pytest, reports one of those missing items (because those tests are in a unittest.TestCase.subTest context). If this failure were fixed, and the suite re-run, it would report a new failure for test_module2.py being missing. The output fron the first pytest run, trimmed down to just the relevant parts, is:

**================= short test summary info ==================**

FAILED /../tests/unit/test_my_package/
  test_project_test_modules_exist.py
    ::test_ProjectTestModulesExist
    ::test_source_modules_have_test_modules - 
AssertionError: False is not true :
    The expected test-module at test_my_package/test_module2.py
    does not exist

All that a developer needs to do to make this test pass is create the missing test-modules, the test_module.py and test_module2.py modules called out in the test-failure report above. As soon as those test-modules simply exist, the test is satisfied.

That pretty much wraps up this installment. The next version and article will focus on what needs to be done to make the same sort of test-process, running in the prescribed test-modules that this version requires, to apply a PACT approach to the members of the source modules.