The PACT for module-members

Tip

This version of the package was tagged as v.0.0.2 in the project repository, and can be examined there if desired. There are some issues with this version that I discovered while writing this article that will be corrected in the package version that was in progress at the time, but functionally the 0.0.2 version does what it was intended to do.

The next layer in of the PACT approach is concerned with prescribing required test-case classes for each member of a given source module. Given the goal that I mentioned in the first article in the series:

Goal

Every code-element in a project’s source tree should have a corresponding test-suite element.

…and this version’s focus on module members, it’s probably worth talking about what kinds of members this version is going to focus on. From a purely technical standpoint, there’s nothing that functionally prevents value-only members — attributes — from having tests written against them, but that is not my primary focus. What I’m concerned with, at least for now, are members that have actual code behind them, that do things, or represent things in the structure of the source code. In short, I’m concerned with testing functions and classes. With that in mind, the specific variation of the “every code-element” rule above for the purposes of this version of the package could be summarized as:

Goal

Every source function and class should have a corresponding test-case class.

For example, given a my_module.py with a my_function function, and a MyClass class, the goal here is to assert that the related test_my_module.py in the test-suite has test-case classes for each: test_my_function and test_MyClass. Following the same pattern mentioned in the previous article, the success or failure of the test provided by the mix-in class is simply no failed assertions that the expected test-case classes exist in the test-module that the test itself lives in.

At a high level, there are several similarities between this test-process and the previous version’s test: There is a class (ExaminesModuleMembers) that defines a single test-method (test_source_entities_have_test_cases) that actually executes the test in question, and there are several supporting properties that the mix-in class provides to facilitate that process. The relationships between those elements can be diagrammed as:

The overall process executed by the test_source_entities_have_test_cases test-method breaks down as follows:

• The test-method iterates over a collection of expected_test_entities names, asserting that each expected member exists, that each existing member is derived from unittest.TestCase, and that each existing member is also derived from another class that the PACT package provides (more on that later). Each of these iterations’ assertions happen inside a subTest context, allowing each individual potential failure to be captured and reported on independently from any others.
  
• The expected_test_entities property uses the source_entities property to define the base names for the expected classes, prepending each name with the TEST_PREFIX string to generate the final expected name for each member in the collection.
  
• It also uses the test_entities property in its iteration, which simply collects all of the member-names in the test_module.
• The source_entities property collects the names of all of the callable members of the target_module.
• The target_module simply returns the module designated by the namespace provided in the TARGET_MODULE class-attribute. This process uses functionality from the built-in importlib package to perform an actual import of the target module.
• Similarly, the test_module is just a reference to the actual test-module that the test is defined in.

The importlib functionality that is used is wrapped so that it can check to see whether the specified namespace import is already present: If it is, whether as a general import at the module level, or as a result of some other test-class executing a similar import, it avoids re-executing the import process, but still stores the imported result for access elsewhere.

Finding the various members of both the target_module and test_module is handled by functionality provided by the built-in inspect module. The basic acquisition of module-members is a simple call to the getmembers function it provides, and the class-vs.-function determination is handled by checking whether a given member of the relevant module is a class with the isclass function. In all of those contexts, if a given element (by name) is not a class, it has already been determined to be a callable, and is assumed to be a function instead.

The differentiation between classes and functions, for testing purposes, is eventually going to be handled by a pair of other classes, not yet defined in detail, that will encapsulate the test-process requirements for each. There are some differences between those source elements that make this distinction necessary, even if it’s only stubbed out at this point in the package’s code:

• A function’s tests will, ultimately, be concerned with requiring a “happy-paths” test-method for the function, which is expected to test all the rational subsets of “good” arguments.
• Tests for functions that have arguments will also be expected to provide “unhappy-path” tests for each individual parameter. That is, given a function with arg, *args, kwdonlyarg, and **kwargs parameters, the test-case for that function will be expected to have an unhappy-path test for arg, args, kwdonlyarg, and kwargs, testing for a reasonable subset of invalid argument-values for those parameters.
• Tests for classes, on the other hand, will need to account for test-cases for the methods of that class — which will follow the same basic rules as function-tests — as well as for properties and other data-descriptors.

Those points of differentiation do not need any real functionality behind them at this stage in development, but they do need to be accounted for. Following the naming convention that’s been established up to this point, those classes will be named ExaminesSourceClass and ExaminesSourceFunction, and they will live in an otherwise empty (for now) module_members.py module.

Taking all of those into account, the package structure in v.0.0.2 looks like this:

With just the changes made in this version, a pattern for manually implementing a full PACT test-suite is starting to emerge. With the previous version, all that needed to be done was to create the initial test-module, populate it with a bare-bones PACT-based test-case class, then run the test-suite, correct any failures reported, and repeat until there were no more failures. The same basic pattern can be applied at the module-members level, though it could get very tedious for larger code-bases that don’t have tests in place. Before that process starts, the the example described earlier, with the my_function function and MyClass class living in my_package/my_module.py, and the test_my_package/test_module.py test-module required looks something like this:

project-name/
├─ Pipfile
├─ Pipfile.lock
├─ .env
├─ src/
│   └─ my_package/
│      └─ module.py
│         ├─ ::MyClass
│         └─ ::my_function()
└─ tests/
    └─ unit/
        └─ test_my_package/
           ├─ test_project_test_modules_exist.py
           │  └─ ::test_ProjectTestModulesExist
           └─ test_module.py

Setting up the first test-case class, deriving from unittest.TestCase and the newly-minted ExaminesModuleMembers, like this:

class test_ProjectTestMembersExist(
    unittest.TestCase,
    ExaminesModuleMembers
):
    TARGET_MODULE='my_package.module'

…then running the revised test-module, either directly from the IDE, or by re-running the suite from the command-line with…

# Assuming pipenv is in play, omit "pipenv run" if not...
pipenv run python -m unittest discover -s tests/unit/test_my_package

…yields two failures, one for each of the missing-but-expected test-case classes that need to be defined. The relevant lines of the test-failure outputs are:

=================================================================
...
AssertionError: False is not true :
    No test-case class named test_Class is defined in
    test_module.py
=================================================================
...
AssertionError: False is not true :
    No test-case class named test_Function is defined in
    test_module.py

Tip

The test-suite could just as easily be run with pytest, but, while it will run all of the subTest-context tests, it will only report on the first failure in one of those contexts: The test_Function failure, in this case.

Adding a bare-bones test-case class for any single failure reported, like this example handling the missing test_Function test-case:

class test_Function(unittest.TestCase, ExaminesSourceFunction):
    pass

…will resolve that failure, leaving only:

=================================================================
...
AssertionError: False is not true :
    No test-case class named test_Class is defined in
    test_module.py

The test-process, as noted earlier, will also assert that an expected test-case class is derived from both unittest.TestCase and one of the ExaminesSource* classes. So, for example, starting with class that has neither of those requirements:

class test_Class:
    pass

…will raise a test failure looking like this when the test is run:

=================================================================
AssertionError: False is not true :
    The test_Class class is exepcted to be a subclass of
    unittest.TestCase, but is not:
    (<class 'test_module.test_Class'>, <class 'object'>)

Adding the missing requirement:

class test_Class(unittest.TestCase):
    pass

…and re-running the test will raise a different failure, intended to require that the relevant ExaminesSource* class will also be in place:

=================================================================
AssertionError: False is not true :
    The test_Class class is exepcted to be a subclass of
    ExaminesSourceClass, 
    but is not: (<class 'test_module.test_Class'>,
    <class 'unittest.case.TestCase'>, <class 'object'>)

Once all of the required parent classes are in place, though, these tests will pass.

Tip

Obviously, knowing that any given test-case class will need to be defined with both of the relevant parent classes will speed things along substantially. In cases where that is not known ahead of time — perhaps in conjunction with an AI agent like Claude Code, for example — the intent was to provide enough information that a developer completely unfamiliar with the PACT requirements, or an assistant system that needs that information, can iterate against failures until they are all resolved.

By the time all of the required test-cases are in place in the example project, it looks like this:

project-name/
├─ Pipfile
├─ Pipfile.lock
├─ .env
├─ src/
│   └─ my_package/
│      └─ module.py
│         ├─ ::MyClass
│         └─ ::my_function()
└─ tests/
    └─ unit/
       └─ test_my_package/
          ├─ test_project_test_modules_exist.py
          │  └─ ::test_ProjectTestModulesExist
          └─ test_module.py
             ├─ ::test_MyClass
             └─ ::test_my_function

So, at this point, the PACT test-processes can identify and require missing test-modules (from the previous version), and missing test-case classes for source-module members that those test-modules relate to. The process for implementing the complete set of required tests is growing in complexity, and may be painful (but at least be tedious) for large bodies of source-changes that need to be accounted for. The tests put in place still make no assumptions about whether they should be executed: it would be quite possible to apply any of the various unittest.skip* decorators decorators, or perhaps the expectedFailure decorator for certain types of test contexts, without changing the pass/fail of the test itself. At this level, skipping the PACT tests would involve skipping all of the tests in the relevant test-case class, though — that seems an unlikely need, but it’s worth bearing in mind should the need arise.

The configuration involved in the ExaminesModuleMembers class has only briefly been touched on with the mention of the TARGET_MODULE and TEST_PREFIX class-attributes noted earlier. The TARGET_MODULE attribute must be defined in order for the import-process that provides target_module, and the source_entities that are retrieved from that module. The TEST_PREFIX, like its counterpart in the ExaminesProjectModules class previously defined and discussed in the previous article, defines the prefix for the names of the test-case classes. It has the same default value: test_.

This layer, captured in the 0.0.2 version of the package, handles the detection of expected test-case classes, as well as asserting that they are of an appropriate type, for each member of a source module that testing is concerned with. The next layer in, which will be covered in the 0.0.3 version and related article here, will concern itself with requiring test-methods in the ExaminesSourceClass and ExaminesSourceFunction classes that were stubbed out here.