The PACT for functions

Note

This version of the package was tagged as v.0.0.3 in the project repository, and can be examined there if desired. This post will cover only part of the changes in that version, with the balance in the next post.

The next layer in from the module-members testing discussed in the previous article is focused on testing that the members identified there have all of the expected test-methods. As noted there, this testing-layer is concerned with callable members: functions and classes, specifically. Beyond the fact that they are both callable types, there are significant differences, which is why the wrappers for testing functions and classes were broken out into the two classes that were stubbed out earlier: the ExaminesSourceClass and ExaminesSourceFunction classes.

The fundamental difference that has to be accounted for that led to those two classes being defined is in how those member-types are called, and what happens when they are called. In the case of a function, the resulting output is completely arbitrary, at least from the standpoint of testing. A function accepts some collection of arguments, defined by its signature parameters, does whatever it’s going to do with those, and returns something, even if that return value is None.

A class, on the other hand, is always expected to return an instance of the class — an object — when it is called. That instance will have its own members, which are equally arbitrary, but can include both methods (which are essentially functions) and properties. It’s important to note that these properties have code behind them that make them work — they are a specific type of built-in data descriptor type, with methods that are automatically recognized and called by the Python interpreter when a get, set, or delete operation is called against the property or descriptor of a given object.

Class attributes, without any backing logic or code, fall into the same sort of testing category that module attributes do, also noted in a previous article. Specifically, while it’s absolutely functionally possible to test a class attribute, that attribute is, by definition, mutable: There is nothing preventing user code from altering or even deleting a class attribute, or that attribute as it is accessed through a class instance. Since the primary focus of the PACT testing idea is testing the contracts of test-targets, and attributes are as mutable as they are, they really cannot be considered as “contract” elements of a class.

After thinking all of those factors through, I decided that my next step, the first new code that would be added to this version of the package, would be focused on testing functions. The main thought behind that decision is that establishing both the rule-sets and implementation patterns for function-test requirements would provide most of the rules and implementation patterns for class-members as well: Methods of classes are just functions with an expected scope argument (self or cls), and properties and other data descriptor implementations are just classes with a known set of methods. On top of that, functions implicitly have contracts, represented by their input parameters and output expectations, so accounting for those in defining the test-expectations for functions would carry over to methods and properties later.

The test-expectations that I landed on after thinking through all of that boiled down to:

Goal

1. Every function should have a corresponding happy-paths test method, testing all permutations of a rational set of arguments across the parameters of the function.
2. Every function should also have a corresponding unhappy-path test-method for each parameter.

The goals for each of these test-expectations are still similar to the goals for test-expectations in previous versions’ tests: To assert that a given, expected test-entities exist for each source-entity. Because functions have their own child entities — the parameters that they expect — those expectations need to account for those parameter variations in some manner. My choice of these two was based on a couple of basic ideas: The happy-path test for a given function is, ultimately, intended to prove that the function is behaving as expected when it is called in an expected fashion. I fully expect that happy-path tests might be fairly long, testing a complete, rational subset of “good” parameter values across all of the logical permutations that a given function will accept. I do not expect that happy-path tests would need to be (or benefit from being) broken out into separate tests for each general permutation-type, though.

The unhappy-path tests, to my thinking, should build on the happy-path test processes as much as possible. Specifically, I’m intending that each unhappy-path test will use a happy-path argument-set as a starting-point for its input to the target function, but replace one of the arguments with a “bad” value for each rational type or value that can be considered “bad.”

An example seems apropos here, since I haven’t been able to come up with a more concise way to describe my intentions without resorting to code. Consider the following function:

def send_email(address: str, message: str, *attachments: dict):
    """
    Sends an email message.

    Parameters:
    -----------
    address : str
        The address to send the message to
    message : str
        The message to send. May be empty.
	attachments : dict
        A collection of attachment specs, providing a
        header-name (typically "Content-Disposition")
        and value ("attachment"), a filename (str, or
        tuple with encoding specifications), and a
        file pointer to the actual file to be attached.
	"""
	# How this function actually works is not relevant
    # at this point
	...

The happy-path test-method for this function, test_send_email_happy_paths, would be expected to call the send_email function with both a general email address and a “mailbox” variant (john.smith@test.com and John Smith <john.smith@test.com>), with both an empty message, and a non-empty one, and with zero, one, and two attachments arguments. That’s a dozen variations, but they should be relatively easy to iterate through, even if they have to be split out in the test-code, whether for readability, or to check that some helper function was called because of circumstances for a given call to the target function (for example, an attachment-handler sub-process).

The unhappy-path tests, and what they use for their arguments break out in more detail based on which “unhappy” scenario they are intended to test:

• test_send_email_bad_address would be expected to test for invalid email address values, and possibly for non-string types if there was type-checking involved in the processing for them, but could use the same message and attachments values for each of those checks.
• test_send_email_bad_message could use the same address and attachments values, since it would be concerned with testing the value and/or type of some invalid message arguments.
• test_send_email_bad_attachments could use any valid address and message values, as it would be testing for invalid attachments elements.

As with the test-expectations established in previous versions, the test-process for functions is only concerned with asserting that all of the expected test-methods exist. That is a hard requirement that’s implemented by application of the pact testing mix-ins. The intention behind that is to promote implementation of test-methods when possible/necessary, or to promote them being actively skipped, with documentation as to the reason why it is being skipped. That’s worth calling out, I think:

Warning

The pact processes do not prevent a developer from creating an expected test-method that simply passes. That risk should be mitigated by application of some basic testing discipline, or by establishing a standard for tests that are not implemented!

Using the send_email function as an example, and actively skipping tests for various reasons with the unittest.skip decorator, the test-case class might initially look something like this, after implementing the happy-paths tests that were deemed more important:

class test_send_email(unittest.TestCase, ExaminesSourceFunction):
    """Tests the send_email function"""

    def test_send_email_happy_paths(self):
        """Testing send_email happy paths"""
        # Actual test-code omitted here for brevity

    @unittest.skip('Not implemented, not a priority yet')
    def test_send_email_bad_address(self):
        """Testing send_email with bad address values"""
        self.fail(
            'test_send_email.test_send_email_bad_address '
            'was initially skipped, but needs to be '
            'implemented now.'
        )

    @unittest.skip('Not implemented, not a priority yet')
    def test_send_email_bad_message(self):
        """Testing send_email with bad message values"""
        self.fail(
            'test_send_email.test_send_email_bad_message '
            'was initially skipped, but needs to be '
            'implemented now.'
        )

    @unittest.skip('Not implemented, not a priority yet')
    def test_send_email_bad_attachments(self):
        """Testing send_email with bad attachments values"""
        self.fail(
            'test_send_email.test_send_email_bad_attachments'
            'was initially skipped, but needs to be '
            'implemented now.'
        )

This approach keeps the expected tests defined, but they will be skipped, and if that skip decorator is removed, they will immediately start to fail. Since the skip decorator requires a reason to be provided, and that reason will appear in test outputs, there is an active record in the test-code itself of why those tests have been skipped, and they will appear in the test logs/output every time the test-suite is run.

That covers what the goal is, in some detail. The implementation, how it works is similar, in many respects, to other test-case mix-ins already in the package from previous versions. For function testing, the new code was all put in place in the ExaminesSourceFunction class that was stubbed out in v.0.0.2. With the implementation worked out, that class’ members can be diagrammed like this:

As with previous mix-ins, the entire process starts with the test-method that the mix-in provides, test_source_function_has_expected_test_methods, and the process breaks out as:

• test_source_function_has_expected_test_methods compares its collection of expected_test_entities against the actual test_entities collection, causing a test failure if any expected test-methods in the first do not exist in the second.
• The expected_test_entities collection is built using the target_function to retrieve the name of the function and its parameters, along with the TEST_PREFIX, HAPPY_SUFFIX and INVALID_SUFFIX class attributes, which provide the test_ prefix for each method, the happy-path suffix for that test-method, and invalid-parameter suffixes for each parameter in the target_function parameter-set.
• The target_function is retrieved using the name specified in the TARGET_FUNCTION class attribute, finding that function in the target_module, which is imported using the namespace identified in the TARGET_MODULE class-attribute.
• The test_entities method-name set is simply retrieved from the class, using the TEST_PREFIX to assist in filtering those members.

Many of the defaults for the various class attributes have already been discussed in previous posts about earlier versions of the package. The new ones, specific to the ExaminesSourceFunction class are shown in the class diagram for the package at this point:

• The HAPPY_SUFFIX, used to indicate a happy-paths test-method, defaults to '_happy_paths';
• The INVALID_SUFFIX, is used to append a '_bad_{argument}' value to unhappy-path test-methods, where the {argument} is replaced with the name of the parameter for that test-method. For example, the address, message, and attachments parameters/arguments noted earlier in the example for the send_email function.
• The TARGET_FUNCTION provides the name of the function being tested, which is used to retrieve it from the target_module, which behaves in the same fashion as the property by the same name in the ExaminesModuleMembers mix-in from v.0.0.2.

When v.0.0.2 was complete, the example project and its tests ended up like this:

project-name/
├─ Pipfile
├─ Pipfile.lock
├─ .env
├─ src/
│   └─ my_package/
│      └─ module.py
│         ├─ ::MyClass
│         └─ ::my_function()
└─ tests/
    └─ unit/
       └─ test_my_package/
          ├─ test_project_test_modules_exist.py
          │  └─ ::test_ProjectTestModulesExist
          └─ test_module.py
             ├─ ::test_MyClass
             └─ ::test_my_function

With a bare-bones my_function implementation like this:

def my_function():
    pass

…running the test_my_function test-case class, or the entire test-suite, immediately starts reporting a missing test-method:

================================================================
FAIL: test_source_function_has_expected_test_methods
...
[Verifying that test_my_function.test_my_function_happy_paths
exists as a test-method]
----------------------------------------------------------------
...
AssertionError: False is not true :
    Missing expected test-method - test_my_function_happy_paths
----------------------------------------------------------------

Adding the required test-method, being sure to use the skip-and-fail pattern shown earlier, like this:

class test_my_function(unittest.TestCase, ExaminesSourceFunction):
    TARGET_MODULE='my_package.module'
    TARGET_FUNCTION='my_function'

    @unittest.skip('Not yet implemented')
    def test_my_function_happy_paths(self):
        self.fail(
            'test_my_function.test_my_function_happy_paths '
            'was initially skipped, but needs to be implemented now.'
        )

…allows the test-case to run successfully, skipping that test-method in the process, and reporting the reason for the skip, provided that the test-run is sufficiently verbose:

test_my_function_happy_paths
    (test_my_function.test_my_function_happy_paths)
    skipped 'Not yet implemented'

If the function is altered, adding a positional argument, an argument-list, a keyword-only argument, and a typical keyword-arguments parameter, like so:

def my_function(arg, *args, kwonlyarg, **kwargs):
    pass

…then the test-expectations pick up the new parameters, and raise new test failures, one for each:

================================================================
FAIL: test_source_function_has_expected_test_methods 
...
[Verifying that test_my_function.test_my_function_bad_args
exists as a test-method]
...
----------------------------------------------------------------
...
AssertionError: False is not true :
    Missing expected test-method - test_my_function_bad_args
...
================================================================
...
[Verifying that test_my_function.test_my_function_bad_kwonlyarg
exists as a test-method]
...
----------------------------------------------------------------
...
AssertionError: False is not true :
    Missing expected test-method - test_my_function_bad_kwonlyarg
...
================================================================
...
[Verifying that test_my_function.test_my_function_bad_kwargs
exists as a test-method]
...
----------------------------------------------------------------
...
AssertionError: False is not true :
    Missing expected test-method - test_my_function_bad_kwargs
...
================================================================
...
[Verifying that test_my_function.test_my_function_bad_arg
exists as a test-method]
...
----------------------------------------------------------------
...
AssertionError: False is not true :
    Missing expected test-method - test_my_function_bad_arg
...
----------------------------------------------------------------

Adding those expected test-methods, unsurprisingly, allows the tests to pass, reporting on the skipped test-methods in the same manner as shown earlier:

class test_my_function(unittest.TestCase, ExaminesSourceFunction):
    TARGET_MODULE='my_package.module'
    TARGET_FUNCTION='my_function'

    @unittest.skip('Not yet implemented')
    def test_my_function_bad_arg(self):
        self.fail(
            'test_my_function.test_my_function_bad_arg '
            'was initially skipped, but needs to be '
            'implemented now.'
        )

    @unittest.skip('Not yet implemented')
    def test_my_function_bad_args(self):
        self.fail(
            'test_my_function.test_my_function_bad_args '
            'was initially skipped, but needs to be '
            'implemented now.'
        )

    @unittest.skip('Not yet implemented')
    def test_my_function_bad_kwargs(self):
        self.fail(
            'test_my_function.test_my_function_bad_kwargs '
            'was initially skipped, but needs to be '
            'implemented now.'
        )

    @unittest.skip('Not yet implemented')
    def test_my_function_bad_kwonlyarg(self):
        self.fail(
            'test_my_function.test_my_function_bad_kwonlyarg '
            'was initially skipped, but needs to be '
            'implemented now.'
        )

    @unittest.skip('Not yet implemented')
    def test_my_function_happy_paths(self):
        self.fail(
            'test_my_function.test_my_function_happy_paths '
            'was initially skipped, but needs to be '
            'implemented now.'
        )

When run, the test-case for the function reports the skipped methods, and their reasons, as expected:

test_my_function_bad_arg
    (test_my_function.test_my_function_bad_arg)
    skipped 'Not yet implemented'

test_my_function_bad_args
    (test_my_function.test_my_function_bad_args)
    skipped 'Not yet implemented'

test_my_function_bad_kwargs
    (test_my_function.test_my_function_bad_kwargs)
    skipped 'Not yet implemented'

test_my_function_bad_kwonlyarg
    (test_my_function.test_my_function_bad_kwonlyarg)
    skipped 'Not yet implemented'

test_my_function_happy_paths
    (test_my_function.test_my_function_happy_paths)
    skipped 'Not yet implemented'

After these additions, this example project looks like this:

project-name/
├─ Pipfile
├─ Pipfile.lock
├─ .env
├─ src/
│   └─ my_package/
│      └─ module.py
│         ├─ ::MyClass
│         └─ ::my_function(arg, *args, kwonlyarg, **kwargs)
└─ tests/
    └─ unit/
       └─ test_my_package/
          ├─ test_project_test_modules_exist.py
          │  └─ ::test_ProjectTestModulesExist
          └─ test_module.py
             ├─ ::test_MyClass
             └─ ::test_my_function
                ├─ ::test_my_function_bad_arg
                ├─ ::test_my_function_bad_args
                ├─ ::test_my_function_bad_kwargs
                ├─ ::test_my_function_bad_kwonlyarg
                └─ ::test_my_function_happy_paths

While the equivalent test-processes for class-members still needs to be implemented, there are already significant gains at this point in prescribing tests for functions. The fact that all of the types of class-members that an active contract testing process really needs to care about are, themselves, just variations of functions means that the processes implemented for function-testing will at least provide a baseline for implementing class-member test expectations. They may even use the exact same code and processes. That said, this post is long enough already, so the implementation and discussion of the class-member pact processes will wait until the next post.

The PACT for module-members

Tip

This version of the package was tagged as v.0.0.2 in the project repository, and can be examined there if desired. There are some issues with this version that I discovered while writing this article that will be corrected in the package version that was in progress at the time, but functionally the 0.0.2 version does what it was intended to do.

The next layer in of the PACT approach is concerned with prescribing required test-case classes for each member of a given source module. Given the goal that I mentioned in the first article in the series:

Goal

Every code-element in a project’s source tree should have a corresponding test-suite element.

…and this version’s focus on module members, it’s probably worth talking about what kinds of members this version is going to focus on. From a purely technical standpoint, there’s nothing that functionally prevents value-only members — attributes — from having tests written against them, but that is not my primary focus. What I’m concerned with, at least for now, are members that have actual code behind them, that do things, or represent things in the structure of the source code. In short, I’m concerned with testing functions and classes. With that in mind, the specific variation of the “every code-element” rule above for the purposes of this version of the package could be summarized as:

Goal

Every source function and class should have a corresponding test-case class.

For example, given a my_module.py with a my_function function, and a MyClass class, the goal here is to assert that the related test_my_module.py in the test-suite has test-case classes for each: test_my_function and test_MyClass. Following the same pattern mentioned in the previous article, the success or failure of the test provided by the mix-in class is simply no failed assertions that the expected test-case classes exist in the test-module that the test itself lives in.

At a high level, there are several similarities between this test-process and the previous version’s test: There is a class (ExaminesModuleMembers) that defines a single test-method (test_source_entities_have_test_cases) that actually executes the test in question, and there are several supporting properties that the mix-in class provides to facilitate that process. The relationships between those elements can be diagrammed as:

The overall process executed by the test_source_entities_have_test_cases test-method breaks down as follows:

• The test-method iterates over a collection of expected_test_entities names, asserting that each expected member exists, that each existing member is derived from unittest.TestCase, and that each existing member is also derived from another class that the PACT package provides (more on that later). Each of these iterations’ assertions happen inside a subTest context, allowing each individual potential failure to be captured and reported on independently from any others.
  
• The expected_test_entities property uses the source_entities property to define the base names for the expected classes, prepending each name with the TEST_PREFIX string to generate the final expected name for each member in the collection.
  
• It also uses the test_entities property in its iteration, which simply collects all of the member-names in the test_module.
• The source_entities property collects the names of all of the callable members of the target_module.
• The target_module simply returns the module designated by the namespace provided in the TARGET_MODULE class-attribute. This process uses functionality from the built-in importlib package to perform an actual import of the target module.
• Similarly, the test_module is just a reference to the actual test-module that the test is defined in.

The importlib functionality that is used is wrapped so that it can check to see whether the specified namespace import is already present: If it is, whether as a general import at the module level, or as a result of some other test-class executing a similar import, it avoids re-executing the import process, but still stores the imported result for access elsewhere.

Finding the various members of both the target_module and test_module is handled by functionality provided by the built-in inspect module. The basic acquisition of module-members is a simple call to the getmembers function it provides, and the class-vs.-function determination is handled by checking whether a given member of the relevant module is a class with the isclass function. In all of those contexts, if a given element (by name) is not a class, it has already been determined to be a callable, and is assumed to be a function instead.

The differentiation between classes and functions, for testing purposes, is eventually going to be handled by a pair of other classes, not yet defined in detail, that will encapsulate the test-process requirements for each. There are some differences between those source elements that make this distinction necessary, even if it’s only stubbed out at this point in the package’s code:

• A function’s tests will, ultimately, be concerned with requiring a “happy-paths” test-method for the function, which is expected to test all the rational subsets of “good” arguments.
• Tests for functions that have arguments will also be expected to provide “unhappy-path” tests for each individual parameter. That is, given a function with arg, *args, kwdonlyarg, and **kwargs parameters, the test-case for that function will be expected to have an unhappy-path test for arg, args, kwdonlyarg, and kwargs, testing for a reasonable subset of invalid argument-values for those parameters.
• Tests for classes, on the other hand, will need to account for test-cases for the methods of that class — which will follow the same basic rules as function-tests — as well as for properties and other data-descriptors.

Those points of differentiation do not need any real functionality behind them at this stage in development, but they do need to be accounted for. Following the naming convention that’s been established up to this point, those classes will be named ExaminesSourceClass and ExaminesSourceFunction, and they will live in an otherwise empty (for now) module_members.py module.

Taking all of those into account, the package structure in v.0.0.2 looks like this:

With just the changes made in this version, a pattern for manually implementing a full PACT test-suite is starting to emerge. With the previous version, all that needed to be done was to create the initial test-module, populate it with a bare-bones PACT-based test-case class, then run the test-suite, correct any failures reported, and repeat until there were no more failures. The same basic pattern can be applied at the module-members level, though it could get very tedious for larger code-bases that don’t have tests in place. Before that process starts, the the example described earlier, with the my_function function and MyClass class living in my_package/my_module.py, and the test_my_package/test_module.py test-module required looks something like this:

project-name/
├─ Pipfile
├─ Pipfile.lock
├─ .env
├─ src/
│   └─ my_package/
│      └─ module.py
│         ├─ ::MyClass
│         └─ ::my_function()
└─ tests/
    └─ unit/
        └─ test_my_package/
           ├─ test_project_test_modules_exist.py
           │  └─ ::test_ProjectTestModulesExist
           └─ test_module.py

Setting up the first test-case class, deriving from unittest.TestCase and the newly-minted ExaminesModuleMembers, like this:

class test_ProjectTestMembersExist(
    unittest.TestCase,
    ExaminesModuleMembers
):
    TARGET_MODULE='my_package.module'

…then running the revised test-module, either directly from the IDE, or by re-running the suite from the command-line with…

# Assuming pipenv is in play, omit "pipenv run" if not...
pipenv run python -m unittest discover -s tests/unit/test_my_package

…yields two failures, one for each of the missing-but-expected test-case classes that need to be defined. The relevant lines of the test-failure outputs are:

=================================================================
...
AssertionError: False is not true :
    No test-case class named test_Class is defined in
    test_module.py
=================================================================
...
AssertionError: False is not true :
    No test-case class named test_Function is defined in
    test_module.py

Tip

The test-suite could just as easily be run with pytest, but, while it will run all of the subTest-context tests, it will only report on the first failure in one of those contexts: The test_Function failure, in this case.

Adding a bare-bones test-case class for any single failure reported, like this example handling the missing test_Function test-case:

class test_Function(unittest.TestCase, ExaminesSourceFunction):
    pass

…will resolve that failure, leaving only:

=================================================================
...
AssertionError: False is not true :
    No test-case class named test_Class is defined in
    test_module.py

The test-process, as noted earlier, will also assert that an expected test-case class is derived from both unittest.TestCase and one of the ExaminesSource* classes. So, for example, starting with class that has neither of those requirements:

class test_Class:
    pass

…will raise a test failure looking like this when the test is run:

=================================================================
AssertionError: False is not true :
    The test_Class class is exepcted to be a subclass of
    unittest.TestCase, but is not:
    (<class 'test_module.test_Class'>, <class 'object'>)

Adding the missing requirement:

class test_Class(unittest.TestCase):
    pass

…and re-running the test will raise a different failure, intended to require that the relevant ExaminesSource* class will also be in place:

=================================================================
AssertionError: False is not true :
    The test_Class class is exepcted to be a subclass of
    ExaminesSourceClass, 
    but is not: (<class 'test_module.test_Class'>,
    <class 'unittest.case.TestCase'>, <class 'object'>)

Once all of the required parent classes are in place, though, these tests will pass.

Tip

Obviously, knowing that any given test-case class will need to be defined with both of the relevant parent classes will speed things along substantially. In cases where that is not known ahead of time — perhaps in conjunction with an AI agent like Claude Code, for example — the intent was to provide enough information that a developer completely unfamiliar with the PACT requirements, or an assistant system that needs that information, can iterate against failures until they are all resolved.

By the time all of the required test-cases are in place in the example project, it looks like this:

project-name/
├─ Pipfile
├─ Pipfile.lock
├─ .env
├─ src/
│   └─ my_package/
│      └─ module.py
│         ├─ ::MyClass
│         └─ ::my_function()
└─ tests/
    └─ unit/
       └─ test_my_package/
          ├─ test_project_test_modules_exist.py
          │  └─ ::test_ProjectTestModulesExist
          └─ test_module.py
             ├─ ::test_MyClass
             └─ ::test_my_function

So, at this point, the PACT test-processes can identify and require missing test-modules (from the previous version), and missing test-case classes for source-module members that those test-modules relate to. The process for implementing the complete set of required tests is growing in complexity, and may be painful (but at least be tedious) for large bodies of source-changes that need to be accounted for. The tests put in place still make no assumptions about whether they should be executed: it would be quite possible to apply any of the various unittest.skip* decorators decorators, or perhaps the expectedFailure decorator for certain types of test contexts, without changing the pass/fail of the test itself. At this level, skipping the PACT tests would involve skipping all of the tests in the relevant test-case class, though — that seems an unlikely need, but it’s worth bearing in mind should the need arise.

The configuration involved in the ExaminesModuleMembers class has only briefly been touched on with the mention of the TARGET_MODULE and TEST_PREFIX class-attributes noted earlier. The TARGET_MODULE attribute must be defined in order for the import-process that provides target_module, and the source_entities that are retrieved from that module. The TEST_PREFIX, like its counterpart in the ExaminesProjectModules class previously defined and discussed in the previous article, defines the prefix for the names of the test-case classes. It has the same default value: test_.

This layer, captured in the 0.0.2 version of the package, handles the detection of expected test-case classes, as well as asserting that they are of an appropriate type, for each member of a source module that testing is concerned with. The next layer in, which will be covered in the 0.0.3 version and related article here, will concern itself with requiring test-methods in the ExaminesSourceClass and ExaminesSourceFunction classes that were stubbed out here.

The PACT for test-modules

Tip

This version of the package was tagged as v.0.0.1 in the project repository, and can be examined there if desired.

I summarized the goal that the Python package that this series of articles is concerned with in the first article in the series as:

Goal

Every code-element in a project’s source tree should have a corresponding test-suite element.

My first step in pursuing this goal, after some thinking on where I wanted to start, and how I wanted to proceed, was to start with the code-elements that could only be described as file-system entities: modules (files) and packages (directories). The corresponding test-elements rule above, with that focus, could be implemented as some test or set of tests that simply asserted that for every given file-system object in a project’s source-tree, there was a corresponding file-system object in the relevant test-suite of the project. For example, given a source module at my_package/module.py, there should be a test_my_package/test_my_module.py in the related test-suite, and if that test-module did not exist, that was grounds for a failure of the test in question. For the purposes of the testing implemented in this version, the goal above could be restated as

Goal

Every source module and directory should have a corresponding test module and directory.

I did not expect the basic mechanics of that test-process to be difficult to implement. Even though I added some layers of functionality for various interim steps in the processes needed to get to that point, that expectation panned out about as I expected. The most significant initial challenge, as it turned out, was figuring out how to identify a project’s root directory. To understand why that was significant, I’ll summarize where the final version ended up.

The projects.py module (goblinfish.testing.pact.projects) provides a single class, ExaminesProjectModules, that provides a “baked in” test-method (test_source_modules_have_test_modules) that performs the actual test. That method relies on a handful of properties that are built in to the class itself:

Everything in the process that eventually get executed by test_source_modules_have_test_modules starts with determining the project_root. There are several common project-structure variations, often depending on how project package dependencies and the project’s Python Virtual Environment (PVE) are managed.

When pipenv manages packages, there is no set project structure, but it is a pretty safe bet that the Pipfile and Pipfile.lock files will live at the project root, and the project_root property uses the presence of either of those as an indicator to determine the value for the property. A fairly detailed but typical project structure under pipenv will likely look much like this:

project-name/
├── Pipfile
├── Pipfile.lock
├── pyproject.toml             # Build system configuration (if applicable)
├── src/                       # Source code directory
│   └── project_name/          # Main package directory
│       └── ...                # Package modules and subdirectories
├── tests/                     # Test suites directory
│   ├── unit                   # Unit tests
│   │   └── test_project_name  # Test suite for the project starts here
│   └── ...                    # Other test-types, like integration, E2E, etc.
├── scripts/                   # Scripts directory (optional)
│   └── ...
├── data/                      # Data directory (optional)
│   └── ...
├── docs/                      # Documentation directory (optional)
│   └── ...
├── README.md                  # Project description
├── LICENSE                    # License file
└── .gitignore                 # Git ignore file

The poetry package- and dependency-manager tool, as of this writing, shows a different project structure created using its tools:

project-name
├── pyproject.toml
├── README.md
├── project_name               # This is the SOURCE_DIR
│   └── ...
└── tests                      # Test suites directory
    └── ...

The pyproject.toml file in this structure is expected to live at the project root, so the project_root property will also look for that file as an indicator.

The uv package- and dependency-manager tool has several project-creation options, but all of them generate a pyproject.toml file, several will create a src directory, and none appear to generate a tests directory, so the defaults noted above will work for determining the source- and test-directory properties without modification, though the creation of a tests/unit directory in the project will be necessary.

Between those tools’ varied project-structure expectations and the src-layout and flat-layout structures that have been categorized with respect to their use and expectations for packaging with the setuptools package, it seems safe to assume that my preferred structure, the one shown above in the discussion of pipenv, is a reasonable default to expect, though I planned to allow configuration options to handle other structures. In any event, so long as there is a Pipfile, Pipfile.lock or pyproject.toml file in the project, that directory can be assumed to be the root directory of the project, and the src and tests/unit directories can be found easily enough from there. The basic implementation rules for this version’s test, then, boil down to:

• The root directory for project source code, modules and directories alike, will start at a directory named src at the project root.
• The root directory for project tests in general will start in a tests directory adjacent to the src directory (also in the project root), with unit tests living in a unit directory under that.

In that context, the test-method doesn’t really need to do much more than getting a collection of the source files, generating a collection of expected test files, then asserting that every test-file in that expected list exists. The final shape of the ExaminesProjectModules class can be diagrammed like this:

Tying the process of the test_source_modules_have_test_modules test-method to the class members in that diagram:

• test_source_modules_have_test_modules iterates over the members of the instance’s expected_test_entities, generating an expected-path Path value for each of those members under the instance’s test_dir, and asserting that the expected path exists.
• The expected_test_entities builds a path-string, with each directory- and module-name in the instance’s source_entities prefixed with the TEST_PREFIX attribute-value. For example, if there is a module at src/package/module.py, the expected_test_entities value will include a test_package/test_module.py.
• The source_entities property simply returns a set of path-strings for each module under the project’s source-directory, which is named in the SOURCE_DIR class-attribute, resolved in the instance’s source_dir property, and lives under the instance’s `project_root.
• The project_root property finds the project’s root directory Path by walking up the file-system from the location of the module that the test-case class lives in, and looking for any of the files noted earlier that indicate the root directory of a project at each iteration until it either finds one, or reaches the root of the file-system, raising an error if no viable root directory could be identified.
• The test_dir property resolves a Path under the project_root that is named in the TEST_DIR class attribute.
• The last remaining property, test_entities is actually not used in this process, and I toyed with the idea of removing it, but eventually decided against doing so because I’m planning to eventually make use of it. Functionally, it behaves in much the same way as the source_dir property, building a Path that resolves from the project_root to a subdirectory named in the TEST_DIR class-attribute.

The SOURCE_DIR, TEST_DIR and TEST_PREFIX class attributes are where project-specific configuration can be implemented for projects that use some other project-structure. A test-module in a project that doesn’t need to change those could be as simple as this:

import unittest

from goblinfish.testing.pact.projects import \
    ExaminesProjectModules

class test_ProjectTestModulesExist(
	unittest.TestCase, ExaminesProjectModules
):
    pass

if __name__ == '__main__':

    # Run tests locally using unittest to facilitate detailed
    # views of failures in subTest contexts.
    unittest.main()

If a project used a code directory for its source-code, and a unit-tests directory for its unit tests, and used test as the prefix for test-elements (instead of test_), handling those changes would only require the addition of three lines of class-attribute code:

import unittest

from goblinfish.testing.pact.projects import \
    ExaminesProjectModules

class test_ProjectTestModulesExist(
	unittest.TestCase, ExaminesProjectModules
):
    SOURCE_DIR = Path('code')
    TEST_DIR = Path('unit-tests')
    TEST_PREFIX = 'test'

if __name__ == '__main__':

    # Run tests locally using unittest to facilitate detailed
    # views of failures in subTest contexts.
    unittest.main()

That’s really all that would be needed in a project’s test-code in order to actively test that source modules have corresponding test-modules. When the test-suite containing this module is executed, it will generate separate failures for each test module that is expected but does not exist. For example, given a project with module.py and module2.py modules in it that do not have unit-test modules defined, the test would report both of those as missing if the test-suite was executed with unittest like so:

==============================================================
FAIL: test_source_modules_have_test_modules 
  (...test_source_modules_have_test_modules)
  [Expecting test_my_package/test_module.py test-module]
Test that source modules have corresponding test-modules
--------------------------------------------------------------
Traceback (most recent call last):

  File "/.../goblinfish/testing/pact/projects.py",
  	line 187, in test_source_modules_have_test_modules

AssertionError: False is not true :
    The expected test-module at test_my_package/test_module.py
    does not exist

...

AssertionError: False is not true :
    The expected test-module at test_my_package/test_module2.py
    does not exist
---------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (failures=2)

The same test-suite, run with pytest, reports one of those missing items (because those tests are in a unittest.TestCase.subTest context). If this failure were fixed, and the suite re-run, it would report a new failure for test_module2.py being missing. The output fron the first pytest run, trimmed down to just the relevant parts, is:

**================= short test summary info ==================**

FAILED /../tests/unit/test_my_package/
  test_project_test_modules_exist.py
    ::test_ProjectTestModulesExist
    ::test_source_modules_have_test_modules - 
AssertionError: False is not true :
    The expected test-module at test_my_package/test_module2.py
    does not exist

All that a developer needs to do to make this test pass is create the missing test-modules, the test_module.py and test_module2.py modules called out in the test-failure report above. As soon as those test-modules simply exist, the test is satisfied.

That pretty much wraps up this installment. The next version and article will focus on what needs to be done to make the same sort of test-process, running in the prescribed test-modules that this version requires, to apply a PACT approach to the members of the source modules.

Prescribing Active Contract Testing (PACT) in Python

I’m going to start this series of articles by stating, for the record:

Note

I am very opinionated when it comes to testing code. Writing tests may not be my least favorite part of software engineering, though I absolutely acknowledge that tests in general, and unit tests in particular are a critical part of writing good, solid, usable code. To be clear, I don’t dislike writing tests, I just like that process less than writing documentation for code, and that less than writing the code itself, and I want the test processes themselves to do certain things to make writing tests easier, or at least more effective.

This series of articles is, ultimately, about a Python package — goblinfish-testing-pact — that I’ve written to try and improve my quality of life with respect to writing unit test suites. Along the way, I’ll discuss my discoveries and thought processes that led to the functionality provided by the package. I’m going to focus on unit testing here, though some aspects of the discussion here may also apply to other types of tests.

Why unit test anyway?

The simplest answer to this question is some variation of “ensuring that each executable component in a codebase performs as expected, accepting expected inputs, and returning expected results.” The majority of the time, that will map one-to-one with the idea of eliminating bugs from the code being tested, or at least reducing their likelihood. It’s still possible for code that is well and thoroughly tested to have bugs surface in it, though — it’s not a panacea for bug-free code. Writing tests so that they can be executed on demand, especially during whatever build/deploy processes — regression testing — also helps considerably to ensure that when changes are made to a codebase, no new bugs get introduced.

For me, that equates to spending less time hunting down bugs, freeing up more time to write new and interesting code, which aligns nicely with my preferences about how I spend the time I have available for development.

When I write tests, I want to be able to iterate over sets and collections of good and bad values both, and I want the results of those tests to be “non-blocking” for other tests against those collections. That is, if there are sixteen variations of arguments to pass, and variations 7 and 9 are going to fail, I want to see both of those failures at the same time, rather than having to run a test, fix #7, then run the test again to discover that #9 is failing too, and fix that. The built-in unittest package that ships with Python supports this, by allowing subTest context managers to run multiple related but independent tests within an iteration. While I prefer pytest for its test-discovery, which makes running test-suites quite a bit easier, unittest.TestCase.subTest is, to my thinking, a much better mechanism for organizing larger groups of related tests than anything that is offered by pytest out of the box.

I do not like the idea of having arbitrary code-coverage metrics that can impact the pass/fail of a build process. At the same time, the coverage package provides some very useful reporting capabilities on what lines of code in the source were not exercised by a test-suite, and I do like having that available as a sanity-check, to provide visibility into testing gaps. That segues neatly into an idea that I’ve been working on for many years now, that I’ve come to think of as prescribing active contract testing: Prescribing in the sense of stating, as a rule, that the active contracts of callables (their input parameter expectations) should all be tested. To that end, I’ve been working off and on for several years towards writing a package that implements those prescriptions, with a fairly simple basic rule:

Goal

Every code-element in a project’s source tree should have a corresponding test-suite element.

In practice, that can be elaborated a bit into some more specific rules for different types of code elements:

• Every source-code directory that contributes to an import-capable namespace should have a corresponding test-suite directory.
• Every source-code module should have a corresponding test-suite module.
• Every source-code callable module-member (function or class) should have a corresponding test-suite-module member.
• Every source-code class-member that either is a method, or has methods behind it that make it work (standard @property implementations, for example, as well as any classes that implement the standard data-descriptor interface, see the Properties implementation of the Descriptor HowTo Guide for an example) should have corresponding test-methods in the test-suite.
• Test-methods for source-code elements that accept arguments should include tests for both happy path scenarios, and for unhappy paths for each parameter.

These are, I feel, just testing based on logical extensions of the object-oriented idea of contracts (or interfaces) into the overall code-structure: a set of defined rules that a given code-element is expected to conform to. Each of the code-elements noted above has a contract of sorts associated with it:

• Modules (and their parent packages, where applicable) can be imported, as can their individual members. That implies, to my thinking, a contract that those packages, modules, and testing for those module-members should be accounted for.
• Every function accepts some number of parameters/arguments. That is another contract, expressing the input expectations for each of those code-elements, and test-entities for those expectations should be accounted for.
• Members of classes — methods and properties are, under the hood, just special cases of, or collections of special cases of functions, and follow the same test-entity accountability rules.

Note, if you will, my use of accounted for in that last list. Even a fairly small codebase whose test-suite follows these prescriptions could easily yield a very large number of test-entities. A single module, with two functions in it, that have, say, half a dozen parameters used between them, lands on none test-entities: One test-module, containing two test-case classes, two happy-path test-methods between them, and six unhappy-path test-methods, one for each of the arguments for either of those functions. I’ve been writing code for decades now, and testing it for more than half of that time, and in that time I’ve seen, firsthand, that not all tests are equally important from a product or service delivery perspective. Requiring that test-entities can be verified as existing, even if they are actively skipped provides what I believe to be a near-optimal balance between “testing everything” and real-world priorities. Even if test-methods or whole test-cases are skipped, they at least exist, and provided that the mechanism for skipping them requires some documentation — a simple string saying we chose not to test this because {some reason} — that encourages making conscious decisions about testing priorities.

So, what I’ve really been working towards might be described as a meta testing package: Something that analyzes the structure of the code (and of the corresponding test-suite code), and asserts that the “accounted for” rules for each entity are being followed, without having to care whether the test-entities involved are even implemented – just that they are accounted for.

The balance of the articles in this series will dive in to the implementations of those meta-testing processes at various levels. Here is what my plan is, broken out by version number:

• v.0.0.1 will contend with the source- and test-entities whose existence is a function of the structure of files and directories in the project: Packages (directories) and modules (files).
• v.0.0.2 will focus on determining whether the members within a given source module — the functions and classes contained — have corresponding test-case classes defined in the corresponding test-modules.
• v.0.0.3 will implement the accountability for test-methods within the test-case classes that were verified in the previous version.

Past that, I had originally planned for the v.0.0.4 version to incorporate the test-processes that the package provides as a logical next step, but after actually putting the package in to use, I started discovering minor little tweaks that needed to be made. Additional tweaks that surfaced as I was putting the package into use as part of the code for a book I’m writing — the second edition of my Hands-On Software Engineering with Python book — led to more tweaks, which I made and released as the v.0.0.5 version. After that, as I was working through some writing for the v.0.0.2 release, I noticed some copypasta-level mistakes, a general lack of attention that I’d paid to documentation within the code, and that I had never actually written, which were all added to the v.0.0.6 release.

That’s where things stood as I was writing this update to the original article post on LinkedIn. My next steps, after I get all the article content from the broken LinkedIn attempts copied over here (and updated where necessary) is going to pick up where I’d originally intended to go with the  v.0.0.4 version. While it may be possible to test incrementally as I go, my previous efforts with this package concept over the years has shown me that I will almost certainly spend a lot more time revising existing tests than I would spend writing them all from scratch once all the package source-elements are in place. I’m tentatively expecting that v.0.0.7 will really just involve implementing unit tests for the package, along with any changes that need to be accounted for as tests reveal issues.

After that, as things surface that need attention, I will revise the code and post new articles as needed. At some point in the foreseeable future, there will be a v.1.0 release, which may or may not warrant a dedicated article. At some point after that, after I’ve had some time to noodle on things, I’m planning to issue v1.1 and v.1.2 releases that will include a command-line tool for stubbing out tests that follow the strategies and expectations of the module, and support for testing Pydantic models, not necessarily in that order.

Logging Strategies for Observability and Troubleshooting

My first effort is a little Python package that provides a decorator/context-manager class that makes capturing and logging process execution times easier and more consistent. I picked this one to start with for a variety of reasons:

• The code involved was relatively minimal, so I could write it in a couple of afternoons, including testing, giving me time to focus on learning the newest standard processes involved in creating Python packages.
• That also allowed me to work through the CI/CD processes that I wanted to build for any future Python packaging projects, specifically Bitbucket’s Pipelines.
• It integrates nicely, I think, with standard AWS CloudWatch log ➔ metrics-filters ➔ dashboards/alarms processes and resources.
• It helps keep logs that are used to capture metrics consistent and more efficient (though in fairness, that efficiency gain is, perhaps, trivial).

The back-story behind this package is fairly straightforward, though I’ve abstracted the circumstances and details for various reasons. Assume that you have a system that runs in AWS. It could be an API Gateway backed by one to many Lambda Functions, a Fargate task, or anything, really, that has inner functions or methods that you need visibility into the successes and errors, and elapsed execution time of for every request. For the sake of illustration, let’s assume a Fargate task that: - Accepts some identifiers for data that needs to be retrieved from APIs that you have no control over. For purposes of illustration, I’ll call them Person, Place and Thing APIs. - Retrieves the actual data for the identified items from their relevant APIs. - Does some manipulation of the data to assemble it into a payload for a call to another API. - Makes that API call to analyze the collected data. - Writes that analysis to a data store.

Putting all of that together looks something like this:

There are, I think, two main categories of logging-related messages to be concerned with. The first is messages for troubleshooting purposes: information that would help an engineer track down which parts of a process are succeeding and which are failing. The approach that I have advocated for over the last several years is a fairly simple rule-set: - Entry-point functions or methods should info-log their initial call: my_entry_point called, for example. - Entry-point functions or methods should debug-log their inputs. In Python, this can be as simple as logging f'{vars()}' at the top of the code, but other, more sophisticated approaches are possible (or even desirable) too. - All functions or methods should log their exit state. That takes two basic forms: - For successful completion without errors, info-logging that is sufficient: my_child_function completed successfully, for example. In addition, debug-logging what will be returned is advised. - Any errors raised should be captured and logged, with the inputs that led to the error. Logging the stack-trace at a debug level is also advised.

When everything is running according to plan, that leads to one log-message per function or method called, plus one at the start of the entry-point, each of which is intentionally small to reduce the amount of logged data being paid for. For example (these examples are all asyncio-based code, unless noted otherwise):

[INFO]  12:05:57,008: main called
[INFO]  12:06:01,857: get_place_data completed without error
[INFO]  12:06:02,241: get_person_data completed without error
[INFO]  12:06:02,811: get_thing_data completed without error
[INFO]  12:06:05,387: analyze_data completed without error
[INFO]  12:06:05,748: write_data completed without error
[INFO]  12:06:05,748: async_main completed
[INFO]  12:06:05,750: main completed

This captures the entire process wrapped by the main function, and shows that all of the relevant sub-processes ran to completion without error. This is, if not the optimum level of logging information for troubleshooting purposes, at least very close to it. In situations where there are no explicit errors being raised, but the results of a process are suspect or just wrong, this level of logging can show at least things over and above the “there were no errors” scenario: - Whether the process as a whole was executing as expected. - Which child processes executed (if there are any), and which did not.

It also affords the ability to simply turn debug-level logging on to see more details:

[INFO]  14:08:08,408: main called
[DEBUG] 14:08:08,408: main: {}
[DEBUG] 14:08:08,408: async_main called: {}
[DEBUG] 14:08:11,206: get_person_data called: {}
[DEBUG] 14:08:11,206: get_place_data called: {}
[DEBUG] 14:08:11,206: get_thing_data called: {}

[INFO]  14:08:13,873: get_thing_data completed without error
[DEBUG] 14:08:13,874: {'thing_data': ('the Revolver', {})}

[INFO]  14:08:13,977: get_place_data completed without error
[DEBUG] 14:08:13,978: {'place_data': ('The Library', {})}

[INFO]  14:08:14,001: get_person_data completed without error
[DEBUG] 14:08:14,001: {'person_data': ('Professor Plum', {})}

[DEBUG] 14:08:14,001: analyze_data called: {
        'payload': {
	        'person_data': {'Professor Plum': {}},
	        'place_data': {'The Library': {}},
	        'thing_data': {'the Revolver': {}}}
        }
[INFO]  14:08:16,537: analyze_data completed without error
[DEBUG] 14:08:16,537: {
		'person_data': {
			'Professor Plum': {}},
			'place_data': {'The Library': {}},
			'thing_data': {'the Revolver': {}}
		}

[DEBUG] 14:08:16,537: write_data called: {
		'payload': {
			'person_data': {'Professor Plum': {}},
			'place_data': {'The Library': {}},
			'thing_data': {'the Revolver': {}}}
		}
[INFO]  14:08:17,145: write_data completed without error
[DEBUG] 14:08:17,145: True

[INFO]  14:08:17,145: async_main completed
[INFO]  14:08:17,146: main completed

In situations where errors arise, this approach will usually log the source error, any “intermediate” errors that happen as a result, and a “final” error, making the process of tracking down the source of the error, and the inputs that led to it, relatively simple:

[INFO]  14:02:13,003: main called
[ERROR] 14:02:15,672: RuntimeError in get_thing_data: with {}:
                      Thing API failure
[INFO]  14:02:18,108: get_person_data completed without error
[INFO]  14:02:18,294: get_place_data completed without error
[ERROR] 14:02:18,295: RuntimeError in async_main: 1 child processes failed: 
                      [RuntimeError('Thing API failure')]
[INFO]  14:02:18,296: main completed

This still provides a reasonable view of a process’ execution — usually enough for an engineer to be able to replicate the error locally, since the inputs that caused the error are captured. But, if more detail is needed, the same switch over to debug-level logging can provide more detail.

The second main category of logging messages is concerned with providing data-points that can be picked up by metrics filters, and used to provide observability into the performance of a system or its process. In an AWS CloudWatch context, these data points can be collected with a Metric Filter, and those collected metrics can drive displays in CloudWatch Dashboards and be used to set up Alarms that can be used to notify someone when performance is outside an expected range or passing some value threshold. The metric filter setup uses one of several filter syntaxes to identify names in the associated log group, and defines how the value for that key is determined.

If storage of logs is an ongoing cost concern, note that the metrics extracted from logs persist after the logs they were extracted from have been deleted. That allows the log retention policy to be set for a much shorter timeframe — just long enough that the passage of a long weekend won’t completely eliminate them, for example — without affecting the metrics extracted in any meaningful fashion.

The metrics themselves will be captured in one-minute buckets by default, and over time it appears that those buckets are merged into hourly data-sets. I’ve observed this, but cannot confirm whether the higher resolution time scale is really not available, or determine what the age is when that happens. I’m reasonably certain, from memory, that it’s over a month, and it seems likely that it’s closer to three months.

One of the more efficient logging output formats, from a metric filter standpoint, is to log data as a JSON object, with fields and values that filter definitions can leverage. By way of example, capturing the execution times of the functions in the hypothetical Fargate task above might issue a log-message with the following values (pretty-printed for ease of reading; there’s no need for it to be pretty in the log message itself):

{
	"async_main": 2.4285507202148438,
	"get_place_data": 2.770256280899048,
	"get_person_data": 2.859272003173828,
	"get_thing_data": 2.9412269592285156,
	"async_results": 2.9431209564208984,
	"analyze_data": 2.139224052429199,
	"write_data": 0.7172739505767822,
	"main": 8.23118805885315
}

Functionally, there’s no reason that these metrics values would have to be logged in a single line: Logging each value independently is (trivially) less efficient, though, and in my experience, troubleshooting efforts that need access to the raw metrics data are easier to deal with if they are all in one place.

With all of those factors in mind, I wrote a small Python package to capture elapsed time values with associated names, and write the entire collection of them out to a single log-line: the # goblinfish-metrics-trackers package. It also provides a mechanism for adding arbitrary metrics to that output, allowing a single log-message to capture both “latencies” (elapsed times for any processes of interest) and other points of interest (like error-counts). To reduce the amount of data storage in the logs and allow for better consistency in the metrics filters later on, the elapsed times are output as milliseconds, with no more than three decimal places of precision (e.g. 1234.567). An example of the final output for the Fargate structure above, pretty-printed, is:

{
   "latencies":{
      "async_main":2869.3,
      "get_place_data":2324.621,
      "get_thing_data":2427.277,
      "get_person_data":2871.321,
      "async_results":2872.099,
      "analyze_data":2199.275,
      "main":7942.849
   },
   "metrics":{
      "write_data_errors":1
   }
}

The corresponding general logging output for the test-run that generated this log message was:

[INFO]  09:38:18,861: main called
[INFO]  09:38:23,246: get_thing_data completed without error
[INFO]  09:38:23,545: get_place_data completed without error
[INFO]  09:38:23,859: get_person_data completed without error
[INFO]  09:38:26,426: analyze_data completed without error
[ERROR] 09:38:26,426: RuntimeError in write_data: Write data failure
[ERROR] 09:38:26,426: RuntimeError in async_main: Write data failure
[INFO]  09:38:26,427: main completed

The function that raised the error (an asynchronous function in this case), incorporating the logging policy described earlier, using a WRITE_FAIL_RATE value of 0.99 to force a fake error, and capturing that error metric was:

# Module "Constants" and Other Attributes
tracker = ProcessTracker()

# ...

async def write_data(payload):
    inputs = vars()
    logging.debug(f'write_data called: {inputs}')
    try:
        if random.random() < WRITE_FAIL_RATE:
            raise RuntimeError('Write data failure')
        sleep_for = random.randrange(250, 1_000) / 1000
        with tracker.timer('write_data'):
            await asyncio.sleep(sleep_for)
    except Exception as error:
        logging.error(
            f'{error.__class__.__name__} in write_data: {error}'
        )
        logging.debug(f'write_data inputs: {inputs}')
        tracker.set_metric('write_data_errors', 1)
        raise
    else:
        result = True
        logging.info('write_data completed without error')
        logging.debug(f'{result}')
        return result

The synchronous equivalent of this function can use the @tracker.track decorator on the function. When the decorator is applied to an async function, it still runs without error, but it only captures the elapsed time needed for an asynchronous promise or future to be created, which is not terrifically useful in general. Apart from that, it is nearly identical:

# Module "Constants" and Other Attributes
tracker = ProcessTracker()

# ...

@tracker.track
def write_data(payload):
    inputs = vars()
    logging.debug(f'write_data called: {inputs}')
    try:
        if random.random() < WRITE_FAIL_RATE:
            raise RuntimeError('Write data failure')
        sleep_for = random.randrange(250, 1_000) / 1000
        time.sleep(sleep_for)
    except Exception as error:
        logging.error(
            f'{error.__class__.__name__} in write_data: {error}'
        )
        logging.debug(f'write_data inputs: {inputs}')
        # Here is the erro-count metric
        tracker.set_metric('write_data_errors', 1)
        raise
    else:
        result = True
        logging.info('write_data completed without error')
        logging.debug(f'{result}')
        return result

The general logging output also looks pretty much the same as the output for the asynchronous function version, though as I’m writing this, it also generated a stack trace.

The CloudFormation resource definitions for the related metric filters don’t need to be terribly complicated — There are only seven properties for either that must be provided, with one more (FilterName) that is more a convenience item. The CloudFormation YAML required for the latency and errors metrics capture for the data-writer function ends up looking like this:

# ...
Resources:
  # ...
  DataWriterLatencyMetricFilter:
    Type: AWS::Logs::MetricFilter
    Properties:
      FilterName: data-writer-latency-metric-filter
      # This is what triggers picking up the value
      FilterPattern: {$.latencies.write_data>0}
      # This is the source of the log-messages to be evaluated
      LogGroupName: !Ref DataWriterLogGroup
      MetricTransformations: 
        - MetricName: data-writer-latency
          # This is a grouping name, so that all the metrics for
          # the application will be kept in the same namespace
          MetricNamespace: !Ref ApplicationName
          # This is the value that will be picked up
          MetricValue: {$.latencies.write_data}
          # This is the unit-type for the metric
          Unit: Milliseconds
  # ...
  DataWriterErrorCountMetricFilter:
    Type: AWS::Logs::MetricFilter
    Properties:
      FilterName: data-writer-error-count-metric-filter
      FilterPattern: {$.metrics.write_data_errors>0}
      LogGroupName: !Ref DataWriterLogGroup
      MetricTransformations: 
        - MetricName: data-writer-latency
          MetricNamespace: !Ref ApplicationName
          MetricValue: {$.metrics.write_data_errors}
          Unit: Count

Full examples are available in the package repository, both modeled to simulate the Fargate example above: - An asynchronous version - A synchronous version

Moving my code-blogging efforts here

As I'm between jobs (again), I decided to use some of my free time to start collecting and packaging various personal projects, and to write about them. Initially, I tried posting these on LinkedIn, but their article system apparentlyhas issues with preserving things like images and code-snippets, at least when articles are published on a scheduled basis. As a result, I'm moving those efforts here instead — I've generally had good luck in the past with Blogger, though it's not as sophisticated as I might like.

The stories behind many of these packages is vaguely similar, generally following this pattern:

• Some (usually minor) issue surfaces at work;
• The team I'm on comes up with some quick solution, typically a pretty good one, but maybe not optimal or elegant;
• At some subconscious level, the problem nags at me, and I dream up some solution;
• I write the code to implement a better solution (even if better is just a matter of my opinion).

Since many of these issues and solutions have come up across several projects, or even several jobs, I decided to collect them into my own personal toolkit, initially. After being laid off at the beginning of 2025, I decided that I might as well formally package them up: even if I'm the only person who uses any of them, at least I could simply search under my common package prefix on PyPI, or in my public Bitbucket project for what I need. If my activities happened to benefit anyone else, so much the better, and maybe having some published packages out there would also serve as some sort of resumé boost during my ongoing job-hunt. I've had my blogging efforts be a significant differentiator in the past.

On a longer-term basis, I have some thoughts about several larger scoped projects that many of these package projects could be useful for. One in particular that I'd like to pursue is a website management/digital CMS/blogging platform of my own design, written to run using cloud-resident serverless resources and functionality. While I'm not displeased with Blogger as an outlet for this content, it's not what I'd really like, either. Hopefully I won't be engaged in the job-hunt for long enough to actually see that come to fruition, but if I do, I do.

My current plan is to publish an article twice a week, for as long as I have code that I can spare the time to work on, starting with re-publishing the articles that I originally wrote on LinkedIn.