With the release of v.0.0.3
of the package, the main functionality that I’d had in mind was
complete. I’d also gotten the build-process worked out to my
satisfaction, built as a Bitbucket Pipeline controlled by a
definition file in the repo. I’d originally planned for
v.0.0.4 to be spent with implementing the unit tests that
I’d only stubbed out, and making alterations as needed when or if those
actual tests surfaced issues that needed to be corrected. However, when
a real-world opportunity to apply PACT testing to code that I hadn’t
been involved with surfaced, I changed those plans in order to try to
make that viable.
The
v.0.0.4 release that fulfilled that desire focused
mainly on adding the last pieces needed for the build-process to publish
the package to PyPI, where it can be found today as goblinfish-testing-pact.
Previous experience with PyPI-compatible repositories (JFrog’s Artifactory,
specifically) had, some years back, left me with the impression that
there was no guarantee that any PyPI repository would prevent
overwriting of an existing package-version, and that if such an
overwrite occurred, it would mess with the SHA-256 hashes that were
attached to a package, making an existing installation invalid. In order
to at least try to prevent those sorts of complications from
arising, I decided to implement a test-module
(tests/packaging/check_pypi.py) that would actively look in
a specified PyPI repository for the current package version, using the
pyproject.toml file to identify the package name and
version. If the test’s execution failed it would prevent an overwrite of
an existing version of the package. The relevant pieces of the
pyproject.toml file were the project’s name
and version:
# Taken from the v.0.0.4 pyproject.toml file
[project]
name = "goblinfish-testing-pact"
version = "0.0.4"These data-points are used to build the PyPI URL, read the content from that page, and search for a package name and version that match in the response. If any matches are found, that indicates that a version of the package with the current version specification has already been published, and a test failure is raised, which will terminate the execution of the pipeline. The test-code itself is quite simple, bordering on painfully brute force, but it works:
PROJECT_ROOT = Path(__file__).parent.parent.parent
PROJECT_TOML = PROJECT_ROOT / 'pyproject.toml'
project_data = loads(PROJECT_TOML.read_text())
def test_package_new_in_pypi():
"""
Tests that the current version of the package does not exist
in the public pypi.org repository.
"""
root_url = 'pypi.org'
pypi_site = HTTPSConnection(root_url)
pypi_name = project_data['project']['name'].replace('_', '-')
package_name = project_data['project']['name'] \
.replace('-', '_')
pypi_version = project_data['project']['version']
url = f'/simple/{pypi_name}/'
pypi_site.request('GET', url)
response = pypi_site.getresponse()
package_list = response.read().decode()
search_string = f'{package_name}-{pypi_version}'
assert package_list.find(search_string) == -1, \
f'Found {search_string} in the versions list at '
f'{root_url}{url}'The circumstances around applying the package’s test prescriptions
brought to my attention that there were other types of data descriptors
that needed to be accounted for than just the basic
property types that were in place. My initial efforts were
limited to identifying property object class members using
the
inspect.isdatadescriptor function., which I expected
would satisfy the need to identify both property objects
and any other objects that implemented the
descriptor protocol. However, I encountered an odd bit of code that
isdatadescriptor did not identify, so I made
changes to accommodate that, checking for both the standard
property methods (fget, fset, and
fdel) and implementations that had any of the
descriptor-protocol methods (__get__, __set__,
and __delete__), even if those objects did not have
all of them. As an interim solution to the odd case I
encountered, that seemed to do the trick, but I wasn’t happy with where
that left things, and so I planned to re-examine that later.
I cannot disclose the specific details of the code that raised this concern (it was work done under an NDA), but I can provide an example of the sort of thing I encountered. If a complete descriptor class is defined as:
class CompleteDescriptor:
def __set_name__(self, owner, name):
self._prop_name = name
self._name = f'_{owner.__name__}__{name}'
def __get__(self, obj, type=None):
if obj is None:
return self
if hasattr(obj, self._name):
return getattr(obj, self._name)
raise RuntimeError(
f'{owner}.{self._prop_name} has not been set'
)
def __set__(self, obj, value):
setattr(obj, self._name, value)
def __delete__(self, obj):
if hasattr(obj, self._name):
delattr(obj, self._name)
return None
raise RuntimeError(
f'{owner}.{self._prop_name} has not been set'
)… and a class is defined that uses that plus some variations that
omit the __delete__ and/or __set__ methods of
the descriptor:
class Thingy:
get_only = GetOnlyDescriptor()
get_set_only = GetSetDescriptor()
get_del_only = GetDelDescriptor()
get_set_del = CompleteDescriptor()…then the following inspect.isdatadescriptor-based
code:
from inspect import isdatadescriptor
print(
'isdatadescriptor(get_only) '.ljust(34, '.')
+ f' {isdatadescriptor(Thingy.get_only)}'
)
print(
'isdatadescriptor(get_set_only) '.ljust(34, '.')
+ f' {isdatadescriptor(Thingy.get_set_only)}'
)
print(
'isdatadescriptor(get_del_only) '.ljust(34, '.')
+ f' {isdatadescriptor(Thingy.get_del_only)}'
)
print(
'isdatadescriptor(get_set_del) '.ljust(34, '.')
+ f' {isdatadescriptor(Thingy.get_set_del)}'
)…yields the following output:
isdatadescriptor(get_only) ....... False isdatadescriptor(get_set_only) ... True isdatadescriptor(get_del_only) ... True isdatadescriptor(get_set_del) .... True
The two key pieces of information that were missed in the original code that prompted this investigation, from the Descriptor HowTo Guide are:
If an object defines
__set__()or__delete__(), it is considered a data descriptor. Descriptors that only define__get__()are called non-data descriptors (they are often used for methods but other uses are possible).
and
To make a read-only data descriptor, define both
__get__()and__set__()with the__set__()raising anAttributeErrorwhen called. Defining the__set__()method with an exception raising placeholder is enough to make it a data descriptor.
The issue that originally sparked this was an attempt to create a
data descriptor that did not implement either __set__ or
__delete__ — making it a non-data descriptor, as
noted above. This was a distinction that I wasn’t aware of until it
surfaced in the code I was poking around at.
ImportantIt’s also important to note that a class that implements the descriptor protocol is not, itself a data descriptor. A data descriptor is an instance of a class that implements the protocol!
The v.0.0.4 release also completed the stubbing out of
unit tests, providing the prescribed test methods for the properties
that were not tested prior to that in v.0.0.3.
The v.0.0.5
release was really nothing more than adding the license
and some manual
setup instructions, showing how to start a PACT test-suite, and how
to iterate against the various types of test failures that would occur
until all the prescribed test-entities were accounted for.
The v.0.0.6
release was mostly concerned with cleaning up and adding items to
the code-base in preparation for starting this series of articles.
I noticed a few other random things that I want to fix, or at least
think about in v.0.0.7 as well, and a couple of items that
I didn’t discuss in the post for v.0.0.3. The one item to
definitely be fixed is the handling of test-method name expectations for
dunder methods in the source entities, like __init__.
Somehow, while working through the name-generation process for
test-methods, I missed accounting for the combination of the test prefix
(test_) being added to a dunder-method name, leading to
three underscores in the expected test-method name:
test___init__, for example, instead of
test__init__. That’s not a functional issue; the
test-methods are still being prescribed, but it’s one of those little
things that will annoy me if I don’t fix it.
The item that I’ll need to think on is whether or not to prescribe
_unhappy_paths test-methods for methods and functions that
have no arguments, or at least no arguments that aren’t part of an
object scope (self and cls). Based on some
common patterns that I’ve seen and used in work projects, I would plan
for relevant _happy_paths test-methods to check for things
like cached returns and returns of appropriate types, but I have this
nagging suspicion that I missed something in the potential for
unhappy-path results to occur as well.
The first of the two ideas that I have implemented but didn’t think to discuss is how inherited member tests play out. I did set things up so that test-methods in a given class would not include members that were inherited from some other class. The logic here is that if the PACT processes are applied as expected, there would be a prescribed test for every class-member in the related test-case for the class that they are defined in. If that source class is used as a parent class for a different class, and the members of that parent class are not overridden, then the prescribed tests of the parent class’ members would provide the required testing. If a parent class member was overridden, then the prescribed test requirements on the child class would detect that, and require a new test-method in the test case class that relates to the child. The function used to determine inheritance status is already in the code, and is quite simple — Its documentation is longer than its code:
def is_inherited(member_name: str, cls: type) -> bool:
"""
Determines whether a member of a class, identified by its
name, is inherited from one or more parent classes of a
class.
Parameters:
-----------
member_name : str
The name of the member to check inheritance status of
cls : type
The class that the member-name is checked for.
Returns:
--------
True if the named member is inherited from a parent.
False otherwise
"""
parents = cls.__mro__[1:]
result = any(
[
getattr(parent, member_name, None) \
is getattr(cls, member_name)
for parent in parents
]
)
logger.debug(f'is_inherited({member_name}, {cls}): {result}')
return resultWhile I’m on the topic of test prescriptions and their relationship to inheritance, it also feels worthwhile to note that abstract members of classes will have prescribed test requirements too. That was conscious decision made during a previous pass at the idea that this package implements, and though I didn’t make a conscious decision about it in this iteration, the logic behind the previous decision still holds true, I think: If the PACT processes are intended to assure that the contracts of source entities are tested, and given that an abstract member of a class is part of that class’ interface, it follows that there should be test-methods associated with abstract members of classes. What has not been given any consideration yet, in that area, is whether the prescribed tests should include all of the requirements expectations for a concrete member. Doing so would increase the number of test-methods to no good end, I feel, so I’m inclined to add detection of abstract state for a class-member wherever possible, and limit the test prescription accordingly, but I will have to think more on that before I settle on that as the preferred path. For example, given this abstract class:
from abc import ABC, abstractmethod
class BaseThingy(ABC):
@abstractmethod
def some_method(arg, *args, kwdonly1, *kwargs):
pass…the current test-prescription process would require
test_some_method_happy_paths,
test_some_method_bad_arg,
test_some_method_bad_args,
test_some_method_bad_kwdonly1, and
test_some_method_bad_kwargs test-methods, all of which
would really only be able to test that an object of a class deriving
from BaseThingy could not be instantiated without
implementing some_method.
Those notes get this series of posts up to date with respect to the
package version that’s in the repo and installable from PyPI as of the
beginning of May, 2025. With everything I’ve covered in this post in
mind, the work for the v.0.0.7 release, whenever I can get
to starting it, will include:
- Implement all unit tests.
- Rework the property/data-descriptor detection to handle properties specifically, and other descriptors more generally:
- Member properties can use
isinstance(target, property), and can be checked forfget,fsetandfdelmembers not beingNone. - Other descriptors can use
inspect.isdatadescriptor, but are expected to always have the__get__, and at least one of the__set__and__delete__members shown in the descriptor protocol docs. - Set test-method expectations based on the presence of get, set and delete methods discovered using this new breakout.
- Update tests as needed!
- Correct maximum underscores in test-name expectations: no more than two (
__init__, not___init__). - Think about requiring an
_unhappy_pathstest for methods and functions that have no arguments (or none butselforcls). - Give some thought to whether to require the full set of test-methods for abstract class members, vs. just requiring a single test-method, where the assertion that the source member is abstract can be made.
The full implementation of the existing unit tests is, to my thinking, a critical step in this. After all, part of the purpose that they serve is to provide regression testing, and it just feels like the balance of the code is stable enough that regression makes sense to add now, rather than waiting for another round of changes. The balance of the items in that list may take a while to work through to my satisfaction: As I’m writing this, there are 90 test-methods that I have yet to examine and potentially implement just to get that first item checked off, and since that needs to happen first, I may not post about the PACT stuff again for a bit.
