Local AWS API Gateway development with Python: Dissecting decorators

Yes, I'm still poking around at the goblinfish.testing.pact package in the background. It's going slowly, since I've been dealing with job-hunting stuff, prepping for a contract job, and still working on the second edition of my first book. It was in the context of that last item that inspiration struck on this package project. As I'm writing this, I had just gotten to the point of starting to write about serverless application development with the services that are officially part of the collection of resources that are, presumably official AWS Serverless Application Model (SAM) services (this is based on their inclusion in the AWS SAM resources and properties documentation), and finished the content about local and server- or container-resident applications. I had focused on Flask and FastAPI in that content because those were, as noted in my previous post, the most popular options (that I saw in the wild, at least).

The inspiration that struck was that it should be possible to extend the various decorators that both of those frameworks use in order to wrap a Lambda Function's entry-point handler-function, translating the native ASGI or WSGI request-input into a Lambda Proxy Integration input, call the wrapped/decorated Lambda function code, and translate its return-value back to an ASGI- or WSGI-compatible response (assuming that the Lambda returns a Lambda Proxy Integration output).

To understand how this will work, it will probably be useful to take a look at what decorators actually do. Here's an example of a simple decorator, applied to a simple function, and called, with print and pprint calls sprinkled through the code at key points to whow what's happening:

from pprint import pprint

head_len = 56

print('-- Defining simple_decorator '.ljust(head_len, '-'))

def simple_decorator(target):
    print(f'simple_decorator({target.__name__}) called.')
    def _inner(*args, **kwargs):
        print('_inner(...) called:')
        pprint(vars())
        print(f'Calling {target.__name__} from _inner')

        # This is where the original function is called
        result = target(*args, **kwargs)

        print(f'Returning results from call to\\\n  {target}')
        return result
    print(f'Returning {_inner.__name__}.')

    # This is where the replacement function comes from
    return _inner

print(
    '-- Applying simple_decorator to simple_function'
    .ljust(head_len, '-')
)

@simple_decorator
def simple_function(arg, *args, kwdonlyarg, **kwargs):
    print('simple_function(...) called:')
    pprint(vars())
    return vars()

if __name__ == '__main__':
    print('-- Calling simple_function '.ljust(head_len, '-'))
    call_result = simple_function(
        'arg', 2, 3, 4, kwdonlyarg=True, kwarg1=object()
    )
    pprint(call_result)

When this code is executed, the output it returns is:

-- Defining simple_decorator ---------------------------
-- Applying simple_decorator to simple_function---------
simple_decorator(simple_function) called.
Returning _inner.
-- Calling simple_function -----------------------------
_inner(...) called:
{'args': ('arg', 2, 3, 4),
 'kwargs': {
     'kwarg1': <object object at 0x1022b85a0>,
     'kwdonlyarg': True
 },
 'target': <function simple_function at 0x102352480>}
Calling simple_function from _inner
simple_function(...) called:
{'arg': 'arg',
 'args': (2, 3, 4),
 'kwargs': {'kwarg1': <object object at 0x1022b85a0>},
 'kwdonlyarg': True}
Returning results from call to\
  <function simple_function at 0x102352480>
{'arg': 'arg',
 'args': (2, 3, 4),
 'kwargs': {'kwarg1': <object object at 0x1022b85a0>},
 'kwdonlyarg': True}

Step by step, this unfolds as:

• The simple_decorator is defined. It doesn't actually get called just yet, so none of the print calls generate any output.
• The simple_decorator decorator is applied to the simple_function function as part of the definition of that function.
  • The Python runtime passes the simple_function as an argument — This is an important item to keep in mind, since the target name association with simple_function remains available in the entire scope of the simple_decorator call, even inside the _inner function.
  • A new instance of the _inner function is created.
  • When that function is called, it calls the original target function.
  • The new _inner function instance is returned.
  • The Python runtime replaces the original simple_function with the _inner function returned
• From that point on, any calls made to the original simple_function name actually call the _inner function instance that was returned during the decoration process. Whether the original simple_function actually gets called as part of the decorated-function process is determined by whether it is called in the _inner function.

This nesting of functions, including the nested call to the original target function, and the persistance of those nested functions outside the scope where they were defined is an example of a closure.

FastAPI- and Flask-application endpoint decorators also accept arguments, though. The quickstart examples for each show that for the FastAPI @app.get() and Flask @app.route() decorations:

# FastAPI bare-bones/quick-start code
from fastapi import FastAPI

app = FastAPI()

@app.get("/")
async def root():
    return {"message": "Hello World"}

# Flask bare-bones/quick-start code
from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello_world():
    return "
Hello, World!
"

Decorators that accept arguments are a bit more complicated: they have three layers of functions instead of two. The outermost layer is where the arguments for the decoration are retrieved — the "/" in the @app.get("/") and @app.route("/") decorations in the quick-start examples above would be captured at this level, and available to the nested functions within it, for example. The next layer in behaves just like the outermost layer of a simple decorator, accepting the target of the decoration. The innermost layer also behaves like the innermost layer of a simple decorator, actually calling the original target function, with whatever addition al logic is needed before and after that call happens. An example decorator that accepts one argument (decorated), and printing noteworthy occurrences in the process similar to the simple example above is:

from pprint import pprint

head_len = 56

print('-- Defining outer '.ljust(head_len, '-'))

def outer(decorated):
    print(f'outer(...) called:')
    pprint(vars())
    def _middle(target):
        print(f'_middle(...) called:')
        pprint(vars())

        # The "decorated" argument passed above is still
        # available, but only appears in vars() if it is used
        print(f'decorated (from outer): {decorated}')

        def _inner(*args, **kwargs):
            print('_inner(...) called:')
            pprint(vars())

            # The "decorated" argument passed above is still
            # available but only appears in vars() if it is used
            print(f'decorated (from outer): {decorated}')

            print(f'Calling {target.__name__} from _inner')
            result = target(*args, **kwargs)
            print(
                f'Returning results from call to\\\n  {target}'
            )
            return result
        print(f'Returning {_inner.__name__}.')
        return _inner
    return _middle

print(
    '-- Applying outer to target_function'
    .ljust(head_len, '-')
)

@outer(True)
def target_function(arg, *args, kwdonlyarg, **kwargs):
    print('target_function(...) called:')
    pprint(vars())
    return vars()

if __name__ == '__main__':
    print('-- Calling target_function '.ljust(head_len, '-'))
    call_result = target_function(
        'arg', 2, 3, 4, kwdonlyarg=True, kwarg1=object()
    )
    pprint(call_result)

…and running this code results in this output:

-- Defining outer ------------------------
-- Applying outer to target_function------
outer(...) called:
{'decorated': True}
_middle(...) called:
{
    'decorated': True,
    'target': <function target_function at 0x104d6d120>
}
decorated (from outer): True
Returning _inner.
-- Calling target_function -----------------------------
_inner(...) called:
{'args': ('arg', 2, 3, 4),
 'decorated': True,
 'kwargs': {
     'kwarg1': <object object at 0x104a785a0>,
     'kwdonlyarg': True
 },
 'target': <function target_function at 0x104d6d120>}
decorated (from outer): True
Calling target_function from _inner
target_function(...) called:
{'arg': 'arg',
 'args': (2, 3, 4),
 'kwargs': {'kwarg1': <object object at 0x104a785a0>},
 'kwdonlyarg': True}
Returning results from call to\
  <function target_function at 0x104d6d120>
{'arg': 'arg',
 'args': (2, 3, 4),
 'kwargs': {'kwarg1': <object object at 0x104a785a0>},
 'kwdonlyarg': True}

Important

It may seem odd, but any arguments provided to a decorator will not be available within the function being decorated, even though they are demonstrably available to the inner function that is calling the target function. The target function, though it is referenced inside the scope of the decorator, is not actually in that scope, so the preservation of those names and values provided by the closure does not carry through to the target function.

It is also worth noting that because of the way decorators work, and how they are defined, it's possible to call a decorator as a function, and pass the target function as an argument to the result of the decorator function call. That is, running this code after the outer decorator is defined...

def external_function(*args, **kwargs):
    print('external_function(...) called:')
    pprint(vars())
    return vars()

local_function = outer('no-decorator')(external_function)
print('-- Calling local_function '.ljust(head_len, '-'))
call_result = local_function(
    'arg', 2, 3, 4, kwdonlyarg=True, kwarg1=object()
)
pprint(call_result)

... will yield this output (truncated to just the new items):

-- Calling local_function ------------------------------
_inner(...) called:
{'args': ('arg', 2, 3, 4),
 'decorated': 'no-decorator',
 'kwargs': {
     'kwarg1': <object object at 0x1044d85a0>,
     'kwdonlyarg': True
 },
 'target': <function external_function at 0x104572340>
}
decorated (from outer): no-decorator
Calling external_function from _inner
external_function(...) called:
{'args': ('arg', 2, 3, 4),
 'kwargs': {
     'kwarg1': <object object at 0x1044d85a0>,
     'kwdonlyarg': True
 }
}
Returning results from call to\
  <function external_function at 0x104572340>
{'args': ('arg', 2, 3, 4),
 'kwargs': {
     'kwarg1': <object object at 0x1044d85a0>,
     'kwdonlyarg': True
 }
}

Another key factor to keep in mind when working with decorators is that all they really do, when it comes right down to it, is replace the target being decorated with the callable returned by the decorator. That, in turn, means that a decorator function can, in fact, return a completely different function. That is probably not a common use case, but it is a legitimate one, and that capability opens the door for the process that I have in mind. At a high level, after doing some work with throwaway code for this project, what I see as the most likely path forward looks something like this:

• Defining a class deriving from the FastAPI and Flask classes provided by those packages;
• Overriding the relevant decorators (FastAPI.get and the other HTTP-verb decorator-methods FastAPI provides, and the Flask.route method), such that:
  • The override accepts a target function that can be pointed at a Lambda Handler elsewhere in a project, either by importing it and passing it, or by specifying its namespace and letting the decorator perform the import.
  • The override would create a new wrapper function.
  • The wrapper function handles converting the FastAPI or Flask request into a Lambda Handler event and context.
  • The wrapper function calls the Lambda Handler with those items.
  • The wrapper function handles converting the response from the Lambda Handler back into whatever response format is expected by FastAPI or Flask before returning it.
• As part of that decoration process, the override would also be responsible for calling the original decorator — the function that it overrides — with all of the normal arguments that decorator expects or accepts, but pointing it at the wrapper function. This should insure that the normal behaviour expected of the FastAPI or Flask decorators is preserved with respect to registering and configuring endpoint handler functions.

With all this done, I would expect that manually generating an endpoint-to-Lambda-Function mapping would look something like this:

from goblinfish.aws.local.fastapi_apigw import FastAPI
from my_lambda_functions import root_get

app = FastAPI()

@app.get(root_get, '/')
async def root():
    # No code is actually needed here, since this would be
    # *completely replaced* by the decorator.
    ...

Ideally, I would also like to be able to support the function-call-only structure shown above, allowing the same definition above to be executed something like this:

from goblinfish.aws.local.fastapi_apigw import FastAPI
from my_lambda_functions import root_get

app = FastAPI()

def mapping_failed(*args, **kwargs):
    # This function should ALWAYS be replaced by an application
    # endpoint mapping, but in case it isn't it will raise an
    # error.
    raise Exception('Mapping failed; check your code!')

root = app.get()

@app.get(root_get, '/')(mapping_failed)

This post has gotten far longer than I had originally intended — decorators are a complex subject, and I could probably be fairly accused of rambling on about them. With all of the exploration into them that has happened here, though, I feel comfortable moving forward with a first attempt my goal. That is where I will pick up in my next post about this package.