Python decorator patterns

Introduction

In Python, functions are first class citizens: functions can be passed to other functions, can be returned from functions, and can be created on the fly. Let's see an example:

# define a function on-the-fly
pow2 = lambda x: x**2
print(pow2(2))

# take a function as a parameter
def print_twice(func: Callable, arg: Any):
    print(func(arg))
    print(func(arg))
print_twice(pow2, 3)

# take a function as a parameter and return a new function
def hello():
    print('Hello world!')
def loop(func: Callable, n: int):
    for _ in range(n):
        func()
loop(hello, 3)

Output:

4
9
9
Hello world!
Hello world!
Hello world!

Decorators in Pythons are syntactic sugar for passing functions to functions and returning a new function. Let's see how this works and how we can put it to use in practice. The code for this article is on Github.

@measure: decorator functions without arguments

Let's take a useful example of measuring how long it takes to execute a function. The best would be if we could easily annotate an existing function and get "free" measurements. Let's look at the following two functions:

from timeit import default_timer as timer
from time import sleep

def measure(func: Callable):
    def inner(*args, **kwargs):
        print(f'---> Calling {func.__name__}()')
        start = timer()
        func(*args, **kwargs)
        elapsed_sec = timer() - start
        print(f'---> Done {func.__name__}(): {elapsed_sec:.3f} secs')
    return inner

def sleeper(seconds: int = 0):
    print('Going to sleep...')
    sleep(seconds)
    print('Done!')

measure() is a function which takes a function func() as an argument, and returns a function inner() declared on the inside. inner() takes whatever arguments are passed in and passed them along to func(), but wraps this call in a few lines of to measure and print the elapsed time in seconds. sleeper() is a test function which explicitly sleeps for a while so we can measure it.

Given these, we can construct a measured sleeper() function like:

measured_sleeper = measure(sleeper)
measured_sleeper(3)

Output:

---> Calling sleeper()
Going to sleep...
Done!
---> Done sleeper(): 3.000 secs

This works, but if we're already using sleeper() in a bunch of places, we'd have to replace all those calls with measured_sleeper(). Instead, we can:

sleeper = measure(sleeper)

Here we are replacing the sleeper reference in the current scope to point to the measured version of the original sleeper() function. This is exactly the same thing as putting the @ decorator in front of the function declaration:

@measure
def sleeper(seconds: int = 0):
    print('Going to sleep...')
    sleep(seconds)
    print('Done!')

So @decorators are just syntactic sugar to passing a newly defined function to an existing decorator function, which returns a new function, and having the original function name point to this new function!

@repeat: parameterized decorator function

In the above example we took an existing function sleeper() and decorated it with a function-taking-and-returning-a-function measure(), ie. a @decorator. What if we want to pass arguments to the decorator function itself? For example, imagine we have a function, and we want to repeat it n times. To accomplish this, we just have to add one more inner function:

def repeat(n: int = 1):
    def decorator(func: Callable):
        def inner(*args, **kwargs):
            for _ in range(n):
                func(*args, **kwargs)
        return inner
    return decorator

@repeat(n=3)
def hello(name: str):
    print(f'Hello {name}!')

hello('world')

Output:

Hello world!
Hello world!
Hello world!

@trace: Decorating a class with a function

We can also decorate classes, not just functions. As an example, assume we have an existing class Foo, and we would like to trace it, ie. get a print() each time a method is called, without having to manually change each method. So we'd like to be able to put @trace before the class definition and get this functionality for free, like:

@trace
class Foo:
    i: int = 0
    def __init__(self, i: int = 0):
        self.i = i
    def increment(self):
        self.i += 1
    def __str__(self):
        return f'This is a {self.__class__.__name__} object with i = {self.i}'

What does trace() look like? It must accepts a cls argument (the newly defined class, Foo in our case), and return a new/modified class (with added tracing):

def trace(cls: type):
    def make_traced(cls: type, method_name: str, method: Callable):
        def traced_method(*args, **kwargs):
            print(f'Executing {cls.__name__}::{method_name}...')
            return method(*args, **kwargs)
        return traced_method
    for name in cls.__dict__.keys():
        if callable(getattr(cls, name)) and name != '__class__':
            setattr(cls, name, make_traced(cls, name, getattr(cls, name)))
    return cls

The implementation is quite straightforward. We go through all methods in cls.__dict__.items(), and replace the method with a wrapped method, which we manufacture with the inner make_traced() function. It works:

f1 = Foo()
f2 = Foo(4)
f1.increment()
print(f1)
print(f2)

Output:

Executing Foo::__init__...
Executing Foo::__init__...
Executing Foo::increment...
Executing Foo::__str__...
This is a Foo object with i = 1
Executing Foo::__str__...
This is a Foo object with i = 4

@singleton: The singleton pattern

A second example of decorating a class with a function is implementing the common singleton pattern:

In software engineering, the singleton pattern is a software design pattern that restricts the instantiation of a class to one "single" instance. This is useful when exactly one object is needed to coordinate actions across the system.

Our implementation as a Python decorator @singleton:

def singleton(cls: type):
    def __new__singleton(cls: type, *args, **kwargs):
        if not hasattr(cls, '__singleton'):
            cls.__singleton = object.__new__(cls) # type: ignore
        return cls.__singleton                    # type: ignore
    cls.__new__ = __new__singleton                # type: ignore
    return cls

As mentioned in the Enum articles, the __new__() class method is called to construct new objects, before __init__() is called on the newly created instance to initialize it. So, to get singleton behaviour, we just need to override __new__() to always return a single instance. Let's test it:

@singleton
class Foo:
    i: int = 0
    def __init__(self, i: int = 0):
        self.i = i
    def increment(self):
        self.i += 1
    def __str__(self):
        return f'This is a {self.__class__.__name__} object with i = {self.i}'

@singleton
class Bar:
    i: int = 0
    def __init__(self, i: int = 0):
        self.i = i
    def increment(self):
        self.i += 1
    def __str__(self):
        return f'This is a {self.__class__.__name__} object with i = {self.i}'

f1 = Foo()
f2 = Foo(4)
f1.increment()
b1 = Bar(9)
print(f1)
print(f2)
print(b1)
print(f1 is f2)
print(f1 is b1)

Output:

This is a Foo object with i = 5
This is a Foo object with i = 5
This is a Bar object with i = 9
True
False

@Count: Decorating a class with a class

The reason the above code works is that in Python, class declarations are really just syntactic sugar for a function which constructs a new type object. For example, a class Foo declared above can also be defined programatically like:

def make_class(name):
    cls = type(name, (), {})
    setattr(cls, 'i', 0)
    def __init__(self, i): self.i = i
    setattr(cls, '__init__', __init__)
    def increment(self): self.i += 1
    setattr(cls, 'increment', increment)
    def __str__(self): return f'This is a {self.__class__.__name__} object with i = {self.i}'
    setattr(cls, '__str__', __str__)
    return cls

Foo = make_class('Foo')

But, if that's the case, we can not just decorate a function with a function, a class with a function, but also a class with a class. Let's see an example of this with the @Count pattern, where we want to count the number of instances created. We have an existing class, and we'd like to be able to just put @Count before class definition, and get a "free" count of instances created, that we can then access using the decorator Count class. The solution:

class Count:
    instances: DefaultDict[str, int] = defaultdict(int) # we will use this as a class instance
    def __call__(self, cls): # here cls is either Foo or Bar
        class Counted(cls): # here cls is either Foo or Bar
            def __new__(cls: type, *args, **kwargs): # here cls is Counted
                Count.instances[cls.__bases__[0].__name__] += 1
                return super().__new__(cls) # type: ignore
        Counted.__name__ = cls.__name__
        # without this ^ , self.__class__.__name__ would
        # be 'Counted' in the __str__() functions below
        return Counted

The trick is that when a class is decorated with Count, its __call__() method is invoked by the runtime, and the class is passed in as cls. Inside, we construct a new class Counted, which has cls as its parent, but overrides __new__(), and increments a counter in the Count class variable instances (but otherwise created a new instance and returns it). The newly constructed Counted class (whose name is overridden) is then returned, and replaces the original defined class. Let's see it in action:

@Count()
class Foo:
    i: int = 0
    def __init__(self, i: int = 0):
        self.i = i
    def increment(self):
        self.i += 1
    def __str__(self):
        return f'This is a {self.__class__.__name__} object with i = {self.i}'
@Count()
class Bar:
    i: int = 0
    def __init__(self, i: int = 0):
        self.i = i
    def increment(self):
        self.i += 1
    def __str__(self):
        return f'This is a {self.__class__.__name__} object with i = {self.i}'

f1 = Foo()
f2 = Foo(6)
f2.increment()
b1 = Bar(9)
print(f1)
print(f2)
print(b1)
for class_name, num_instances in Count.instances.items():
    print(f'{class_name} -> {num_instances}')

Output:

This is a Foo object with i = 0
This is a Foo object with i = 7
This is a Bar object with i = 9
Foo -> 2
Bar -> 1

@app.route: Building a Flask-like application object by decorating functions

Finally, many of us have used Flask, and have written HTTP handler functions along the lines of:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello():
    return 'Hello, World!'

This is yet another creative use of decorator patterns. Here we're building up an app object by adding our custom handler functions, but we don't have to worry about defining our own class derived from Flask, we just write flat functions which we decorate. This functionality is straightforward to duplicate as a toy Router class:

class Router:
    routes: dict[str, Callable] = {}

    def route(self, prefix: str):
        def decorator(func: Callable):
            self.routes[prefix] = func
        return decorator

    def default_handler(self, path):
        return f'404 (path was {path})'

    def handle_request(self, path):
        longest_match, handler_func = 0, None
        for prefix, func in self.routes.items():
            if path.startswith(prefix) and len(prefix) > longest_match:
                longest_match, handler_func = len(prefix), func
        if handler_func is None:
            handler_func = self.default_handler
        print(f'Response: {handler_func(path)}')

The only trick here is that the Router::route() can act like a decorator, and returns a function. Example usage:

app = Router()

@app.route('/')
def hello(_):
    return 'Hello to my server!'

@app.route('/version')
def version(_):
    return 'Version 0.1'

app.handle_request('/')
app.handle_request('/version')
app.handle_request('does-not-exist')

Output:

Response: Hello to my server!
Response: Version 0.1
Response: 404 (path was does-not-exist)

@decorator vs @decorator()

In the @measure example, we wrote:

@measure
def sleeper(seconds: int = 0):
    ...

Could we also write @measure() before the def? No! We would get an error:

measure() missing 1 required positional argument: 'func'

But, in the app.route() example, we do write the () parentheses. The situation is simple: roughly speaking, @decorator def func gets replaced by func = decorator(func). If we write @decorator() def func, it gets replaced by func = decorator()(func). So in the latter case, decorator() is run, and it needs to return a function which accepts a function as as an argument, and returns a function. This is how all the examples where the decorator takes an argument are structured.

Conclusion

In Python functions are first class citizens, and decorators are powerful syntactic sugar exploiting this functionality to give programmers a seemingly "magic" way to construct useful compositions of functions and classes. This is an important language feature that sets Python apart from traditional OOP languages like C++ and Java, where achieving such functionality requires more code, or more complex templated code. This dynamic nature of Python creates more runtime overhead compared to a language like C++, but it makes the code easier to wrote and comprehend. This is a win for programmers and projects; in most real-world software engineering efforts runtime performance is not a bottleneck.

Thanks Zsolt for bugfixes and improvement suggestions.