Meta Programming Part 1 - Decorators

A decorator creates "wrapper" function around a function that we provide.

Below, "meta" is the decorator which can be used with any function.
In [106]:
def meta(func):
    def wrapper(*args, **kwiargs):
        print(func.__name__)
        return func(*args, **kwiargs)
    return wrapper

somefunc
Parameters: 1,2
Usage:
In [107]:
@meta
def somefunc(x ,y):
    print("Parameters: " + str(x) + ',' + str(y))
    
somefunc(1,2)

somefunc
Parameters: 1,2
When using decorators, we loose meta information about the function. The Python returns information about the wrapper function and not about the original function. It is the same with help(). To avoid this, we use wraps. @wraps copies the metadata to the decorator.
In [108]:
somefunc
Out[108]:
<function __main__.meta.<locals>.wrapper(*args, **kwiargs)>
In [109]:
help(somefunc)

Help on function wrapper in module __main__:

wrapper(*args, **kwiargs)
Same decorator with @wraps. Here @wraps copies the information about the passed function and preserve it for documentation.
In [110]:
from functools import wraps

def meta(func):
    @wraps(func)
    def wrapper(*args, **kwiargs):
        print(func.__name__)
        return func(*args, **kwiargs)
    return wrapper

@meta
def somefunc(x ,y):
    print("Parameters: " + str(x) + ',' + str(y))
So when we get info or help about the decorated function, we get info about the original function and not about the decorator. Note the difference between the info returned by Python before and after using the @wraps.
In [111]:
somefunc
Out[111]:
<function __main__.somefunc(x, y)>
In [112]:
help(somefunc)

Help on function somefunc in module __main__:

somefunc(x, y)
A debugging decorator
In [113]:
def debug(func):
    funcname = func.__qualname__
    @wraps(func)
    def wrapper(*args, **kwiargs):
        print(funcname)
        return func(*args, **kwiargs)
    return wrapper

def somefunc(x ,y):
    print("Parameters: " + str(x) + ',' + str(y))
   
## diffrent kind of usage
somefunc = debug(somefunc)
In [114]:
somefunc("y","z")

somefunc
Parameters: y,z
In [115]:
somefunc
Out[115]:
<function __main__.somefunc(x, y)>
In [116]:
help(somefunc)

Help on function somefunc in module __main__:

somefunc(x, y)
Benefit of using the decorator is, in this case, we can have all the debugging logic in one place which make its logic change or disabling the debug much easier.
To enable a kill switch for debug, let us modify the decorator debut().
In [117]:
def debug(func):
    if DEBUG not in os.environ:
        return func(*args, **kwiargs)
    funcname = func.__qualname__
    @wraps(func)
    def wrapper(*args, **kwiargs):
        print(funcname)
        return func(*args, **kwiargs)
    return wrapper
Decorator with args. We can create decorators with some arguments passed to it which can be used for more detailed debugging.
In [118]:
def debug(somearg=""):
    def decorator(func):
        funcname = somearg + func.__qualname__
        @wraps(func)
        def wrapper(*args, **kwiargs):
            print(funcname)
            return func(*args, **kwiargs)
        return wrapper
    return decorator

@debug(somearg="The function name is ")
def somefunc(x ,y):
    print("Parameters: " + str(x) + ',' + str(y))
In [119]:
somefunc
Out[119]:
<function __main__.somefunc(x, y)>
In [120]:
somefunc("x","y")

The function name is somefunc
Parameters: x,y
To make the argument optional.
In [123]:
from functools import wraps, partial

def debug(func=None, *, somearg=""):
    if func is None:
        return partial(debug, somearg=somearg)
    funcname = somearg + func.__qualname__
    @wraps(func)
    def wrapper(*args, **kwiargs):
        print(funcname)
        return func(*args, **kwiargs)
    return wrapper
Call the function without passing the argument to decorator
In [124]:
@debug
def somefunc(x ,y):
    print("Parameters: " + str(x) + ',' + str(y))

somefunc
Out[124]:
<function __main__.somefunc(x, y)>
In [101]:
somefunc("This is test", " and works")

somefunc
Parameters: This is test, and works
And with the arguement.
In [127]:
@debug(somearg="The function name is ")
def somefunc(x ,y):
    print("Parameters: " + str(x) + ',' + str(y))
In [128]:
somefunc("This is test", " and works")

The function name is somefunc
Parameters: This is test, and works
To cover all the methods in a class with a decorator, one way of doing is:
In [129]:
class sample:
    @debug
    def somefunc(self):
        pass
    @debug
    def otherfunc(self):
        pass
    @debug
    def onemore(self):
        pass
It would be better to cover all the methods in a class in one go. We can achieve it by manipulating the key, pair values in class metadata and add the @debug to every method. Here is one way of doing it:
In [134]:
def debugmethods(cls):
    for key, val in vars(cls).items():
        if callable(val):
            setattr(cls, key, debug(val))
    return cls
In [139]:
class sample:
    def somefunc(self):
        pass
    def otherfunc(self):
        pass
    def onemore(self):
        pass

sample = debugmethods(sample) # class sample's metadata is modificed here.
In [144]:
s = sample()
s
Out[144]:
<__main__.sample at 0x7f9bc1e274d0>
In [145]:
s.somefunc()

sample.somefunc
In [141]:
s.otherfunc()

sample.otherfunc
In [142]:
s.onemore()

sample.onemore
Here are what we did above:
  1. Walk through class dict.
  2. Identify callables.
  3. Wrap with a decorator.
How about debugging class attributes? Let us see how to do that here:
In [163]:
def debugattr(cls):
    originalgetattribute = cls.__getattribute__
    def __getattribute__(self, name):
        print("Attribute: ",name)
        return originalgetattribute(self, name)
    cls.__getattribute__ = __getattribute__
    return cls

@debugattr
class sampleattr:
    def __init__(self,x,y):
        self.x = x
        self.y = y
In [165]:
attr = sampleattr(5,6)
In [166]:
attr.x

Attribute:  x
Out[166]:
5
In [167]:
attr.y

Attribute:  y
Out[167]:
6
What if we want to add the debug wrapper to all the classes at once? The solution would be to create a metaclass with wrapper and inherit other classes from it. Before we get into the code, few points to consider:
  1. Type is top of the heirarchy of classes in Python.
  2. Types are their own class (built-in).
  3. There are many types and are instances of type class.
  4. Classes are instances of types.
Defining a New Metaclass:
  • Inherit from type class.

  • Redefine new or init methods in the inherited class.

    class custype(type):

    def __new__(cls, name, bases, clsdict):  
    clsobj = super().__new__(cls, name, bases, clsdict)  
    return clsobj

  • When we use this new type class:

    class myclass(metaclass=custype)  # this step creates a new class based on the metaclass or type "custype".
In the metaclass definition "debugmeta" below, we get the class to be created normally using built-in metaclass and then we immediately wrap it by class decorator.
In [186]:
class debugmeta(type): # type is top in the hierarchy.
    def __new__(cls, clsname, bases, clsdict):
        clsobj = super().__new__(cls, clsname, bases, clsdict) # a normal class is created.
        clsobj = debugmethods(clsobj) #  all the methods in the class are wrapped by debug wrapper.
        return clsobj

To use it, set it in the base class which will get inherited to all the classes.

In [188]:
class Base(metaclass=debugmeta):
    pass

class myclass1(Base):
    def somefunc(self):
        pass
    def otherfunc(self):
        pass
    def onemore(self):
        pass
    
cls1 = myclass1()
cls1
Out[188]:
<__main__.myclass1 at 0x7f9bc44d0d10>
In [190]:
cls1.somefunc()

myclass1.somefunc
In [196]:
class myclass2(Base):
    def cls2somefunc(self):
        pass
    def cls2otherfunc(self):
        pass
    def cls2more(self):
        pass
    
cls2 = myclass2()
cls2
Out[196]:
<__main__.myclass2 at 0x7f9bc4355110>
In [197]:
cls2.cls2somefunc()

myclass2.cls2somefunc

Decorators - Main Ideas:

  • ##### It is mostly about wrapping / rewriting
    • ##### Decorators: Functions
    • ##### Class Decorators: Classes
    • ##### Metaclassess: Class heirarchies

Let us continue in Part 2 with more advanced techniques.

Reference: David Beazley - Python 3 Metaprogramming https://www.youtube.com/watch?v=sPiWg5jSoZI