Add first article

1 files changed, 700 insertions, 0 deletions

diff --git a/content/python-best-practices.rst b/content/python-best-practices.rst
new file mode 100644
index 0000000..fd8b7c3
--- /dev/null
+++ b/content/python-best-practices.rst
@@ -0,0 +1,700 @@
+Good practices in Python
+========================
+
+:date: 2013-01-20 17:34
+
+This post is a collection of various facts about Python:
+
+* common mistakes that I encounter frequently when reading code written by
+  myself or other people.
+
+* specific aspects of the Python language that are not very well-known and that
+  I think should be used more.
+
+* general recommendations regarding Python.
+
+Please note that I do not consider myself a Python expert, and it is possible
+that the following text contains inaccurate statements.
+
+Also, due to its very nature, this post is rather unstructured. The table of
+contents should help you jumping directly to the part you are interested in.
+
+.. contents:: :local:
+
+List comprehensions
+-------------------
+
+*List comprehensions* give Python users a very concise and powerful syntax to
+build a list from another list (or any iterable object). The syntax is the
+following:
+
+.. code-block:: python
+
+    result_list = [expression(item) for item in original_list if condition(item)]
+
+which means that ``result_list`` will be a list containing ``expression(item)``
+(any expression that you can computed from ``item``) where ``item`` is an
+element of ``original_list`` for which ``condition(item)`` (a boolean
+expression involving ``item``) is ``True``. The boolean condition which allows
+you to filer the list is optional.
+
+For example, to compute the list of squares of elements in a list, instead of:
+
+.. code-block:: python
+
+    l = [1, 2, 3]
+    result = []
+    for i in l:
+        result.append(i*i)
+
+which is particularly inefficient because of the repeated use of the ``append``
+function, one could use a *list comprehension*:
+
+.. code-block:: python
+
+    l = [1, 2, 3]
+    result = [i*i for i in l] # result = [1, 4, 9]
+
+In addition to being shorter, the above code is also faster (around 3x
+improvement) because you build the list in one instruction.
+
+Another example to compute the list of square roots of all non-negative
+elements of a list (you could get in big troubles computing the square root of
+a negative element):
+
+.. code-block:: python
+
+    from math import sqrt
+
+    l = [4, -3, 9]
+    result = [sqrt(i) for i in l if i>=0] # result = [2, 3]
+
+The same syntax also exists for dictionaries, this is called *dict
+comprehensions* (very original, isn't it?). For example, to transform a list of
+(name, phone number) pairs into a dictionary, for faster lookup:
+
+.. code-block:: python
+
+    l = [("Barthes", "+33 6 29 64 91 12"), ("Dumbo", "+001 650 472 4243")]
+    d = {name: phone for (name, phone) in l}
+
+You can get more details about *list comprehensions* on the `dedicated section <http://docs.python.org/tutorial/datastructures.html#list-comprehensions>`__ of the official documentation.
+
+The multiples faces of ``in``
+-----------------------------
+
+The ``in`` keyword has many different meanings and makes Python code so easy to
+write that people often forget to use it.
+
+* ``in`` gives a universal syntax to iterate over iterable objects. For
+  example, to iterate over a list, instead of:
+
+  .. code-block:: python
+
+    l = [1, 2, 3]
+    for i in range(len(l)):
+        print l[i]
+
+  you could simply write:
+
+  .. code-block:: python
+
+        l = [1, 2,3]
+        for i in l:
+            print i
+
+  similarly, to iterate over a dictionary, instead of:
+
+  .. code-block:: python
+
+        d = { ... }
+        for key in d.keys():
+            print d[key]
+
+  you could write:
+
+  .. code-block:: python
+
+        d = { ... }
+        for key in d:
+            print d[key]
+
+* ``in`` also allows you to test whether an element belongs to some structure:
+  list, dictionary (or any iterable object), occurrence of a substring inside
+  a string. For example:
+
+  .. code-block:: python
+
+        l = [line for line in open("server.log") if "Connected" in line]
+
+  will return the list of lines from the file ``server.log`` containing
+  ``Connected`` as a substring.
+
+Manipulating lists with atomic instructions
+-------------------------------------------
+
+More generally, it is advised to avoid iterating over a list with a ``for``
+loop. ``for`` loops are slow in Python and writing an operation over a list as
+a single instruction allows Python to optimize the execution of the code
+internally.
+
+*List comprehensions* often help in replacing an iteration by a single
+instructions. Here are a few other functions which can be helpful in this
+regard:
+
+* ``join`` can be useful to format a list. For example, to print the list of
+  words whose first letter is ``a`` in a list of words. Instead of:
+
+  .. code-block:: python
+
+    l = [ ... ]
+    result = ""
+    for word in l:
+        if word[0] == 'a':
+            result += word + " "
+    print result
+
+  you could do:
+
+  .. code-block:: python
+
+    l = [ ... ]
+    print " ".join([word for word in l if word[0] == 'a'])
+
+* ``sum``. To sum the elements of a list.
+
+* ``map`` to apply a given function to all elements in a list. For example to
+  reverse all the words in a list:
+
+  .. code-block:: python
+
+    l = ["Dumbo", "Polochon"]
+
+    def reverse(word):
+        return word[::-1]
+
+    m = map(reverse, l) # m = ['obmuD', 'nohcoloP']
+
+*slices* are also very useful when it comes to manipulating lists (or sublists)
+in blocks. Remember that if ``l`` is a list (or any iterable)
+``l[begin:end:step]`` will extract all the elements from the index ``begin``
+(included) to the index ``end`` (excluded) with a step of ``step`` (this last
+parameter being optional).
+
+If the ``begin`` parameter is omitted, it is given 0 as a default value.
+Similarly, the default value of ``end`` when unspecified is ``len(l)`` (the
+numbers of elements in ``l``). A negative value for ``begin`` or ``end`` will
+be subtracted from the end of the list. For example, to extract all the element
+from a list but the last one:
+
+.. code-block:: python
+
+    l = [1, 2, 3]
+    m = l[:-1] # m = [1, 2]
+
+Using a negative value for the ``step`` parameter can be useful to walk through
+an iterable object in reverse order as shown in the example given above to take
+the mirror image of a word:
+
+.. code-block:: python
+    
+    word = "dumbo"
+    drow = word[::-1] # drow = "obmud"
+
+which compensates for the scandalous lack of a ``reverse`` function for strings
+in Python.
+
+Exceptions
+----------
+
+Exceptions provide a powerful tool found in many high-level programming
+languages, and which are often under-used. They allow for a less defensive
+programming style by handling errors *as they appears* instead of making test
+*beforehand* to prevent them from happening.
+
+In Python, every time you are trying to execute an illegal operation (*e. g.*
+trying to access an element outside a list's boundaries, dividing by zero,
+etc.) instead of simply crashing the program, Python raises an exception which
+can be caught, which gives the programmer a last chance of fixing the problem
+before the program ultimately crashes.
+
+The syntax to catch exceptions in Python is the following:
+
+.. code-block:: python
+
+    try:
+        .... # piece of code potentially raising the exception named Kaboum
+    except:
+        .... # piece of code to be executed is the above code raises the Kaboum exception
+
+For example, if a line of code contains a division by a number which could
+(rarely) be equal to zero, instead of systematically checking that the number
+is non zero, it is much more efficient to encapsulate the line within a ``try
+... except ZeroDivisionErro:`` to handle specifically the rare cases where the
+number will be zero. This is the well known principle: *better ask for
+absolution than permission*.
+
+Another example, when trying to access a key which does not exist in
+a dictionary, Python raises the ``KeyError`` exception. This exception can be
+used to initialize the value associated with a key which does not exist yet in
+the dictionary. For example, to compute a dictionary of word counts in a text,
+you can often find:
+
+.. code-block:: python
+
+    text = "..."
+    result = {}
+    for word in text.split():
+        if word in result:
+            result[word] += 1
+        else:
+            result[word] = 1
+
+You could instead use the ``KeyError`` exception to your advantage to avoid the
+systematic ``if`` test:
+
+.. code-block:: python
+
+    test = "..."
+    result = {}
+    for word in text.split():
+        try:
+            result[word] += 1
+        except KeyError:
+            result[word] = 1
+
+The difference with the previous code is that *most of the time*, this code
+will behave exactly as if the body of the ``for`` loop only contained the
+instruction ``result[word] += 1`` which is a significant speedup compared to
+the first code where a test was computed for each iteration of the loop.
+
+See the `dedicated page <http://docs.python.org/tutorial/errors.html#handling-exceptions>`__ in the official documentation.
+
+Values equivalent to ``True`` or ``False``
+------------------------------------------
+
+If ``test`` is a boolean variable (equal to ``True`` or ``False``), we known
+that it is redundant to write:
+
+.. code-block:: python
+
+    if test == True:
+        ...
+
+instead of:
+
+.. code-block:: python
+
+    if test:
+        ...
+
+More generally, Python has automatic conversion rules from standard types to
+booleans to allow a shorter syntax in conditional tests:
+
+* as in the vast majority of programming languages, a positive integer is
+  converted to ``True`` and zero is converted to ``False``
+* an string is converted to ``False`` if and only if it is empty. For example,
+  to test whether or not a string ``title`` is empty, you can simply write:
+
+  .. code-block:: python
+
+    if title:
+        ...
+
+  instead of:
+
+  .. code-block:: python
+
+   if len(title) > 0:
+        ...
+
+* the ``None`` value, which is a constant used when a variable has not been
+  specified is converted to ``False``. To test that a variable ``var`` is not
+  equal to ``None``, you could write:
+
+  .. code-block:: python
+
+    if not var:
+        ...
+
+  **Beware**, the above code will not allow you to distinguish between the case
+  where ``var`` is ``None`` and the case where ``var`` has a value which is
+  converted to ``False`` by Python (like an empty string or list for example).
+  You need to be careful that this is really what you are trying to test.
+
+Generators
+----------
+
+Generators provide an easy way to create iterator objects (objects over which
+you can iterate). They can be created by using different methods.
+
+Generator expressions
+~~~~~~~~~~~~~~~~~~~~~
+
+*Generators expressions* are exactly similar to *list comprehensions* except
+that the brackets are replaced by parenthesis. Thus, the following code:
+
+.. code-block:: python
+
+  l = [1, 2, 3]
+  m = (i*i for i in l)
+  print '\n'.join(m)
+
+would produce the exact same result had the second line been replaced by:
+
+.. code-block:: python
+
+  m = [i*i for i in l]
+
+The difference between the two codes is that in the case where ``m`` is
+defined by a *list comprehension* the list is integrally computed (and placed
+in memory) when the variable ``m`` is defined. On the contrary, when ``m`` is
+defined by a *gemerator expression*, the elements in ``m`` are generated on
+the go *when needed*: only when trying to iterate over the variable ``m`` (as
+induced by the call to the ``join`` function in the above example) are the
+elements generated.
+
+From the speed of execution point of view, both solutions are equivalent: in
+the end, each element in ``m`` will be computed once and only once. From the
+memory usage point of view however, generators give a clear advantage:
+because the elements are generated dynamically (when needed), one at a time,
+never more than one elememt is stored in memory at the same time. In cases
+when the list is too big to fit into memory, then *generators* could be the
+solution.
+
+When using a ``generator expression`` as the argument of a function, Python
+allows to drop one pair of parenthesis to make the code more readable. For
+example, in the following code:
+
+.. code-block:: python
+
+  l = [1, 2, 3]
+  total = sum((i*i for i in l))
+
+the second line could be replaced by:
+
+.. code-block:: python
+
+  total = sum(i*i for i in l)
+
+Generator functions
+~~~~~~~~~~~~~~~~~~~
+
+A second way to define a *generator* is by writing a function using the special
+keyword ``yield``. When called, this function will return an iterable object
+whose behavior is the following: for each iteration step over the object, the
+function which defined it is executed until a ``yield`` instruction is hit. The
+value following the ``yield`` keyword is returned and can be used during the
+iteration step. The execution of the function is frozen until the next
+iteration step.
+
+For example, let us define the following function:
+
+.. code-block:: python
+
+    def min_max(filename):
+        with open(filename) as f:
+            for line in f:
+                l = map(int, line.split())
+                yield min(l), max(l)
+
+When called, this function will produce an iterable object. When iterating
+over this object, at each iteration one line of ``filename`` will be read,
+and the minimum and maximum value of this line will be returned when the
+``yield`` keyword is reached, freezing the execution of the function until
+the next iteration.
+
+Hence, the following code:
+
+.. code-block:: python
+
+    for (inf, sup) in min_max(filename):
+        print (inf + sup)/2.
+
+is exactly equivalent to:
+
+.. code-block:: python
+
+    with open(filename) as f:
+        for line in f:
+            l = map(int, line.split())
+            inf, sup = min(l), max(l)
+            print (inf + sup)/2.
+
+but allows you to define separately the code which computes the minimum and
+maximum value of the lines, and the code which computes their arithmetic
+mean.
+
+Built-in functions
+~~~~~~~~~~~~~~~~~~
+
+Finally, some built-in functions in Python return generator objects. This is
+the case of the ``xrange`` function which can be used exactly as the ``range``
+function. The difference is that ``range`` computes a list of integers whereas
+``xrange`` defines a generator object, which will generate the elements on the
+go, one at a time. For example a call to ``range(1000000000)`` might induce
+a memory error on your machine (if you do not have enough memory to store this
+list), whereas the same call using the ``xrange`` will not have this issue and
+will behave similarly for purposes of iteration. It is almost always more
+suitable to use ``xrange`` instead of ``range``: in Python 3.x for example,
+``range`` now behaves like ``xrange``.
+
+See more details on the `official documentation <http://docs.python.org/tutorial/classes.html#generators>`__.
+
+Decorators
+----------
+
+*Decorators* provide a very powerful way to alter the behavior of a function
+without redifining it. The syntax is the following:
+
+.. code-block:: python
+
+    @logging
+    def f(x):
+        return x + 1
+
+In the above example, we say that the ``f`` function has been *decorated* with
+the ``logging`` function. ``logging`` must be a function taking another
+function as an argument and the result of decorating the ``f`` function with
+``logging`` is equivalent to:
+
+.. code-block:: python
+
+    def f(x):
+        return x + 1
+
+    f = logging(f)
+
+which means that by decorating ``f``  with ``logging``, ``f`` now behaves as
+the composite function ``logging(f)``.
+
+A simple decorator
+~~~~~~~~~~~~~~~~~~
+
+Imagine that we want the ``logging`` decorator to *log* the calls to the
+function it decorates, by printing them to the standard output. Such
+a decorator could be written like this
+
+.. code-block:: python
+
+    def logging(fun):
+        def aux(*args, **kwargs):
+            print "Calling", fun.__name__
+            fun(*args, **kwargs)
+        return aux
+
+Because ``logging`` could be used to decorate any function, with an arbitrary
+number of arguments and keyword arguments, it is necessary to use the generic
+syntax ``aux(*args, **kwargs)`` which stores all the arguments passed to ``aux`` in
+a list named ``args`` and all the keyword arguments in a dictionary named
+``kwargs``. Note that the exact same arguments are passed to ``fun``, meaning
+that from the argument passing perspective, ``aux`` and ``fun`` behaves exactly
+the same. The only difference between ``aux`` and ``fun`` is that ``aux`` logs
+the call to the standard output before doing the computation made in ``fun``.
+This is the expected behavior of the ``logging`` decorator.
+
+To be perfectly rigorous, the previous decorator should have been written like
+this:
+
+.. code-block:: python
+
+    from functools import wraps
+
+    def logging(fun):
+        @wraps(fun)
+        def aux(*args, **kwargs):
+            print "Calling", fun.__name__
+            fun(*args, **kwargs)
+        return aux
+
+Note that ``aux`` is itself decorated by the ``wraps`` decorator provided by
+the ``functools`` official module. This decorators does some magic to ensure
+that ``aux`` behaves as closely as possible to ``fun``. For example, whithout
+this decorator, the following code:
+
+.. code-block:: python
+
+    @logging
+    def f(x):
+        return x + 1
+
+    print f.__name__
+
+would print ``aux`` to the standard output, instead of the expected ``f``. The
+``wraps`` decorator ensures that the ``__name__`` attribute is preserved
+throughout a decoration.
+
+Let us further assume that you want to extend the ``logging`` decorator to not
+only log the calls, but also keep track of how many times the function has been
+called. 
+
+You could be tempted to write something like:
+
+.. code-block:: python
+
+    from functools import wraps
+
+    def logging(fun):
+        a = 0
+        @wraps(fun)
+        def aux(*args, **kwargs):
+            a = a + 1
+            print "{0} has been called {1} times".format(fun.__name__, a)
+            fun(*args, **kwargs)
+        return aux
+
+However, if you apply this decorator to some function and then call this
+function, you will get an angry face from Python complaining that the variable
+``a`` is unbound. The reason for this is that in the line:
+
+.. code-block:: python
+
+    a = a + 1
+
+Python thinks you are redefining the variable ``a`` and forgets about the fact
+that this variable has already been initialized to 0. So when reaching the ``a
++ 1`` part, ``a`` is no more defined, which causes the error. This is a current
+limitation of Python 2: local variables that have been defined outside the
+current scope are read-only variables.
+
+A standard way to circumvent this limitation is to use a mutable structure for
+``a``: ``a`` itself cannot be redefined, but the structure it is pointed to can
+be modified. In the previous example, this could lead to the following code:
+
+.. code-block:: python
+
+    from functools import wraps
+
+    def logging(fun):
+        a = [0]
+        @wraps(fun)
+        def aux(*args, **kwargs):
+            a[0] = a[0] + 1
+            print "{0} has been called {1} times".format(fun.__name__, a[0])
+            fun(*args, **kwargs)
+        return aux
+
+where ``a`` points to a list a length 1 where the number of calls is stored at
+the first position.
+
+Another example
+~~~~~~~~~~~~~~~
+
+A common example which is often used to illustrate decorators in Python is
+`memoization <http://en.wikipedia.org/wiki/Memoization>__`: when a function is
+computation-heavy, but is often called using the same parameters, you can save
+a lot of time by storing past results returned by the function.
+
+This idea can be nicely implemented in Python by using a decorator. This
+decorator will store past results in a dictionary: when the decorated function
+will be called, the decorator will make a lookup in the dictionary to check
+whether the function has already been called with the same parameter, and
+return the stored value in this case.
+
+Here is how you could write such a decorator for a single argument function:
+
+.. code-block:: python
+
+    from functools import wraps
+
+    def memoize(fun):
+        cache = {}
+        @wraps(fun)
+        def aux(x):
+            if x in cache:
+                return cache[x]
+            else:
+                a = fun(x)
+                cache[x] = a
+                return a
+        return aux
+
+Then if ``f`` is defined like this:
+
+.. code-block:: python
+
+    @memoize
+    def f(x):
+        ... # very long and heavy computation
+
+when calling ``f`` twice with the same parameter, the first call will take
+a lot of time to be computed, whereas the secod call will be almost
+instantaneous.  
+
+Classes with two methods
+------------------------
+
+Let us briefly recall how class work in Python. A class is defined like this:
+
+.. code-block:: python
+
+    class Cipher:
+
+        def __init__(self, key):
+            self.key = key
+
+        def decrypt(self, message):
+            return (message & self.key)
+
+all the methods of a class take as their first argument, which is always named
+``self`` by convention, the instance of the class on which the method is
+called. Thus, if ``a`` is an instance of the ``Cipher`` class, the call
+``a.decrypt(message)`` is equivalent to ``decrypt(a, message)``.
+
+The special function ``__init__`` is the class constructor and is called
+every time an instance of the class is created. It is mainly used to initialize
+some attributes of the instance. An instance of ``Cipher`` class can be created
+like this:
+
+.. code-block:: python
+
+    d = Cipher(key)
+
+A flaw commonly found in code written by people coming from object-oriented
+programing languages is to create classes for everything. This often leads to
+classes containing only two methods, one being the ``__init__`` function. This
+is the case in the class written above as an example. By looking to this
+example a bit closer, you can see that you could completely get rid of the class
+definition: you only need a ``decrypt`` function taking the key as an
+additional argument:
+
+.. code-block:: python
+
+    def decrypt(key, message):
+        return (message & key)
+
+Some people could object that it still makes sense to use a class in the
+example above, if we plan to extend the ``Cypher`` class in the future, for
+example by adding an ``encryp`` function. In my opinion, it is better to start
+by writing your code as simply as possible. If you really need to extend the
+code, then you can start restructuring it and group several related functions
+in a class.
+
+PEP 8
+-----
+
+When writing about good practices in Python, it is impossible not to mention
+the PEP8. It is a set of recommendations regarding coding style in Python.
+These recommendations are of course not absolute rules and should be taken as
+advices. However, I noticed that following these recommendations generally
+leads to greater code readability. Furthermore, as many people who code in
+Python also follow these recommendations, adopting them reduces the gap between
+your code and code written by others: this will save you some time when trying
+to understand code.
+
+Here are a few points extracted from the PEP8:
+
+* you should follow English typographic rules: no space before a colon, no
+  space before a comma, but a space after, etc.
+
+* you should put spaces around operators like the equal sign, plus sign, etc.
+
+* you should try to limit the length of the lines of code to a maximum of 80
+  characters.
+
+More details on the `PEP8 page <http://www.python.org/dev/peps/pep-0008/>`__.
+
+
+    
+
+