Writing the theme was not difficult at all. Liferay has an excellent SDK for writing plugins, which include themes. I didn’t start from scratch but used the great HTML5Goodness responsive theme. The main navigation menu was easy to do since there is a method to iterate over first level pages. However, when I tried to do the same with the footer menu I couldn’t find an easy and clean way to do it. I wanted to put standard links in the footer menu as the Terms of Use, the Privacy Policy and so on in the footer menu but I didn’t know how to organize this stuff in Liferay CMS so I could retrieve them back easily. I thought about using a portlet for the footer but I think that’s not the way to do it since that would affect the portlets layout for all the pages. So I added some variables in the XML that describes the theme and hardcoded the links there. At least I don’t have to change the theme code if we change any of those links.

About the portlets, we needed to write a portlet that pulls the content from an external service via REST calls and render it nicely using some kind of templates. These were the attempts I made and my conclusións:

• Portlet written in Java: this was the obvious choice. The advantages was full access to Liferay API and easyness to integrate the portlet with the SDK stanrdard procedures. The disadvantages were, well, it has to be written in Java. We are far less productive in Java that with other languages. Just for making an HTTP request is quite involved. Hopefully Liferay has APIs for making this easier.
• Portlet written in Javascript: This looked promising and was easy to setup, the problem was the importPackage and importClass functions were not available from the Rhino environment. This made Javascript just a toy language in Liferay since the language itself has no useful standard library and it needs to leverage the net or filesystem or any other calls to the runtime it runs on. This make javascript a very integrable language but also a very dependent language. If we can’t call Liferay API from Javascript and there are no network functions in the language itself we can’t use it for our purposes.
• Portlet written in Python: Our last try was writting the portlet in Python. First we had to update the Jython jar that was included in Liferay since it was a little bit old. Then we added the jyson jar to the jython jar itself to have support for json parsing. We also used Liferay Network APIs since we couldn’t import urllib2 from Python. Finally we even managed to use the Python debugger (PDB) by running Tomcat in the foreground. (bin/catalina.sh run)

One important thing when writing portlets with a scripting language like Python or Javascript is that Liferay will concatenate all your modules into a single big file before running it in the scripting engine (jython or rhino, in our case). This is important to know when reading errors information where the line numbers is not always what we expect. In Python, we can avoid this behavior by changing the PYTHONPATH dinamically at runtime (at the beginning of our script) and then, importing our regular modules will work again.

I’ve been at the PySAML2 sprint with Roland Hedberg and a couple of guys from Zeomega: Baiju M and Chris Austin. It has been a real pleasure to work with this gang. During the nights I hanged out with them and also Brad Allen and other guys from the Texas Python group.

Roland teaching SAML to the Zeomega guys

We managed to clean a lot of stuff at PySAML2, to release a new version, to improve the djangosaml2 package and to start making a Zope2 product for it.

Baiju and me are sprinting, don't you see?

One of these nights we went to a place where there were some good jazz music live. Ireally enjoyed the food, the music and the company of Chris and Baiju.

Today I did some tourism with Baiju and we went to the Atlanta Aquarium and the museum about Coca Cola. I did knew Baiju before since I translated his great book about the Zope Component Architecture but I never meet him in real life. He is a very good developer but also a really nice person. I’m glad I got to spend some time with him.

I’ve also meet other people here at PyCON. Here is just a brief list, sorry if I miss somebody:

• Jeff Rush, a consultant and Zope lover. Very nice guy.
• Laurens Van Houtven, my roommate and a cool guy that hacks on the Twisted project and has different slepping habbits than me
• Sylvia Candelaria, a very smart lady who knows about free software, linguistics, Spain, India and a whole lotta more things. I just talked to her during a lunch but she just blowed my mind with her wisdom.
• Ralph Bean, a guy who was working on an open goverment website during the sprints. He was interested on my work on SAML stuff
• Luke Macken, a Fedora hacker
• Jordan Sissel, a guy who works for logglya nice service about logging with very cool stickers.

Other people I met but I can’t recall their names include people from bitbucket, from eldarion, from linode, and many more!

Well, I guess that’s all I have to say about PyCON 2011 but I’m probably forgetting lots of things since it has been a very exciting week for me. I hope I can make it next year at Santa Clara!

It was 8am and I was ready for more talks!

The second day of the PyCON conference started with a talk by the Dropbox guys. The first half of it was not really interesting but things go better on the second half when they started to explain the technical bits. Then Guido was interviewed and answered some questions that the community had elaborated for him. One of the things he said is that Python 3 is closer than you think. I can tell he is right since many sprints at PyCON were about porting existing libraries to Python 3 and there is a general momentum going in that direction. Then they gave him a big cake representing the 20th birthday of the language.

PyCON moment where Guido is given a big cake representing the 20th python birthday

Then I learned about the new Selenium 2 and that it doesn’t need the ugly selenium-remote-control if you just run it on your own machine so that becomes much easier to use. The speaker was showing how to write tests using the python client library and it looked very very good.

After that talk I went to the zc.buildout talk where Jim Fulton explained how they deploy python projects based on buildout on Redhat based systems. Something very similar to what we do at Yaco. On the reasons they do this is because Zope Corporation hosts their own applications so their sysadmin department and their development department need to play well with each other. He explained how the have two different buidouts for each application: one for the application itself and a different one for its configuration. You can read more about his great talk at http://www.riversnake.com/pycon2011/full/ or, if you are in a hurry, just check out his slides

Jim Fulton speaking about zc.buildout

Then I went to the talk Alex Martelli gave: API design anti-patterns. and these are some of the things I learned:

• You better have a API for your software because you are going to need it sooner or later. Otherwise programmers are going to start using your applications in very nasty ways you can’t even predict.
• APIs don’t just grow up magically, you have to deeply thing about them. It’s harder to change an API than to change the user interface of your software because the consumers of the former (programms) are so much less flexible than the consumers of the later (humans)
• A good way to design an API is to use it by yourself in your own software. This forces you to think what’s the main thing that my program does and what the absolute minimum amount of features that should go into the core. I believe we are doing the right thing here in Merengue

After that, I went to the talk about http libraries in Python. It was given by a Mercurial guy who did a very good analysis of how bad all Python http libraries are except for pycurl, which is very difficult to use. That’s why he wrote his own library for this job and I think it is very good if you need some advanced http features and a very fast library.

To finish the day I went to Tarek’s talk bout distutils 2 and I listened exceited about the state of this project and how it is going to make our lifes so much easier. Distutils 2 is going to be part of Python 3.2 and you will be able to use it as a separated module starting with Python 2.4! There are only two problems to be solved: it is not finished yet (this is easy, just help him!) and it won’t support all the things that setuptools does, like entry_points among other things. We will need some other library for this since there a bunch of us that uses these features. By they way, I’m the one that ask Tarek about entry_points at the end of his talk. I agree with him that they should not be part of distutils2 but that does not solves the problem. We need a full replacement!

Tarek is the man behind distutils2

That afternoon I went to an open space to talk about the future of the Zope community. After that I got to go for dinner with the Zope guys to a nice mediterranean restaurant. I was invited by Zeomega company and I enjoyed the dinner a lot. I’ve been reading and following some of these guys for a long time and it was very cool to talk to them face to face. By the way, I talked with Jim Fulton about Spain since he has been there recently and he liked it a lot.

happiness = getMultiAdapter(zope, dinner)

Next day there were some more nice talks. One guy from Threadless gave a very funny talk. Unfortunately my english is still not that good and I didn’t get all the jokes. After that the people from Disqus gave a very interesting talk about their product and how they did it using python tools: celery (for asynchronous tasks), graphite (for monitoring), senty (for exception monitoring), coverage.py, pep8, pyflakes and may more. The only thing they didn’t like about Python is the current state of packaging, which I have to agree with.

After that I went to see the co-routines talk but I didn’t quite understand them yet. I’ll need some more reading because it looks like they are getting pretty big nowadays.

To end the conference days there was the poster session. I have to admit I didn’t have big expectations on this but I also have to admin that I was nicely surprised. There were lots of good posters and I discovered a bunch of really interesting projects:

• askbot.org , a stackoverflow clone written in Python
• Allura , the code behind Sourceforge
• Fudge, which is very cool for mock testing. For example if you need to test a software that makes http connections but you don’t really want to make them for testing. It’s monkey patching done right.
• Open States , and open goverment initiative
• There was a guy with a wheel chair doing some astonishing job with a kinect device. Sorry I don’t remember the name of his project

You would not believe what this guy just did with a kinect and a lot of courage

By the way, PyCON does a marvelous job by recording all the talks and puting them online. So, in case you couldn’t come, just watch the  talks They are worthy!

I arrived to Atlanta last Wednesday and I was impressed of how big the Hyatt hotel was. Almost like a small city on its own.

This short video shows how huge this hotel is. It’s filmed from the 12th floor

I’m not used to this kind of hotels so I really enjoyed details like the elevators or the huge lobby As I arrived quite late due to some issues about getting out of the gigantic airport the only thing I did that night was having dinner in the Hard Rock Cafe a couuple of blocks next to the hotel.

That's a bunch of floors!

But, anyway, next day I attended to a couple of tutorials, one about Django deployment which was really good and another one about python 3 differences with respect to python 2 which was pretty interesting too.

The Django tutorial is about to start

Then next day (friday) the conference started. Very early in the morning my mind was blown away when I saw how many people there were in the keynote.

Look of the huge room where the PyCon 2011 keynote will take place

This video was shot 10 minutes before the keynote started. During the keynote there was not a single free seat.

The room was pretty big and it was full! Luckily Hilary Mason (bit.ly scientist) did a fantastic job and her talk was really interesting and inspiring. I liked how passionated did she talk about programming and that she believes that programming changes your internal brain structure so you think differently from other non programmers when approaching life’s problems. For good or for bad I think she’s right.

Then I went to a talk about distributed tasks with Celery because I though it was given by Celery author. But it was about how a photographer’s website used Celery to have success. It was nice but not what I expected. Then I moved into “Javascript for people who known Python” by Ian Bicking and it was a good talk, quite fast but good since I knew some of the stuff he was talking about. Even so, I learned a bit. After that talk I went to the “State of pylons/turbogears/repoze” talk and that was just a sneaky title to present the new big thing: Pyramid. I really believe this is going to be a web framework to follow since it has inherited a big community and it has very well written and tested code and documentation. And by the way, the t-shirt is awesome

Later that day I attended to another not so exciting talk like “The development of python and you”. Then I went to couple of talks that I really enjoyed and would say were maybe the best talks of PyCON: “Pluggable Django Patterns” and “Reverse engineering Ian Bicking’s brain”. The first one explained how to write your Django application so others (or even you in another project) can reuse it seamlessly. You should look the talk by yourself because it has tons of little gems. Then the other talk was about understanding how virtualenv and pip (tools maed by Ian Bicking) works and was really really good. One of the things I discovered in this talk is the way pip can install regular distutils projects is by monkey patching their setup.py file or that at the end, pip’s core is just a hack around setuptools

Then I moved into the PyPy talk which was quite boring to me and finally the lighting talks. Some were better than another but I specially liked the one about Qtile, a tiled window manager written in Python. The guy speaking was just hilarious.

I forgot to menion that I meet some old friends this day like Guilherme Salgado (from my days at Async) and John Ehresman, from Wingware. It was really nice to see them again and hear that they are doing pretty well.

The guy on the left is freaking awesome

As this post is already quite long I talk more about my PyCON experiences in a following post.

When I started to thing about using Mercurial at Yaco I sketched up a list of things that needed to be done:

1. Migrate to Trac 12 since it has support for multiple repositories of different types. This is really important not only because you can see your Mercurial repos in Trac but because it allow us to migrate step by step. You can have a Subversion repository for the QA and graphic design people and one or more Mercurial repos for developers. Also it’s up to the project manager to decide wheter he wants to use Subversion or Mercurial. We don’t want to impose a technology when Subversion is just fine for many situations.
2. Add several hooks in different stages of a changeset lifecycle:
   1. Before commit, check basic things about the quality of the code (pyflakes, pep8, pdb, …)
   2. Before commit check that the message mentions one or more Trac tickets
   3. After commit update one or more Trac tickets
3. Serve the repositories via https and add a web interface to them. This mean adding authentication and authorization for them too.
4. Train our people to use the new system
5. Define common workflows for our development patterns

Step 1 was done several months ago and we are really happy with Trac 12.

Step 2 was achieved using hghooks and hghooks.yaco, a couple of python packages I wrote to suite this goal.

Step 3 is the reason of this post. More on this later.

We are between step 4 and 5 and right now and that is causing minor griefs among our developers

So, let’s talk about authentication and authorization in the context of Mercurial repositories. Mercurial comes with a WSGI application for serving the repositories via HTTP called hgwebdir. It’s pretty easy to setup and has a nice looking web interface with lots of cool features.

But we have a couple of problems with hgwebdir:

• It doesn’t handle autthentication at all. This is actually a feature, but we need to add some bits for this.
• It reads authorization from a different file for each repository it handles. This authorization information does not have group support neither. This mean we need to duplicate a lot of information among repositories. Most of our developers have access to most of our repositories, while each customer has (read only) access just to the repositories related to his project.

With Subversion we do LDAP authentication with an Apache module and use mod_authz_svn for authorization purposes. This last module is really good and it allows you to keep all your authorization information in a single file. Really convenient.

Besides that, we would like to experiment with web servers other than Apache like nginx or Cherokee so we don’t want to depend on Apache for authentication neither.

Our solution involves using repoze.who for authentication and repoze.what for authorization. These are great Python packages that allows you to build really flexible middleware stuff. repoze.who and repoze.what don’t do anything useful by their own but define a set of rules and components to interact with the users and applications. Then there are plugins for both packages that really do the authorization and authentication stuff. One nice thing about repoze is that it uses the WSGI standard heavily so it’s easy to integrate them with other WSGI applications like hgwebdir.

WSGI middleware is like onion layers

We use repoze.who.plugins.ldap.LDAPSearchAuthenticatorPlugin for authentication. It connects to our LDAP server and check that the user and password entered by the user matches a user in our directory. The way the user enter his user and password is through authentication basic. repoze.who allows several other mechanisms but that one is fine for us.

Then for authorization we use the repoze.what.plugins.hgwebdir package that I wrote for our needs but trying to make it general enough so it can be used by other people. This plugin allows the system administrator to write all the repositories authorization information in a single file very similar to the one used by mod_authz_svn.

To put everything together we use a buildout that configures Mercurial, a nginx webserver and all the WSGI middleware and applications to serve the repositories.

Let’s see how our final WSGI application looks like:

from optparse import OptionParser
import sys
from wsgiref.simple_server import make_server

import mercurial.util
import mercurial.dispatch

from mercurial.hgweb.hgwebdir_mod import hgwebdir

from repoze.who.plugins.basicauth import BasicAuthPlugin
from repoze.who.plugins.ldap import LDAPSearchAuthenticatorPlugin

from repoze.what.middleware import setup_auth

from repoze.what.plugins.hgwebdir.adapters import HgwebdirGroupsAdapter
from repoze.what.plugins.hgwebdir.adapters import HgwebdirPermissionsAdapter
from repoze.what.plugins.hgwebdir.adapters import get_public_repositories
from repoze.what.plugins.hgwebdir.middleware import protect_repositories

def add_auth(app, authz_file):
   # Defining the group adapters; you may add as much as you need:
   groups = {'all_groups': HgwebdirGroupsAdapter(authz_file)}

   # Defining the permission adapters; you may add as much as you need:
   permissions = {'all_perms': HgwebdirPermissionsAdapter(authz_file)}

   # repoze.who identifiers; you may add as much as you need:
   basicauth = BasicAuthPlugin('Yaco Hg repositories')
   identifiers = [('basicauth', basicauth)]

   # repoze.who authenticators; you may add as much as you need:
   ldap_auth = LDAPSearchAuthenticatorPlugin('ldap://ldap.yaco.es',
                                     'ou=People,dc=yaco,dc=es', returned_id='login')
   authenticators = [('ldap_auth', ldap_auth)]

   # repoze.who challengers; you may add as much as you need:
   challengers = [('basicauth', basicauth)]

   app_with_auth = setup_auth(
                         app,
                         groups,
                         permissions,
                         identifiers=identifiers,
                         authenticators=authenticators,
                         challengers=challengers)
   return app_with_auth

def make_app(hg_config, authz_file):
   original_app = hgwebdir(hg_config)
   secured_app = protect_repositories(original_app,
                         get_public_repositories(authz_file))
   return add_auth(secured_app, authz_file)

The main entry point here is the make_app function. It recieves two filenames, one for configuring the repositories themselves and one for the authorization stuff. As you can see we are adding a couple of layers to the hgwebdir application. The inner one deals with authorization and the outer one with authentication. We have to deactivate hgwebdir authorization support to make it work. Otherwise authorization is done by two different modules and it won’t work. To deactivate it we configure all of our repositories with this option:

[web]
allow_push = *

By default hgwebdir allows read access to everybody and denies write access to everybody. We keep read access as hgwebdir defaults and add write access to everybody. repoze.what.plugins.hgwebdir will handle this stuff for us.

Now, to configure the authorization you just need to write a file like this:

[groups]
group1 = lgs, mviera
group2 = john, peter

[repo1]
@group1 = rw

[repo2]
@group2 = rw
@group1 = r

[repo3]
* = r
@group1 = rw

As I mentioned before, the syntax and semantics are very similar to the ones used in mod_authz_svn. One feature that is not possible with repoze.what.hgwebdir is specific authorization rules to some directories inside a repository.

Final arquitecture

I hope everything is clearer with this last figure.

Before I start with the inner details of how all these buzzwords work together, let met introduce you the antisocial python development law:

isolation = k * repeatability

Where k is a constant that is inversely proportional to the probability of you saying “It works on my machine”.

That’s it, the more isolate you are from the operating system the more easy is to repeat your environment. As a consequence the more you avoid using your system python and packages the easier your coworkers and deployment engineers will get your system up and runing. Let’s see why:

• By not making any assumption of what packages and libraries you have installed on your machine you are democratizing your team. Everybody has the same oportunities to see how the system fails.
• Using your system python is a ticket to the deploy-this-on-the-very-old-production-server-nightmare rollercaster. It may work but it will probably not.
• If one day is working and the next day is not and you swear your god you didn’t touch anything you may not be isolated enough and an update on your operating system or your build environment has changed without you noticing it. Reciprocally you will be able to upgrade your operating system more often without worries of breaking your projects.
• No more lost hours debugging a crash just to realize that you didn’t run a service that you were supposed to run but you are not running because there are so many things you need to do by hand and you don’t have an automated script to do so that your brain collapses.

Everybody has experienced the frustration feeling of not being able to reproduce a problem. So follow my advice and isolate yourself as much as you can.

So you get it but the question is: how do I isolate myself from the scary, treacherous, evil system? Well, like in hospitals and nuclear stations there are several different levels of isolation, from safe, to super safe, to paranoid.

Safe level Option A

Just build your custom python by dowloading the tarball, uncompress it and do the configure, make and make install dance.

Advantages: you don’t mess with the system, the system doesn’t mess with you. No more fear to upgrade your distro, nothing will break. No more fear to update a package you need, your desktop application won’t break. Another advantage is that you can do the same in the deployment server if its operating system is too old and you are stuck with python 2.4.
Disadvantages: be careful when compiling your python or you will get a barebone executable with no support for a bunch of things like ssl, zip, bzip2, readline, … The solution is to install the development packages from your distro before compiling your python.

Safe level Option B

Use virtualenv with –no-site-packages and you will get a directory completely isolated from you system python.

Advantages: very fast and cheap to get started since you don’t need to compile python. It actually uses your system python with all its features but without any of your system packages.
Disadvantages: it still uses your system python so when you deploy you may find your server python is different. Deploy soon and you will avoid nasty surprises.

Safe level Option C

Use buildout with your system python. Use this option for a recently installed operating system while your site-packages is still clean.

Advantages: easy to get started
Disadvantages: you need to be very disciplinated in orde not to polute your system site-packages. For example, everytime you use easy_install or pip it will get dirty. Always install your dependencies through buildout.

Super safe level

Combine Safe level A or B with buildout. Now you can share your custom python or your virtualenv with different Django projects and nothing will get messy since all your dependencies will get installed inside buildout.

Paranoid level

Compile your python, use virtualenv and buildout. This is like having sex with two condoms and anticonception pills. Maybe not the most pleasure thing but certainly the safest.

I’ve been pretty well with custom python and buildout, e.g. super safe level but sometimes, under the right circunstances, I’ve played paranoid level. Some people in my office prefer virtualenv and buildout. It doesn’t matter as long as you understand the risks and are consistent.

The nice thing about using buildout is that it takes care of your python dependencies and other parts of your build automatically, for you. So having your development environment set up is a matter of minutes and it’s pretty automated.

So, what does our standard buildout do for us? A bunch of things:

1. Download a specific Django version and put it on your PYTHONPATH
2. Download all your python dependencies and put them on your PYTHONPATH
3. Download and compile uwsgi
4. Download and compile nginx
5. Set nginx and uwsgi to work together with your Django project
6. Set supervisord to run and monitor everything

And all this stuff is done automatically, with a couple of commands. I think it’s a pretty advanced deployment that you can have in a few minutes. Our buildout.cfg file looks like this:

[buildout]
parts =
      python
      django
      uwsgi-download
      uwsgi-compilation
      nginx
      nginx-conf
      yourproject
      zc.sourcerelease
      releaser
      service

develop = .
eggs = yourproject

[python]
recipe = zc.recipe.egg
interpreter = py
eggs = ${buildout:eggs}

[django]
recipe = djangorecipe
version = http://code.djangoproject.com/svn/django/branches/releases/1.2.X
project = yourproject
settings = settings
wsgi = true
eggs = ${buildout:eggs}

[uwsgi-download]
recipe = gocept.download
url = http://projects.unbit.it/downloads/uwsgi-0.9.5.4.tar.gz
strip-top-level-dir = true
md5sum = 822c846484dcd6cc9f6d79118ed7e97a

[uwsgi-compilation]
recipe = yaco.recipe.uwsgi
uwsgi-location = ${uwsgi-download:location}

[uwsgi-conf]
socket-path = ${buildout:directory}/var/run/uwsgi.sock
pid-path = ${buildout:directory}/var/run/uwsgi.pid
log-path = ${buildout:directory}/var/log/uwsgi.log
python-path = ${buildout:directory}/${django:project}/
wsgi-path = ${buildout:bin-directory}/django.wsgi

[nginx]
recipe = zc.recipe.cmmi
url = http://nginx.org/download/nginx-0.8.34.tar.gz
extra_options =
    --conf-path=${buildout:directory}/etc/nginx/nginx.conf
    --error-log-path=${buildout:directory}/var/log/nginx-error.log
    --http-log-path=${buildout:directory}/var/log/nginx-access.log
    --pid-path=${buildout:directory}/var/run/nginx.pid
    --lock-path=${buildout:directory}/var/lock/nginx.lock
    --add-module=${uwsgi-download:location}/nginx/

[nginx-conf]
recipe = collective.recipe.template
input = ${buildout:directory}/templates/nginx.conf.in
output = ${buildout:directory}/etc/nginx/nginx.conf
port = 8080
project_location = ${buildout:directory}/${django:project}
django_location = ${django:location}
uwsgi_sock_path = ${uwsgi-conf:socket-path}
uwsgi_location = ${uwsgi-download:location}

[supervisor]
recipe = collective.recipe.supervisor
port = 9080
user = admin
password = admin
pidfile = ${buildout:directory}/var/run/supervisord.pid
serverurl = http://localhost:${supervisor:port}
plugins = superlance
programs =
    0 nginx (stderr_logfile=NONE stdout_logfile=${buildout:directory}/var/log/nginx-error.log) ${nginx:location}/sbin/nginx [ -c ${buildout:directory}/etc/nginx/nginx.conf ] true
    1 uwsgi (stderr_logfile=NONE stdout_logfile=${uwsgi-conf:log-path}) ${buildout:bin-directory}/uwsgi [ -p 1 -C -A 4 -m -s ${uwsgi-conf:socket-path} --wsgi-file ${uwsgi-conf:wsgi-path} --pythonpath ${uwsgi-conf:python-path} --pidfile ${uwsgi-conf:pid-path} ] true

Some notes about this file:

• You should replace the string ‘yourproject’ with the name of your Django project
• The python part is useful to have a custom python interpreter in bin/py with the PYTHONPATH configured the same as your Django project. I use it to debug import errors.
• In the Django part it will create yourproject if it does not already exist
• The yaco.recipe.uwsgi recipe is needed in the uwsgi-compilation part because no matter how good uwsgi is, its build system is really bad and hard to integrate with this kind of tools.
• Note that the supervisord web interface will be on port 9080 and the nginx serving your Django on port 8080
• You still need the nginx.conf.in template that the nginx-conf part uses but it’s just a simple nginx configuration file with a couple of string substitutions.

Alright, before running the bootstrap thing, make sure you are a good Python citizen and you have the common project metadata files like README, CHANGES, LICENSE and more important, the setup.py distutils/setuptools file. Buildout is going to need it. You should put your python dependencies in the install_requires argument of the setup function. Something like this:

import os
from setuptools import setup, find_packages

def read(*rnames):
    return open(os.path.join(os.path.dirname(__file__), *rnames)).read()

setup(
    name='yourproject',
    version='0.1.0',
    url='http://url-to-your-project',
    license='GPL',
    description='...',
    long_description=(read('README') + '\n\n' + read('CHANGES.txt')),
    author='Your Name ',
    classifiers=[
        'Development Status :: early stages',
        'Framework :: Django',
        'Intended Audience :: Developers',
        'License :: GPL',
        'Operating System :: OS Independent',
        'Programming Language :: Python',
        ],
    packages=find_packages(yourproject'),
    package_dir={'': yourproject'},
    install_requires=[
        'distribute',
        'django-piston',
        'feedparser',
        'South',
        'MySQL-python',
        'python-dateutil',
        'httplib2',
        'django-celery',
        ],
)

As you can see I have a bunch of dependencies in my example.

Now do the buildout dance:

python bootstrap.py --distribute
bin/buildout

And make sure you are not using your system python in the first command or a kitten will get killed.

After a few minutes you will be able to run the following commands:

bin/django syncdb     # to create your database tables
bin/django runserver  # to run your development server
bin/supervisord       # to run your production lighting fast server

And the best thing is that once you have this all setup, your team partners will have all of this for free and all of you will have exactly the same environment. You won’t hear “It works on my computer” again.

All this is good and pretty fast to deploy but we want more. How about creating a source distribution in a tarball that we can copy to our remote server and run with all our dependencies frozen? You asked it, you got it! Add this to your buildout:

parts += releaser

[releaser]
recipe = zc.recipe.egg:script
eggs = zc.sourcerelease

Now commit that change and run these commands:

bin/buildout
bin/buildout-source-release hhttps://your-svn-repository-url/path/to/your/project buildout.cfg --name yourproject-0.1.0

After a couple of minutes you will have a file called yourproject-0.1.0.tgz with everything your project needs inside. Go to the server, untar it and run these commands:

python install.py bootstrap
python install.py

Just make sure the python on your server is the same version as the python you used to create the source release. This will deploy the buildout without connecting to the network to download anything since it has everything it needs.

Pretty neat, huh? Well, we really want more. How about a rpm package or a debian one? Having your project in your server operating system package is good for updates since it will respect the changes that you did to your configuration files and the system administrator will be very happy.

To get the rpm you just need to write a simple spec that pick the yourproject-0.1.0.tgz file and run the install.py in a specific location. I even build a custom python inside the rpm file to avoid server incompatibilities. I starting to think about creating two separate packages, one for the python and one for the project and make the later depends on the former. That would be much cleaner, but for now the first option does the job perfectly. I’ve never written a spec file before but I must admit it was much easier than I thought. Sadly, I can’t say the same about Debian packages. But the important thing is that the idea is the same.

So, that’s how we build and deploy our final solutions to our clients and we and they are very happy with the outcomes. But wait, there is more. How about nighty builds? Building and deploying rpm packages just for the nighty builds is quite heavy so that’s where we are using Fabric.

Manuel Viera has created a system that roughly does the following:

1. Creates a OpenVZ virtual machine programatically from a CentOS 5.5 or Debian 5.0 template
2. Configure its network programmatically too
3. Execute a fabric script that:
   1. Connects to the fresh virtual machine through ssh
   2. Install a bunch of operating system packages that will be needed to run a buildout. These packages are listed in a file that every project has: deployment.cfg
   3. Creates a user and a group
   4. With that user, fetch, configure, compile and install a custom python. The version of this python is also in the deployment.cfg file.
   5. Copy the project files to the virtual machine or, in some cases, just do a checkout/clone
   6. Run the bootstrap and buildout dance with the python it just compiled.
   7. Run the test suite and fetch the results
   8. Optionally, creates the source distribution and the rpm package
4. Destroy the virtual machine

And we are on the process to run this algorithm for every project, every night. I bet that will give us another point in the Joel Test

Kudos to Manu and his mighty powers.

I’ve been using Evolution, the mail client, since 2000. I think it was the first Linux program along with Netscape that I remember using on a daily basis. What can I say about a program I’ve used for more than 10 years? Well I’m very grateful to it and I hope its development continues and it gets better and better every day. But now it’s time for a change.

Motivation

There are two main reasons I wan to switch from Evolution to Mutt:

• Speed. Evolution is now much faster than it used to be but when you have 5 Gigabytes of email it starts to behave slowly.
• Knowledge. As an engineer I don’t feel comfortably using something I don’t totally understand and I even feel embarrased about my (lack of) understanding of email, something that I use (dammit, sometimes I even depend on) every day. That’s why switching to Mutt makes sense. Its take on email forces you to do it the Unix way. This mean using several small programs together to accomplish your task and by dividing the task into small pieces you understand the big picture much better. Now I can tell the difference between a MTA, a MUA  and a MDA.

General architecture

As you can see in the above figure, there are quite a lot of small programms (blue boxes and red circle) cooperating to deliver my mail for me. I’ll try to explain what each of these ones do and how to configure them in the following sections.

As there are a lot of configuration files involved I won’t paste them here but point to the Mercurial repository where I keep them versioned.

Step 1. Fetching email: getmail

getmail is a MRA or Mail Retrieval Agent. Its main task is fetch your email from a POP3 or IMAP server. Unless you have a fixed IP and you run your own SMTP and IMAP/POP3 server you probably have your email delivered to your ISP or company’s own servers.

Another popular MRA is fetchmail. I use getmail because it’s way simpler to configure and whetever I have to decided between two programs and one of them is written in python I go for that one. I’m very comfortable with Python so if I ever need to change something or go to the source to understand what’s going on, that way is easier for me.

Anyway, this is my configuration file for getmail. You have to call the getmail program every time you want to download email from your IMAP/POP3 server. A typical run looks like this:

[lgs@ticotico ~]$ getmail
getmail version 4.20.0
Copyright (C) 1998-2009 Charles Cazabon.  Licensed under the GNU GPL version 2.
SimpleIMAPSSLRetriever:lgs@correo.yaco.es:993:
Enter password for SimpleIMAPSSLRetriever:lgs@correo.yaco.es:993:  *******
  0 messages (0 bytes) retrieved, 0 skipped

As this is a very tedious task I have set up a cron task that runs every 5 minutes. Yes, I’m that anxious about email But the problem is that if I put this in a cron job it will ask for the password every time it runs. One solution is to put the password in the getmailrc configuration file but I don’t feel like sharing my password with the whole world and I really want to keep my configuration files in Bitbucket. So the other solution is to use expect. This is a common Unix tool for manipulating other programs. This is my getmail-no-password expect script:

#!/usr/bin/expect -f

set timeout 600
spawn /usr/bin/getmail
expect -exact "Enter password for SimpleIMAPSSLRetriever:lgs@correo.yaco.es:993:  "
send -- "put-your-real-password-here\r"
expect eof
exit

So expect will call getmail and as soon as the “Enter password …” phrase goes to stdin it will send my real password, followed by the ‘\r’ character. One minor issue is I have to give expect a timeout. After this time has passed expect will stop running. 600 seconds (10 minutes) is more than enough for running this script in my cron job but the first time I synchronize with my IMAP server it takes way more time than this so I just run getmail and enter my password manually instead of using the getmail-no-password expect wrapper.

This script is not in my Bitbucket repository for obvious reasons

This is my crontab for fetching email:

*/5 * * * * /home/lgs/bin/getmail-no-password > /dev/null

Note that I redirect stdout to /dev/null but not stderr so cron will email me if something goes wrong with getmail or the expect wrapper script.

Step 2. Filtering email: maildrop

As soon as getmail gets the mail it can save it to the final destination, e.g. my Maildir directory, or we can send it to the MDA (Mail Delivery Agent) to do more powerful things. In my case I went for dropmail but as always there are other options like procmail. Procmail is more mature in the Unix world but I like dropmail syntax better.

getmail fetches an email message from the server and then run dropmail with the mail message as the argument. As you can see in my dropmail configuration file (called .mailfilter), I do several things with a mail message:

• Pipe it to through reformail to add a Lines header, which is useful in Mutt.
• Pipe it to spambayes (sb_filter) to add antispam features to my mail system. More on this later
• Send a notification to my GNOME desktop in case the message is not spam. More on this later.
• Filter the message and store it in one of my mail folders depending on several regular expression rules that match agains some header values of the mail message.

Step 3. Anti spam: spam bayes

The typical antispam software in Unix system is spamassassin but I use Spam Bayes because of several reasons:

• I setup spambayes in the previous company I worked for and I was very happy with its results.
• In my current company, spamassassin is used and it’s not very effective. This may be a configuration issue.
• Spamassassin eats a lot of cpu.
• Did I mention that spam bayes is written in python?

The way spam bayes work is quite simple: first you need to train it giving a bunch of spam messages and a bunch of ham (ham is the opposite of spam, in other words, good mail) messages. You do it with the following command line:

sb_mboxtrain.py -g .inbox/ -s .spam/

This will create a hammie.db database in Berkeley format. Now you should write a configuration file to tell spam bayes where this database file is located so it can use it to filter spam messages when called from dropmail. Here is my spambayes configuration file.

If you remember my maildrop configuration file this was the line that calls spam bayes:

xfilter "python -W ignore:::: /usr/bin/sb_filter.py -d ~/.hammie.db"

What’s that -W ignore::: argument to Python? Well spam bayes spit some warnings when run under python 2.5 or 2.6 and that’s not good for maildrop because python writes the warnings to stderr and maildrop thinks there is a problem with that process and aborts the mail delivery. So by calling python with the -W ignore::::: argument we tell it to keep the warnings quiet and don’t write them to stderr. I hope spam bayes get updated soon and I can remove this hack.

The good thing about spam bayes is that it can learn with your help. Any time it fails when filtering a message you can retrain it fix its behaviour. By the way, all that spam bayes does is adding some headers to the message and then you need to add some rules to your maildrop configuration file to put the message where it belongs. The interesting thing is that sometimes a message is not spam or ham because spam bayes is not sure. You can store these messages in a folder different than the spam folder so you can look it more frequently and teach spam bayes with them.

Step 4. Reading and composing emails: mutt

From all text based email clients mutt is the most popular. Israel Herraiz told me about Sup and it looks very promising. I think I’ll give it a try pretty soon. But for now, I’ll stick to Mutt for a while.

Mutt is the most complex software from all the programs I mentioned in this post, so its configuration file is the biggest one. Hopefully I have added enough comments to make it easier to understand the options that I use.

Step 5. Sending emails: msmtp

The last step consist in sending emails. Until recently Mutt didn’t know how to talk SMTP, the protocol for sending emails and due to popular demand that feature has been added. But in keeping the original Mutt spirit and thus, the Unix way of using one program for each task, I decided to go for msmtp, a small client for sending emails. As always, there are plenty of options in Linux land but I think msmtp does a very decent job. As usual, you can check my msmtp configuration file in Bitbucket. Remember to set the “set sendmail=”/usr/bin/msmtp” option in Mutt.

In these days full of spam it’s usual that your SMTP server ask you for authorization information unless you are in the same private network. As in the getmail case I didn’t want to write my password in plain text in the msmtp configuration file. Fortunately msmtp has support for GNOME (and MacOS X) keyring so you can simple store your password in the safe keyring of your operating system and msmtp will pick it up for you there. The only issue is that you need to be logged in a graphical GNOME session in order have access to the keyring.

Anyway, to store your password in the keyring use the following command changing your username and server for your own values:

./msmtp-gnome-tool.py –set-password –username lgs –server correo.yaco.es

Unfortunately you need the source code of msmtp to use this command since it is not in the msmtp rpm package that I installed in my Fedora system. So go ahead and grab it. Once you untar it, it should be in the msmtp/scripts/msmtp-gnome-tool/msmtp-gnome-tool.py

Bonus track 1: notifications

Mutt is great and all but one thing I miss from Evolution is GNOME notifications every time a new mail reaches my inbox. Well, with such flexible system as the one I built, that’s not difficult to have. Just add this line to the maildrop configuration file:

xfilter "python /home/lgs/bin/email_notificator.py"

I added after storing spam in the spam folder since I don’t want to be notified when spam arrives.

Here is the content of my email_notificator.py script:

import email
import subprocess
import sys

if __name__ == '__main__':
    data = sys.stdin.read()

    message = email.message_from_string(data)
    from_ = 'Unknown from'
    if 'From' in message:
        from_ = message['From']

    subject = 'Unknown subject'
    if 'Subject' in message:
        subject = message['Subject']

    cmd = ['/usr/bin/notify-send', '-u', 'critical', '-t', '5000',
           '-i', '/usr/share/icons/gnome/48x48/actions/mail_new.png',
           '%s' % from_, '%s' % subject]
    subprocess.call(cmd, env={'DISPLAY': ':0.0'})
    print data

No rocket science here. In the following screenshot you can see the script in action:

Bonus track 2: logs rotation

Several programs of my email ecosystem log information to different files. As always you should rotate your logs if you don’t want your hard disk to become full really fast. Just add a configuration file for log rotate as mine and add a line to your crontab like this one:

0 9 * * * /usr/sbin/logrotate /home/lgs/mail_logs/logrotate.conf --state=/home/lgs/mail_logs/logrotate.status

Bonus track 3: reading mails from my android device

One of the nice things of being a text only program is that you can use mutt from any computer that can connect through a ssh link to the computer you have your mail and mutt installed. In Android there is the fantastic ConnectBot application which let you do just that. See this screenshot in you don’t believe it

Mutt in Android

The only gotchas of using Mutt from my Android phone is that the GNOME integration gets in my way: now I can’t send emails since msmtp will look for the password in the GNOME keyring and there is no such thing in Android. One way to fix this issue is to configure another account in msmtp which doesn’t use the keyring but ask for the password manually and switch to that account with a key combination when using it from the phone.

TODO

• Mark the spam message as already readed so I don’t get distracted every time a spam message arrives my spam folder. Then once in a while I can review them looking for false positives.
• Addressbook integration.
• PGP integration

References

• Mutt Manual
• Mut Wiki
• Mutt newbie guide
• Mutt overview for newbies
• The Woodnotes Guide to the Mutt Email Client
• Spam Bayes wiki

In current Pycha versions there is a padding option but it is not what you expect. The current padding represents the space between the limit of the graphics area and the chart area. The labels and ticks are drawn in this space. What this means is the programmer has to manually adjust the padding in order to make the labels render correctly. When you are using pycha for drawing one or two charts in an environment under your control this is not a big issue. When you draw lots of charts from wild data sources, autopadding helps a lot.

Figure 1. Pie Chart with manual padding

In Figure 1 you can see old style pie charts where we had to manually adjust the padding to make the labels fit. As a side effect the pie was not centered in the graphics area. Here the padding options are: {‘top’: 0, ‘right’: 10, ‘left’: 70, ‘bottom’: 0}. Imagine what happen if the dataset changes: we would probably need to change the padding.

So, what exactly is autopadding? Well, it’s the feature that lets pycha compute the labels and ticks area automatically. Now the padding that you specify is real empty space.

Figure 2. Pie chart with autopadding

For most of the charts that pycha supports this feature is not very difficult. Except for the pie charts. In this case, it took me a while to get it working and the results, while good enough, are not perfect.

In figure 2 you can see such results. As you can see the pie is smaller now and the slices are ordered in a different way. The order really matters and I will probably add an option in the future to indicate the starting angle (now is 0) so there is a small degree of control left to the programmer. Note that this time the four paddings are equal to zero.

Let’s describe the problem: given a rectangular area (width and height) and a list of tuples made of a label and an angle, compute the circunference radius and the label positions so that the pie area is maximized and the labels do not override the pie.

For each label the angle associated with it is the angle that split its pie slice into two slices of the same side. If you have the initial and final angle of each slice (as I do) computing this angle is as easier as summing both and dividing by two.

Step 1. So the first thing I do is compute the center of the area and the bounding boxes of the labels. Any decent graphics library has a function for that and Cairo is no different: cairo_text_extents.

Step 2. Now for each tuple in my label-angle list I draw a ray that starts at the center of the chart and goes into the direction given by the angle. Remember that a rect can be defined with a point and an angle.

Figure 3. Pie Chart in debug mode showing the rays in red

Step 3. I calculate the intersection of these rays with the rectangle that defines my graphics area (drawn in blue in the Figure 3 because debug mode is enabled). Clipping is not useful here since I want the coordinates for every intersection, the rays are not drawn in the final result unless debug mode is enabled. In order to make this computation easier I divide the area in four parts delimited by the two diagonals of the rectangle. Every ray is defined by an ecuation of the form y = mx + n where m is tan (angle) and I know which of the four sides of the rectangle the ray is going to intersect bases on the angle. As every side of the rectangle is also defined by a very simple ecuation, all I have to do is solve an ecuation system for each ray.

Figure 4. Division of the graphics area using the two diagonals

Nice. We have now a center and a bunch of rays and intersection points. We also have bounding boxes for the labels. Now I have to place each bounding box with two constraints:

• Its center lies on its associated ray
• One of its side lies on one of the sides of the big chart rectangle

Step 4. It’s easy to guess what side of the bounding box matches with which side of the chart rectangle. A little bit of trigonometry will do the rest.

Figure 5. Initial labels position

Step 5. Ok, now I have all the labels, or more correctly, their bounding boxes, positioned in the chart rectangle, as far as I can from the center. Now the maximun radius for the pie is the minimum of the distances from the bounding boxes to the center. For every bounding box there is one side that is closer than the others to the center, so the distance we want is the distance from that side to the center. Again, having divided the area in those four parts makes this step this easier. In our example shown in Figure 5 the label that restricts the pie radio is the one that reads “stackedbar.py (6.9%)“.

The good news are that we have achieved one of our goals: compute the radius of our pie chart. The bad news are that was the easy part. In the second round of our algorithm we are going to move the labels bounding boxes as close as we can to the pie. Otherwise they don’t look good and the conection between the slice and its label is lost. As you may guess all the labels can be moved closer to the center except for one. That’s the label that gave as the radius of the pie circunference.

Step 6. So let’s try to move the labels. For this task I will divide the chart rectangle in four sides but this time I’ll use two perpendicular rects, one horizontal and one vertical. So I’ll get my circunference divided in four quadrants. For each label I’ll compute the intersection of its ray and our pie circunference. Remember that now we know its radius.

Figure 6. Division by two perpendicular rects that touch the center

Step 7. For each label place one corner of its bounding box in the point computed in step 6. For the first quadrant this corner is the left down corner, for the second quadrant it’s the right down corner, for the third quadrant it’s the right top corner and for the fourth quadrant it’s the left top corner. This is a feasible position but it doesn’t look very nice so we will try to improve it.

Step 8. For each label draw another circunference with the same center as the pie circunference and the radius equal to the distance between the center of the bounding box and the center of the circunference. Now let’s compute the intersection between this second circunference and the ray of the same label. Voilá, that’s the point where we are going to move the center of the bounding box.

Figure 6. Detail of label reposition

So that’s the end of the algorithm, the labels are now positioned closer to the pie circunference but as I said before, it’s not perfect since for those labels that are near the top and bottom side of the circunference the label could be closer than it is. I may have used a soft computing algorithm that moves the bounding box along its ray until touches the pie circunference but that would be slower.

By the way, I’m using a named branch for this feature and hopefullly it will get merged into the main branch for the 0.7.0 release of pycha.

Anyway, this is what I did to increase the hard disk space without reinstalling the whole thing:

1. Create a new hard disk with Virtual Machine Manager. I created it with another 2 GB and selected VirtIO as the disk type.
2. Start the virtual machine and create a partition in the new disk. /dev/vdb1 is the new disk:

      Device Boot      Start         End      Blocks   Id  System
   /dev/vdb1               1        4063     2047720+  83  Linux

3. Initialize a physical volume on the new disk:

   pvcreate /dev/vdb1

4. Add this physical volume to the logical volume group:

   vgextend VolGroup00 /dev/vdb1

5. Extend the logical volume group:

   lvextend /dev/VolGroup00/LogVol00 -L+1.9G
   Rounding up size to full physical extent 1.91 GB
   Extending logical volume LogVol00 to 3.84 GB
   Logical volume LogVol00 successfully resized

6. Resize the filesystem:

   resize2fs /dev/VolGroup00/LogVol00
   resize2fs 1.39 (29-May-2006)
   Filesystem at /dev/VolGroup00/LogVol00 is mounted on /; on-line resizing required
   Performing an on-line resize of /dev/VolGroup00/LogVol00 to 1007616 (4k) blocks.
   The filesystem on /dev/VolGroup00/LogVol00 is now 1007616 blocks long.

Thanks to LVM now I have 2 more gigabytes on my virtual disk and no reinstallation was needed.

Os dejo la secuencia de fotos con el making of de la funda.

También os dejo los planos en vectorial por si alguno se anima. Estos planos se han mejorado tras la primera funda. Mi madre ya está deseando hacer más para mejorar el resultado.

Nota: la inspiración original es esta. Me gustó tanto que fui a comprarmela pero en aquel momento no tenían en stock así que me puse manos a la obra y seguí el principio punki del “do it yourself”.