Memory Dump

After months of planning and thinking about it we are starting to adding support for Mercurial at Yaco. It’s far from finished but it is already tested in a couple of projects. We used and still use Subversion for most of our projects and our workflow is really tied to it.

When I started to thing about using Mercurial at Yaco I sketched up a list of things that needed to be done:

1. Migrate to Trac 12 since it has support for multiple repositories of different types. This is really important not only because you can see your Mercurial repos in Trac but because it allow us to migrate step by step. You can have a Subversion repository for the QA and graphic design people and one or more Mercurial repos for developers. Also it’s up to the project manager to decide wheter he wants to use Subversion or Mercurial. We don’t want to impose a technology when Subversion is just fine for many situations.
2. Add several hooks in different stages of a changeset lifecycle:
   1. Before commit, check basic things about the quality of the code (pyflakes, pep8, pdb, …)
   2. Before commit check that the message mentions one or more Trac tickets
   3. After commit update one or more Trac tickets
3. Serve the repositories via https and add a web interface to them. This mean adding authentication and authorization for them too.
4. Train our people to use the new system
5. Define common workflows for our development patterns

Step 1 was done several months ago and we are really happy with Trac 12.

Step 2 was achieved using hghooks and hghooks.yaco, a couple of python packages I wrote to suite this goal.

Step 3 is the reason of this post. More on this later.

We are between step 4 and 5 and right now and that is causing minor griefs among our developers

So, let’s talk about authentication and authorization in the context of Mercurial repositories. Mercurial comes with a WSGI application for serving the repositories via HTTP called hgwebdir. It’s pretty easy to setup and has a nice looking web interface with lots of cool features.

But we have a couple of problems with hgwebdir:

• It doesn’t handle autthentication at all. This is actually a feature, but we need to add some bits for this.
• It reads authorization from a different file for each repository it handles. This authorization information does not have group support neither. This mean we need to duplicate a lot of information among repositories. Most of our developers have access to most of our repositories, while each customer has (read only) access just to the repositories related to his project.

With Subversion we do LDAP authentication with an Apache module and use mod_authz_svn for authorization purposes. This last module is really good and it allows you to keep all your authorization information in a single file. Really convenient.

Besides that, we would like to experiment with web servers other than Apache like nginx or Cherokee so we don’t want to depend on Apache for authentication neither.

Our solution involves using repoze.who for authentication and repoze.what for authorization. These are great Python packages that allows you to build really flexible middleware stuff. repoze.who and repoze.what don’t do anything useful by their own but define a set of rules and components to interact with the users and applications. Then there are plugins for both packages that really do the authorization and authentication stuff. One nice thing about repoze is that it uses the WSGI standard heavily so it’s easy to integrate them with other WSGI applications like hgwebdir.

WSGI middleware is like onion layers

We use repoze.who.plugins.ldap.LDAPSearchAuthenticatorPlugin for authentication. It connects to our LDAP server and check that the user and password entered by the user matches a user in our directory. The way the user enter his user and password is through authentication basic. repoze.who allows several other mechanisms but that one is fine for us.

Then for authorization we use the repoze.what.plugins.hgwebdir package that I wrote for our needs but trying to make it general enough so it can be used by other people. This plugin allows the system administrator to write all the repositories authorization information in a single file very similar to the one used by mod_authz_svn.

To put everything together we use a buildout that configures Mercurial, a nginx webserver and all the WSGI middleware and applications to serve the repositories.

Let’s see how our final WSGI application looks like:

from optparse import OptionParser
import sys
from wsgiref.simple_server import make_server

import mercurial.util
import mercurial.dispatch

from mercurial.hgweb.hgwebdir_mod import hgwebdir

from repoze.who.plugins.basicauth import BasicAuthPlugin
from repoze.who.plugins.ldap import LDAPSearchAuthenticatorPlugin

from repoze.what.middleware import setup_auth

from repoze.what.plugins.hgwebdir.adapters import HgwebdirGroupsAdapter
from repoze.what.plugins.hgwebdir.adapters import HgwebdirPermissionsAdapter
from repoze.what.plugins.hgwebdir.adapters import get_public_repositories
from repoze.what.plugins.hgwebdir.middleware import protect_repositories

def add_auth(app, authz_file):
   # Defining the group adapters; you may add as much as you need:
   groups = {'all_groups': HgwebdirGroupsAdapter(authz_file)}

   # Defining the permission adapters; you may add as much as you need:
   permissions = {'all_perms': HgwebdirPermissionsAdapter(authz_file)}

   # repoze.who identifiers; you may add as much as you need:
   basicauth = BasicAuthPlugin('Yaco Hg repositories')
   identifiers = [('basicauth', basicauth)]

   # repoze.who authenticators; you may add as much as you need:
   ldap_auth = LDAPSearchAuthenticatorPlugin('ldap://ldap.yaco.es',
                                     'ou=People,dc=yaco,dc=es', returned_id='login')
   authenticators = [('ldap_auth', ldap_auth)]

   # repoze.who challengers; you may add as much as you need:
   challengers = [('basicauth', basicauth)]

   app_with_auth = setup_auth(
                         app,
                         groups,
                         permissions,
                         identifiers=identifiers,
                         authenticators=authenticators,
                         challengers=challengers)
   return app_with_auth

def make_app(hg_config, authz_file):
   original_app = hgwebdir(hg_config)
   secured_app = protect_repositories(original_app,
                         get_public_repositories(authz_file))
   return add_auth(secured_app, authz_file)

The main entry point here is the make_app function. It recieves two filenames, one for configuring the repositories themselves and one for the authorization stuff. As you can see we are adding a couple of layers to the hgwebdir application. The inner one deals with authorization and the outer one with authentication. We have to deactivate hgwebdir authorization support to make it work. Otherwise authorization is done by two different modules and it won’t work. To deactivate it we configure all of our repositories with this option:

[web]
allow_push = *

By default hgwebdir allows read access to everybody and denies write access to everybody. We keep read access as hgwebdir defaults and add write access to everybody. repoze.what.plugins.hgwebdir will handle this stuff for us.

Now, to configure the authorization you just need to write a file like this:

[groups]
group1 = lgs, mviera
group2 = john, peter

[repo1]
@group1 = rw

[repo2]
@group2 = rw
@group1 = r

[repo3]
* = r
@group1 = rw

As I mentioned before, the syntax and semantics are very similar to the ones used in mod_authz_svn. One feature that is not possible with repoze.what.hgwebdir is specific authorization rules to some directories inside a repository.

Final arquitecture

I hope everything is clearer with this last figure.

Introduction

I’ve been using Evolution, the mail client, since 2000. I think it was the first Linux program along with Netscape that I remember using on a daily basis. What can I say about a program I’ve used for more than 10 years? Well I’m very grateful to it and I hope its development continues and it gets better and better every day. But now it’s time for a change.

Motivation

There are two main reasons I wan to switch from Evolution to Mutt:

• Speed. Evolution is now much faster than it used to be but when you have 5 Gigabytes of email it starts to behave slowly.
• Knowledge. As an engineer I don’t feel comfortably using something I don’t totally understand and I even feel embarrased about my (lack of) understanding of email, something that I use (dammit, sometimes I even depend on) every day. That’s why switching to Mutt makes sense. Its take on email forces you to do it the Unix way. This mean using several small programs together to accomplish your task and by dividing the task into small pieces you understand the big picture much better. Now I can tell the difference between a MTA, a MUA  and a MDA.

General architecture

As you can see in the above figure, there are quite a lot of small programms (blue boxes and red circle) cooperating to deliver my mail for me. I’ll try to explain what each of these ones do and how to configure them in the following sections.

As there are a lot of configuration files involved I won’t paste them here but point to the Mercurial repository where I keep them versioned.

Step 1. Fetching email: getmail

getmail is a MRA or Mail Retrieval Agent. Its main task is fetch your email from a POP3 or IMAP server. Unless you have a fixed IP and you run your own SMTP and IMAP/POP3 server you probably have your email delivered to your ISP or company’s own servers.

Another popular MRA is fetchmail. I use getmail because it’s way simpler to configure and whetever I have to decided between two programs and one of them is written in python I go for that one. I’m very comfortable with Python so if I ever need to change something or go to the source to understand what’s going on, that way is easier for me.

Anyway, this is my configuration file for getmail. You have to call the getmail program every time you want to download email from your IMAP/POP3 server. A typical run looks like this:

[lgs@ticotico ~]$ getmail
getmail version 4.20.0
Copyright (C) 1998-2009 Charles Cazabon.  Licensed under the GNU GPL version 2.
SimpleIMAPSSLRetriever:lgs@correo.yaco.es:993:
Enter password for SimpleIMAPSSLRetriever:lgs@correo.yaco.es:993:  *******
  0 messages (0 bytes) retrieved, 0 skipped

As this is a very tedious task I have set up a cron task that runs every 5 minutes. Yes, I’m that anxious about email But the problem is that if I put this in a cron job it will ask for the password every time it runs. One solution is to put the password in the getmailrc configuration file but I don’t feel like sharing my password with the whole world and I really want to keep my configuration files in Bitbucket. So the other solution is to use expect. This is a common Unix tool for manipulating other programs. This is my getmail-no-password expect script:

#!/usr/bin/expect -f

set timeout 600
spawn /usr/bin/getmail
expect -exact "Enter password for SimpleIMAPSSLRetriever:lgs@correo.yaco.es:993:  "
send -- "put-your-real-password-here\r"
expect eof
exit

So expect will call getmail and as soon as the “Enter password …” phrase goes to stdin it will send my real password, followed by the ‘\r’ character. One minor issue is I have to give expect a timeout. After this time has passed expect will stop running. 600 seconds (10 minutes) is more than enough for running this script in my cron job but the first time I synchronize with my IMAP server it takes way more time than this so I just run getmail and enter my password manually instead of using the getmail-no-password expect wrapper.

This script is not in my Bitbucket repository for obvious reasons

This is my crontab for fetching email:

*/5 * * * * /home/lgs/bin/getmail-no-password > /dev/null

Note that I redirect stdout to /dev/null but not stderr so cron will email me if something goes wrong with getmail or the expect wrapper script.

Step 2. Filtering email: maildrop

As soon as getmail gets the mail it can save it to the final destination, e.g. my Maildir directory, or we can send it to the MDA (Mail Delivery Agent) to do more powerful things. In my case I went for dropmail but as always there are other options like procmail. Procmail is more mature in the Unix world but I like dropmail syntax better.

getmail fetches an email message from the server and then run dropmail with the mail message as the argument. As you can see in my dropmail configuration file (called .mailfilter), I do several things with a mail message:

• Pipe it to through reformail to add a Lines header, which is useful in Mutt.
• Pipe it to spambayes (sb_filter) to add antispam features to my mail system. More on this later
• Send a notification to my GNOME desktop in case the message is not spam. More on this later.
• Filter the message and store it in one of my mail folders depending on several regular expression rules that match agains some header values of the mail message.

Step 3. Anti spam: spam bayes

The typical antispam software in Unix system is spamassassin but I use Spam Bayes because of several reasons:

• I setup spambayes in the previous company I worked for and I was very happy with its results.
• In my current company, spamassassin is used and it’s not very effective. This may be a configuration issue.
• Spamassassin eats a lot of cpu.
• Did I mention that spam bayes is written in python?

The way spam bayes work is quite simple: first you need to train it giving a bunch of spam messages and a bunch of ham (ham is the opposite of spam, in other words, good mail) messages. You do it with the following command line:

sb_mboxtrain.py -g .inbox/ -s .spam/

This will create a hammie.db database in Berkeley format. Now you should write a configuration file to tell spam bayes where this database file is located so it can use it to filter spam messages when called from dropmail. Here is my spambayes configuration file.

If you remember my maildrop configuration file this was the line that calls spam bayes:

xfilter "python -W ignore:::: /usr/bin/sb_filter.py -d ~/.hammie.db"

What’s that -W ignore::: argument to Python? Well spam bayes spit some warnings when run under python 2.5 or 2.6 and that’s not good for maildrop because python writes the warnings to stderr and maildrop thinks there is a problem with that process and aborts the mail delivery. So by calling python with the -W ignore::::: argument we tell it to keep the warnings quiet and don’t write them to stderr. I hope spam bayes get updated soon and I can remove this hack.

The good thing about spam bayes is that it can learn with your help. Any time it fails when filtering a message you can retrain it fix its behaviour. By the way, all that spam bayes does is adding some headers to the message and then you need to add some rules to your maildrop configuration file to put the message where it belongs. The interesting thing is that sometimes a message is not spam or ham because spam bayes is not sure. You can store these messages in a folder different than the spam folder so you can look it more frequently and teach spam bayes with them.

Step 4. Reading and composing emails: mutt

From all text based email clients mutt is the most popular. Israel Herraiz told me about Sup and it looks very promising. I think I’ll give it a try pretty soon. But for now, I’ll stick to Mutt for a while.

Mutt is the most complex software from all the programs I mentioned in this post, so its configuration file is the biggest one. Hopefully I have added enough comments to make it easier to understand the options that I use.

Step 5. Sending emails: msmtp

The last step consist in sending emails. Until recently Mutt didn’t know how to talk SMTP, the protocol for sending emails and due to popular demand that feature has been added. But in keeping the original Mutt spirit and thus, the Unix way of using one program for each task, I decided to go for msmtp, a small client for sending emails. As always, there are plenty of options in Linux land but I think msmtp does a very decent job. As usual, you can check my msmtp configuration file in Bitbucket. Remember to set the “set sendmail=”/usr/bin/msmtp” option in Mutt.

In these days full of spam it’s usual that your SMTP server ask you for authorization information unless you are in the same private network. As in the getmail case I didn’t want to write my password in plain text in the msmtp configuration file. Fortunately msmtp has support for GNOME (and MacOS X) keyring so you can simple store your password in the safe keyring of your operating system and msmtp will pick it up for you there. The only issue is that you need to be logged in a graphical GNOME session in order have access to the keyring.

Anyway, to store your password in the keyring use the following command changing your username and server for your own values:

./msmtp-gnome-tool.py –set-password –username lgs –server correo.yaco.es

Unfortunately you need the source code of msmtp to use this command since it is not in the msmtp rpm package that I installed in my Fedora system. So go ahead and grab it. Once you untar it, it should be in the msmtp/scripts/msmtp-gnome-tool/msmtp-gnome-tool.py

Bonus track 1: notifications

Mutt is great and all but one thing I miss from Evolution is GNOME notifications every time a new mail reaches my inbox. Well, with such flexible system as the one I built, that’s not difficult to have. Just add this line to the maildrop configuration file:

xfilter "python /home/lgs/bin/email_notificator.py"

I added after storing spam in the spam folder since I don’t want to be notified when spam arrives.

Here is the content of my email_notificator.py script:

import email
import subprocess
import sys

if __name__ == '__main__':
    data = sys.stdin.read()

    message = email.message_from_string(data)
    from_ = 'Unknown from'
    if 'From' in message:
        from_ = message['From']

    subject = 'Unknown subject'
    if 'Subject' in message:
        subject = message['Subject']

    cmd = ['/usr/bin/notify-send', '-u', 'critical', '-t', '5000',
           '-i', '/usr/share/icons/gnome/48x48/actions/mail_new.png',
           '%s' % from_, '%s' % subject]
    subprocess.call(cmd, env={'DISPLAY': ':0.0'})
    print data

No rocket science here. In the following screenshot you can see the script in action:

Bonus track 2: logs rotation

Several programs of my email ecosystem log information to different files. As always you should rotate your logs if you don’t want your hard disk to become full really fast. Just add a configuration file for log rotate as mine and add a line to your crontab like this one:

0 9 * * * /usr/sbin/logrotate /home/lgs/mail_logs/logrotate.conf --state=/home/lgs/mail_logs/logrotate.status

Bonus track 3: reading mails from my android device

One of the nice things of being a text only program is that you can use mutt from any computer that can connect through a ssh link to the computer you have your mail and mutt installed. In Android there is the fantastic ConnectBot application which let you do just that. See this screenshot in you don’t believe it

Mutt in Android

The only gotchas of using Mutt from my Android phone is that the GNOME integration gets in my way: now I can’t send emails since msmtp will look for the password in the GNOME keyring and there is no such thing in Android. One way to fix this issue is to configure another account in msmtp which doesn’t use the keyring but ask for the password manually and switch to that account with a key combination when using it from the phone.

TODO

• Mark the spam message as already readed so I don’t get distracted every time a spam message arrives my spam folder. Then once in a while I can review them looking for false positives.
• Addressbook integration.
• PGP integration

References

• Mutt Manual
• Mut Wiki
• Mutt newbie guide
• Mutt overview for newbies
• The Woodnotes Guide to the Mutt Email Client
• Spam Bayes wiki

Simon Ilyushchenko sent me a patch some weeks ago about tick labels in Pycha. The patch fixes the label position in the case where it goes out of the surface dimensions. The problem with his patch is that is too specific and only works well for the most simple case. For example, it doesn’t solve the problem:

• when you have rotated labels
• when you have pie charts
• for the y axis tick labels

And that’s why I didn’t add his patch to pycha. When developing the library I always try to make it as orthogonal and predictable as possible, so if a feature is added and it should work in all places where it makes sense. Users that send me patches are not usually concerned about edge cases but that’s my work as the library author and maintainer.

But the real point is that Simon raised a real issue with Pycha that I was worried about since the very beginning: right now the user has to adjust the padding options to make sure that tick labels and other things are kept inside the surface. Clearly this is suboptimal and for automatically computed charts without human intervention it can be a real issue.

So let me explain how pycha uses its real state space right now. First, the user tells the library how big the rendering surface is going to be: let’s call these parameters surface_width and surface_height. Then there are these padding options: padding.top, padding.bottom, padding.left and padding.right. Let’s see it in the following figure:

Now we have a padding area and a chart area. The chart area is where pycha draws the chart. The axis (when used) are drawn in the boundary of the chart area and the tick labels and axis titles are drawn in the padding rectangle. Other things like the chart title and the legend are also drawn in that rectangle. You can see it in the next figure where the chart area has a blue background:

Now, I want to change this and make Pycha automatically compute all its elements so the padding is actually that, blank space. Optimally Pycha should not draw anything into the padding rectangle. Well, there maybe some exceptions, see the end of this post for more information about them. So let’s define some new areas:

The goal is to make Pycha compute all these areas automatically so the chart area is reduced to accomodate all the other things the library draws in the surface without cluttering the padding area. This is a big change and it won’t happen for 0.5.2 but I’ll try to have it for the 0.6.0 release. Obviously this is a backwards incompatible change and new charts rendered with the same options will look smaller. Maybe I’ll set the default values for the padding options to zero to minimize the update impact.

The way Pycha will compute these areas is actually pretty similar than the way graphical toolkits like Gtk+, Tk, Qt, Swing, GWT do it when using layout containers where the widgets are placed in. There is a layout phase where the area’s dimensions are negociated and calculated and then there is a rendering phase when the elements/widgets are actually drawn. For bar and line charts this may look overkill but for other charts like pie charts is really the only sane way to do it.

The legend is a special case and I’m still not sure about positioning it using ‘absolute’ coordinates or give it its own area that is taken into account for the layout algorithm.