Digital Achievement

Python and Jabber: Head to Head Library Review

Posted by Tim Freund Tue, 12 Aug 2008 03:42:00 GMT

Python is known as a batteries included language, but sometimes it can feel like a home improvement project gone bad. You know, the kind of project where there seems to be too many parts for the work that remains? I had that feeling as I tried to figure out which Jabber/XMPP library would suit my application best. Like any programmer worth their salt, my first tool for research is a visit to Google. Two searches, python jabber and python xmpp, seemed to turn up three viable projects.

• jabber.py (GNU LGPL)
• PyXMPP (GNU LGPL)
• xmpppy (GNU GPL)

There’s also a reference to Twisted’s support for XMPP in the first page of search results, but we’ll save Twisted for another day. I haven’t grasped Twisted enough to feel comfortable writing about it, and the application that I am working on doesn’t use Twisted in any way. Keep an eye on these guys for more information regarding Twisted’s XMPP support.

All three of the options seem like reasonable choices, so we will review each of the packages for the following criteria:

• Installation
• Ease of Use
• Support Options

Installation

The trial begins with an empty virtualenv environment. We have a blank slate into which we can install the projects. We will first attempt easy_install $PROJECT_NAME, and then easy_install $DOWNLOAD_URL. Any library that won’t easily install with easy_install is going to be a source of grief form any team or project that depends on it.

jabber.py

(xmpp)tim@prime:~/src/xmpp$ easy_install jabber.py
...
error: Could not find suitable distribution for Requirement.parse('jabber.py')
(xmpp)tim@prime:~/src/xmpp$ easy_install "http://downloads.sourceforge.net/jabberpy/jabberpy-0.5-0.tar.gz?modtime=1075826815&big_mirror=0" 
...
Finished processing dependencies for jabber.py==0.3-1

It appears that jabber.py isn’t in the PyPI index at all, and when we attempt to download and install jabber.py version 0.5-0, we are told that we have just installed version 0.3-1. I downloaded jabberpy-0.5-0.tar.gz and confirmed that setup.py reports the distribution as version 0.3-1. One other slightly confusing issue: the README file reports jabber.py as a GPL project, but the home page and setup.py files report it as LGPL. The xmpppy project also refers to jabber.py as LGPL. Best 3 out of 4 licensing?

The good news: jabber.py installation is painless on Windows, OS X, and Linux, as long as a download URL is provided.

PyXMPP

(xmpp)tim@prime:~/src/xmpp$ easy_install PyXMPP
...
Reading http://pypi.python.org/simple/
Reading http://pypi.python.org/simple/pyxmpp/
Reading http://pyxmpp.jabberstudio.org/
No local packages or download links found for PyXMPP
error: Could not find suitable distribution for Requirement.parse('PyXMPP')
(xmpp)tim@prime:~/src/xmpp$ easy_install http://pyxmpp.jajcus.net/downloads/pyxmpp-1.0.0.tar.gz
...
Finished processing dependencies for pyxmpp==1.0.0

PyXMPP is in the PyPI index, but the download URL is out of date and the download URL is not automatically discovered from the page at http://pyxmpp.jajcus.net. The package installs without issues once the download URL is passed to the easy_install program, assuming you are not using Windows. Windows users will likely see something like “Python was build with Visual Studio version 7.1, and extensions need to be build with the same version of the compiler, but it isn’t installed.” And with that, we’ve exceeded the scope of this tutorial.

Before we go further, are you running OS X? I know you OS X users were laughing at the Windows guys in the last paragraph because their inferior system left them hanging. Sure, PyXMPP installed just fine on your Mac Book Pro, but open an iTerm session right now and try to run a PyXMPP application. You will probably see ImportError: No module named libxml2, and if you don’t, you’ve run into this problem before and fixed it yourself. Let this be a lesson, karma always finds a way. Google may hold the answer to this problem, however, I would guess that Cuil does not.

So now we have, what, maybe a handful of Linux users still chomping at the bit to try PyXMPP? Let’s get going. One more package to install, and we’re off to the races.

(xmpp)tim@prime:~/src/xmpp$ easy_install dnspython
...
Finished processing dependencies for dnspython

xmpppy

Hey, you Windows and OS X users, you can come back now. This next one is a piece of cake for everybody:

(xmpp)tim@prime:~/src/xmpp$ easy_install xmpppy
...
Installed /home/tim/pyenvs/xmpp/lib/python2.5/site-packages/xmpppy-0.4.1-py2.5.egg
Processing dependencies for xmpppy
Finished processing dependencies for xmpppy

The latest version of the xmpppy project installs automatically with easy_install.

Ease of Use

We are about to write some code, and this is where any head to head competition can turn subjective. For the sake of this comparison, I have written a small base class that does the following:

1. Create a chat bot
2. Send an arbitrary message to an arbitrary user on a supplied list every 10 seconds
3. Accept messages from users: print the message to the console, and thank the user.

Although the messages sent in this demonstration are fairly silly, the concept can easily be applied to real world applications. For instance, bug trackers could send notifications to developers, and developers could request status changes with the help of a chat bot.

We will create an implementation for each of the libraries under review. Before we begin, you may want to review the base class below. You can also check out the whole package from subversion.

jabber.py

Let’s dive in with jabber.py because it comes first alphabetically. That’s the kind of rigorous science we’re doing here, we alphabetize.

The JabberPyBot subclass proceeds in a pretty straightforward way that matches my experiences with writing Jabber client software in Java. We create a Client, connect, authenticate, and prepare to send and receive messages in the __init__ method.

I was able to send messages in very little time, but receiving messages was a little tricky. We registered a message handler when we created our Client object, and I thought that registering the handler would cause a glitter-laden jabber fairy to fly in on gossamer wings and sprinkle a little “we’ll handle these incoming messages automagically” dust over the code.

I ran the code a second time. Surely those dropped incoming messages were a fluke, but running unchanged code a second time produced exactly the same results. I was shocked. Shocked! Truthfully, I was just delaying the inevitable RTM moment. After reading through the example a second time, I saw the process method. That’s the ticket, when called it processes queued messages and fires off the appropriate handlers. We wire the process method into a receiver method that will run in its own thread, and we’re in business.

PyXMPP

The PyXMPP client was the last of the three clients written. So much for rigorous, alphabetized, science, eh? This code was a nice change of pace since the jabber.py and xmpppy clients were so similar, as you will soon find out.

The code for the PyXMPP implementation is just as short as our other two implementations, but something feels a little funny about it. Notice how we need to keep digging into our connection (JabberClient) object and grabbing its stream object to get stuff done? I don’t know about you, but that feels a little dirty. Matter of fact, there’s even a name for the “rule” that we are breaking. Take note, and during your next code review at the office when you see something similar you can say “I don’t know about that, Barry, the AbstractDatabaseManagerManagerFactory doesn’t adhere to the Law of Demeter very well, and I think it will be a maintenance nightmare down the road. Are you going to support this stuff when the API changes in the future?” But who are we kidding? Barry is such an old crank that it’s easier to fix his stuff after it’s been deployed than argue with him about his crazy design right now at 10:55. It is so close to lunch, and we can’t be late for that. Man, I’m hungry. A gyro sounds really good.

What were we talking about? Oh, right, our PyxmppBot and the code that talks to more than just its immediate friends. I feel like we’re giving the library a quick brush off just because it doesn’t adhere to the world view that we’ve adopted in the other two client examples. Let’s take another look at the PyXMPP example. We have formed the opinion that our JabberBot “has-a” connection/client object and interacts with it, but the PyXMPP programmers seem to think that a bot “is-a” client/connection object. The have been doing this stuff longer than we have, so let’s give it one more try.

That feels a little bit cleaner. If Demeter is happy, I’m happy.

xmpppy

The xmpppy client was shockingly similar to the jabber.py client, but the similarities were explained away upon reading the xmpppy home page where it mentions that some of the code and API decisions came directly from jabber.py. That just goes to show, when in doubt, read the docs.

One thing I didn’t quite understand about the xmpppy API was the choice of method names. Why are some methods given capitalized names, while others are lowercase? This is probably a silly complaint on my part, but I just know I will spend a lot of time second guessing the capitalization of method names if I use this library in my applications.

Reading the xtalk.py source provided most of the pointers that I needed to get our client up and running, with only one minor issue that stood in the way. The example calls a method named SendInitPresence which was since changed to sendInitPresence. Again, this made me wonder about the naming conventions in use.

Support and Growth Options

The example we worked through today was pretty basic. Where do these packages leave us as our skill and requirements grow?

jabber.py

Jabber.py is dead, dead, dead, and they are kind enough to tell us that right on the front page. If that’s the case, why did we look into it? Well, it works, and it isn’t licensed under the GPL. The working bit matters to everyone, and the GPL bit matters to some. If you have fairly simple needs and a list of requirements that includes “NO GPL!”, then this may just be the project for you.

PyXMPP

PyXMPP under active development, and it is a capable performer that seems to have a lot of power available for developers. For instance, as your bots grow, you can transfer your efforts into server side Jabber components, and the PyXMPP API will support your efforts. Of course, the fact that a large majority of developers will need to jump through a few hoops to install the library will be a source of frustration.

• API Docs
• Bug Tracker
• Home Page
• Mailing List

xmpppy

The xmpppy project’s last release was in December of 2007, and the mailing list remains active. It installs easily on Windows, OS X, and Linux, and it is a fairly easy to grasp library that could be quickly integrated into existing applications. The fact that it is GPL licensed could cause heartache for some projects or corporations, so proceed with caution if you are working in such a situation.

• API Docs
• Bug Tracker
• Home Page
• Mailing List

Conclusion

Sometimes the best way to pick a library is to sit down and write a little bit with each of the contenders. A little bit of code, and a little bit of reading goes a very long way toward a sound design decision. Sometimes these experiments lead us to question our initial assumptions. In this case, I am reconsidering Twisted to handle my XMPP needs since it is mature, under active development, and installs quickly on all of my potential target platforms.

What did I do right? What did I get wrong? What did I miss? Your comments are appreciated!

Submit to programming.reddit.com

Posted in Code, Python, Tutorials

Schema Migrations in TurboGears with Migrate

Posted by Tim Freund Mon, 06 Aug 2007 00:42:00 GMT

I learned about the importance of schema migrations the hard way. At my previous job, I helped a team upgrade a Java web application. The upgrade involved schema changes, and I had the forethought to script the upgrade and thoroughly test it on the development database. Even with that preparation, the night of the upgrade would teach me two important questions that new developers should always ask of their team. Does the development database schema match the production database schema? And do you know how to restore the database from backups should anything go wrong? The answers to both questions on that night were no and oh no. Sometime after midnight things started working.

A better way existed. I first learned of schema migrations a few months earlier when exposed to ActiveRecord::Migrations. After using them on several projects, I was itching to have the same capability in my Python and Java projects. The Pythonic answer came in the form of Migrate, a schema migration library for SQLAlchemy, and direct support for TurboGears was added with TGMigrate. Having migrate integrated with my projects greatly reduces my blood pressure on deployment days. Of course, I still make sure that my database backups are working.

Those interested in integrating Migrate with a TurboGears project might enjoy the screencast I just completed on the topic. If you use SQLAlchemy but avoid Migrate, I would be interested to hear what is holding you back.

• Add Migrate Support to TurboGears (QuickTime)
• Add Migrate Support to TurboGears (DivX)

Schema refactoring and migration was one of three topics at the last DotNext Kansas City Tech Coffee meeting. Notes on the schema migration talk were posted to Squidoo.

My apologies to those with small screens. I will record my next screencast in a smaller window. Your comments on today's screencast are appreciated.

Posted in Code, Python, TurboGears, Tutorials

Change Your Identity in TurboGears with Entry Points

Posted by Tim Freund Thu, 14 Jun 2007 03:15:00 GMT

Identity defines who we are. Identity is made up of all the little distinguishing traits that differentiate one person from another. We've all changed our identity throughout our lives. We change from student to graduate, single to married, dogless to dogged, and more, but that's not what we're talking about today. Identity is the authentication and authorization framework for TurboGears, and it is easy to extend.

At the core of the Identity framework is an IdentityProvider. The Identity Provider interfaces with an authentication and authorization repository to determine two things: are you who you say you are, and do you belong where you are trying to go. The framework comes with two providers, one each for SQLObject and SqlAlchemy.

We will customize an IdentityProvider to authenticate against an IMAP server in the few steps that follow. This would be helpful for writing a web mail application, and the concept can be applied to other authentication mechanisms as well, including LDAP, Radius, and others.

Action Plan

1. Quickstart a project
2. Create an identity provider
3. Define an entry point for the identity provider
4. Configure the application for the new provider
5. Finish the identity provider
6. Test
7. Relax

The code for this tutorial is available from subversion or as a tar.gz file. It is released under the MIT license. No TurboGears installation? Install it like so.

Step 1: Quickstart a Project

If you don't have an existing TurboGears project to experiment with, now would be a great time to start one.

tim@iris ~/src $ tg-admin quickstart -s -i iddemo
...
tim@iris ~/src/ $ cd iddemo
tim@iris ~/src/iddemo $ tg-admin sql create

Note the -s and -i flags. This is a project with support for SqlAlchemy and Identity.

Step 2: Create an Identity Provider

Any object can be an identity provider as long as it supplies the following methods: validate_identity, validate_password, load_identity, anonymous_identity, authenticated_identity, but it isn't always necessary to write one from scratch. Extending an existing provider often gets an application authenticating as required without much trouble. We will extend the SqlAlchemyIdentityProvider in this example to authenticate against an IMAP server.

from turbogears.identity.saprovider import SqlAlchemyIdentityProvider

class ImapSqlAlchemyIdentityProvider(SqlAlchemyIdentityProvider):
    pass

Step 3: Define an Entry Point

The Identity Framework uses an entry point named turbogears.identity.provider to decide what class to use when authenticating users. We are about to define a new option for this entry point, but further reading on the subject of entry points is recommended. Scroll to the bottom of this entry for a couple of relevant links. It's OK, we have the time.

This entry point step won't be necessary in the future, thanks to this patch, but entry points are a powerful tool and worth learning, regardless.

setup(
    name="iddemo",
... (more setup stuff) ...
    entry_points="""
    [turbogears.identity.provider]
    imapsa = iddemo.identity:ImapSqlAlchemyIdentityProvider
    """,
... (more setup stuff) ...
)
  

To let the setuptools system know about this new identity provider, run the following:

tim@iris ~/src/iddemo $ python setup.py develop

Step 4: Configure the Application

With our imapsa option defined for the turbogears.identity.provider entry point, we can now configure the application to call the new provider. There is a value named identity.provider in app.cfg. We will replace the existing value with imapsa. While app.cfg open is, add the other three lines in the following example. They will be explained in the next step.

...

identity.provider='imapsa'

identity.imapprovider.imap_authoritative=True
identity.imapprovider.server="localhost"
identity.imapprovider.port=143

...

  

The application is now ready to run with the new identity provider. Restart the application if it is currently running so that the configuration change will take effect.

Step 5: Finish the Identity Provider

Now let's dig in and implement the new authentication behavior.

...

class ImapSqlAlchemyIdentityProvider(SqlAlchemyIdentityProvider):
    def __init__(self):
        SqlAlchemyIdentityProvider.__init__(self)
        
        # These three lines get the configuration parameters we set in app.cfg
        self.imap_authoritative = get("identity.imapprovider.imap_authoritative", False)
        self.server = get("identity.imapprovider.server", "localhost")
        self.port = get("identity.imapprovider.port", 143)

        # These four lines make the user and visit classes available for
        # later use
        user_class_path = get("identity.saprovider.model.user", None)
        self.user_class = load_class(user_class_path)
        visit_class_path = get("identity.saprovider.model.visit", None)
        self.visit_class = load_class(visit_class_path)


    def validate_identity(self, user_name, password, visit_key):
        if self.validate_password(None, user_name, password):
            user = session.query(self.user_class).get_by(user_name=user_name)
            if not user:
                if self.imap_authoritative:
                    user = self.user_class()
                    user.user_name = user_name
                    user.save()
                    session.flush()
                else:
                    return None
            link = session.query(self.visit_class).get_by(visit_key=visit_key)
            if not link:
                link = self.visit_class(visit_key=visit_key, user_id=user.user_id)
                session.save(link)
            else:
                link.user_id = user.user_id
            session.flush()
            return SqlAlchemyIdentity(visit_key, user)
        return None

    def validate_password(self, user, user_name, password):
        rc = False
        try:
            imapcon = imaplib.IMAP4(self.server, self.port)
        except:
            log.error("Could not establish connection to server at %s:%d" % (self.server, self.port))
            return rc

        try:
            if imapcon.login(user_name, password)[0] == 'OK':
                rc = True
        except:
            # Probably threw an error for invalid username/password
            log.info("Passwords don't match for user: %s", user_name)
        imapcon.shutdown()
        return rc

Take a look at each method to figure out what they accomplish.

__init__ invokes the constructor of the SqlAlchemyIdentityProvider and then collects the configuration parameters required for authentication.

validate_identity first invokes validate_password. If the password validate succeeds, the user is selected from the database. If the user does not exist in the database, the provider will create the user when the imap_authoritative option is set. Should you not require this capability, you can remove this method entirely and rely upon the implementation in SqlAlchemyIdentityProvider. Finally, this method links the user with the current visit_key.

validate_password handles all IMAP access. It is the perfect method to override if all you need to do is change the identity authentication mechanism.

Step 6: Test

Open controllers.py and change the identity decorator to require an authenticated user when accessing the index page.

class Root(controllers.RootController):
    @expose(template="iddemo.templates.welcome")
    @identity.require(identity.not_anonymous())
    def index(self):
        import time
        flash("Your application is now running")
        return dict(now=time.ctime())

The moment of truth has arrived. Point a browser at the application. You should see a login page, and a valid IMAP username/password should provide access to the application.

Step 7: Relax

We're done here. Go tell your friends and family how cool you are.

Any feedback is appreciated. Leave a comment here, post a response on your own blog, or send an email to tim -at- achievewith -dot- us. You can also usually find me lurking in #turbogears on irc.freenode.net as timphnode.

For more information, follow these links:

• Mini tutorial on eggs, plug-ins, and entry points.
• The setuptools manual section concerning entry points.
• Plugins using Eggs.
• Identity documentation

Posted in Code, Python, TurboGears, Tutorials

Produce PDF Pages with TurboGears, Cheetah, and ReportLab

Posted by Tim Freund Thu, 22 Feb 2007 02:10:00 GMT

HTML is king of Web 2.0, and JavaScript is HTML's chief advisor and errand boy. But the printed page still matters, especially when working with Web 0.2 requirements and processes, and PDF documents provide crisp and consistent print output across platforms.

Follow along with the rest of this document to see how to produce PDF documents with the help of TurboGears, Cheetah, and ReportLab. We will generate form letters in response to a job opening, and in Part II we will print mailing labels for our letters.

The code for this tutorial is available from subversion or as a tar.gz file. It is released under the MIT license.

The Tools

• ReportLab is a Python library that creates professional PDF documents with minimal developer effort. It can be used in any Python application. The <br/> tag doesn't seem to work with the stable release, so try the snapshot instead.
• Cheetah is a flexible template engine that can generate arbitrary documents. It is supported by most Python web frameworks. TurboGears support is provided by the TurboCheetah package.
• TurboGears is a web framework that aims to make creating web apps faster, easier, and more fun. It brings together a wide range of powerful Python libraries into a coherent whole.

The Exercise

We will generate acceptance and rejection letters on behalf of Non Compos Mentis Research, a mysterious scientific research organization located somewhere underneath the the Midwestern United States. They were seeking applicants for an "Evil Genius, Sr." position after their long time employee, Dr. Phinius Fraggenblam, had an accident that lead to early "termination." All of the interviews are complete, the Board has made its decisions, and we have all of the applicant information we need in a model class named pdfdemo.model.Applicant.

This demonstration breaks into three logical steps. We need to

• Create a static PDF just to prove that we can
• Insert data into the PDF
• Serve the PDF to a web browser

Step 1: Generate a PDF from a Text File Using ReportLab

ReportLab allows programmers to specify every detail as they write code to generate PDF documents. For the [pragmatic | impatient | lazy] developer, ReportLab also provides an abstraction layer named PLATYPUS, short for "Page Layout And Typography Using Scripts." PLATYPUS provides a collection of "flowable" elements including paragraphs and page breaks that can be inserted into templated documents.

The Paragraph is a powerful Flowable element. In addition to accepting a style definition from reportlab.lib.styles, the Paragraph element also understands a small set of intra-paragraph XML markup.

Two brief examples follow:

<para>Dear Sir/Madam:</para>

<para>After careful consideration, we regret to inform you that your application
was <font face="times" color="red">rejected</font>.</para>

<para>Please try again next year.  <b>Or else</b>.</para>
<para>Best regards,</para>
<para>The Board<br/>
   Non Compos Mentis Research</para>
   

<para>Dear Sir/Madam:</para>

<para>Your application rose to the top of the millions and millions that we 
received.  We are pleased to extend to you a position on our dynamic team.</para>

<para>We will come for you in the night, unannounced.  Be prepared.</para>
<para><i>Congratulations!</i></para>

<para>The Board<br/>
   Non Compos Mentis Research</para>
   

Given the above, we can generate a static PDF with two pages: one acceptance letter, one rejection letter.

def generate_plain_document():
    """Generates a PDF with two pages, an accept and reject letter.
    
    Output is hardcoded to output.pdf in the current working directory
    """
    output_file = open("output.pdf", "w")
    document = SimpleDocTemplate(output_file)
    components = []
    
    sheets = styles.getSampleStyleSheet()
    # increase font size to 14 points
    sheets['Normal'].fontSize = 14
    # increase line height to 16 points
    sheets['Normal'].leading = 16
    # increase space after a paragraph to 16 points.
    sheets['Normal'].spaceAfter = 16
    paragraphStyle = sheets['Normal']
    
    for letter in (letters.accept_plain, letters.reject_plain):
        # There are multiple paragraph elements (para) defined in our 
        # documents, and each Paragraph object can only accept one.
        # Fortunately ElementTree is a requirement component of TurboGears,
        # and it makes short work of splitting up the document by para element.
        doc = ElementTree.XML(letter)
        for para in doc.findall('para'):
            components.append(Paragraph(ElementTree.tostring(para), paragraphStyle, None))
        components.append(PageBreak())
    
    document.build(components)
    output_file.close()

Step 2: Use a Cheetah Template to Insert Custom Data into the PDF

Now there is the small matter of inserting custom bits of data into the text before ReportLab does the voodoo that it does so well. Note that ReportLab is generating the PDF from a text string.

Also note that Cheetah can generate arbitrary text strings from templates and data with grace and aplomb. Since TurboGears is our target platform, TurboCheetah is installed, and that makes things even easier. Data available to a Cheetah template is passed as a dictionary into the TurboCheetah render method. Variables are rendered in Cheetah templates like so: ${my_variable.some_property}.

Below are examples of a Cheetah template, and a function invoking the TurboCheetah.render method.

<doc>
<para>${applicant.prefix} ${applicant.full_name()}<br/>
${applicant.address1}<br/>
#if $applicant.address2 != None
${applicant.address2}<br/>
#end if
${applicant.city}, ${applicant.state} ${applicant.postal}</para>

<para>Dear ${applicant.prefix} ${applicant.full_name()}:</para>

<para>Your application rose to the top of the millions and millions that we 
received.  We are pleased to extend to you a position on our dynamic team.</para>

<para>We will come for you in the night, unannounced.  Be prepared.</para>

<para><i>Congratulations!</i></para>

<para>The Board<br/>
Non Compos Mentis Research</para>
</doc>

def generate_lettersv1(applicants):
    tcheetah = TurboCheetah()
    pages = []
    # render the templates
    for person in applicants:
        letter_data = {"applicant": person}
        # convert status to a string because render doesn't
        # appreciate unicode template names
        template_name = "pdfdemo.letters.%s" % person.status.__str__().lower()
        rendered = tcheetah.render(letter_data, template=template_name)
        pages.append(rendered)
    
    paragraphStyle = get_stylesheets()['Normal']
    # assemble the PDF from the rendered templates
    components = []
    for page in pages:
        page = ElementTree.XML(page)
        for para in page.findall('para'):
            components.append(Paragraph(ElementTree.tostring(para), 
                                        paragraphStyle, 
                                        None))
        components.append(PageBreak())
    
    # generate the PDF
    output_file = open("output.pdf", "w")
    document = SimpleDocTemplate(output_file)
    document.build(components)
    output_file.close()

Step 3: Render the PDF to a Browswer with TurboGears

The hard work is done, but the code still generates a file on the file system, and we need to view the file in a browser. ReportLab can work with any object that acts like a file, including a StringIO Buffer. A new function, generate_lettersv2, accepts a file like object rather than opening a file itself. The controller then calls generate_lettersv2 with a StringIO object as the "file". This is a breeze.

    @expose(content_type="application/pdf")
    def letters(self, **kw):
        letters_file = StringIO()
        generate_lettersv2(Applicant.select(), letters_file)
        pdf = letters_file.getvalue()
        letters_file.close()
        return(pdf)

Some really old browsers require the URL of a PDF document to end with .pdf. This is accomplished by implementing a default method for the controller.

    @expose()
    def default(self, *path, **kw):
        method_name = path[0]
        if method_name.find('.') != -1:
            method_name = method_name.split('.')[0]
            method = getattr(self, method_name, None)
            if method != None:
                return(method(**kw))
        raise cherrypy.NotFound

Get Up and Stretch. Grab a Soda.

This article has detailed the steps required to produce templated PDF documents using TurboGears, Cheetah, and ReportLab. In the next installment we will generate mailing labels so that the letters we generated today are not lost due to poor handwriting skills. The Non Compos Mentis organization does not tolerate mistakes.

Any feedback is appreciated, both on the technical content and the silly back story. Leave a comment here, post a response on your own blog, or send an email to tim -at- achievewith -dot- us. You can also usually find me lurking in #turbogears on irc.freenode.net as timphnode.

For more information, follow these links:

• ReportLab
• Cheetah
• TurboGears

Posted in Code, Python, TurboGears, Tutorials

GOODCalc, Now Accepting Bad Data

Posted by Tim Freund Thu, 30 Nov 2006 07:32:07 GMT

The GOODCalc of 15 minutes ago didn't handle invalid input very well. This should be fixed now, in two ways:

• More variations in data are now accepted. For instance, $1,000.00 is now a valid monetary amount, and 3.4% is a valid APR. Previously only 1000.00 and 3.4 would have worked.
• Friendly error messages are now given when the program doesn't understand your input. Previously a programmer's stack trace was displayed. Those stack traces confuse people like my mom.

So what are you waiting for? Prove me wrong and find more bugs in GOODCalc, but only if you promise to tell me about your experience here either with a comment or a trackback. :-)

Posted in Code

System.out deemed Unreliable

Posted by Tim Freund Wed, 15 Nov 2006 04:47:00 GMT

Actually, System.out fulfills its purpose perfectly. Data goes in through print methods, and goes out to the console every single time. Before you complain about the misleading title, let me finish. Although great at printing, System.out is a completely unreliable test framework.

Who in their right mind uses System.out as a test framework? At no place in the javadocs does it mention code testing as an alternative use of System.out. How about a quick show of hands. How many people have written code like this:

System.out.println("widget.name == " + session.getWidget().getName());

public class CogAdapter{
  /*
   * other code goes up here...
   */ 

  public static void main(String[] args){
    // the intuitive naming scheme in the catalog is awesome... ugh.
    Cog c = CogswellCogFactory.getCog(CogswellCatalog.X43RNT);
    Sprocket s = SpacelySprocketFactory.getSprocket();

    try{
      c.attach(s);
      System.out.println("cog.attach(sprocket) worked");
      System.out.println("cog.attachedObjects.length == " + 
                          c.getAttachedObjects().size());
    } catch(RatioException re){
      System.out.println("cog & sprocket ratios incompatible");
      System.out.println("cog.ratio == " + c.getRatio());
      System.out.println("sprocket.ratio == " + s.getRatio());
    } catch(JetsonException je){
      System.out.println("manufacturing defect in sprocket: " + je.getMessage());
    }
  }
}

Right, right, cogs don’t really attach directly to sprockets. That’s why this example is in the CogAdapter class. Duh. I didn’t want to pass up a fun Jetsons reference just because sprockets and cogs are not directly interchangeable. The object model of Fred Flintstone’s rock quarry just wasn’t a good fit for use in this example.

At this point System.out has evolved into the cornerstone of a miniature testing framework. Not a good testing framework, but one none the less. This output driven test framework requires labor intensive manual intervention for the tests to have any value. A person must manually run the tests and then manually verify the printed results with the intent of the code.

When I would write tests like this, I would get the test to work, satisfy my original curiosity, and then the test would never run again. At least not until the vendor upgraded their library and changed behavior. Even then, I had to actually remember that I wrote such a test when first working on the project before that test would again be called into service.

I saw the answer to these problems for years before I acted to remedy the situation. These little System.out driven test programs were perfect candidates to become JUnit tests, but I didn’t make that connection. I had only ever heard the “Test First” mantra, and I was obviously already knee deep in source code, so I thought that unit tests weren’t for me. Watching two of Mike Clark’s presentations at a NFJS conference in 2004 got me excited about integrating unit tests into my existing code.

How does JUnit improve upon our System.out test framework?

• JUnit tests can run one at a time, or all at once
• Every major Java IDE understands how to run JUnit
• Every major build system supports running JUnit tests, either through built in plugins or trivial customization.
• The mind numbing task of verifying results is left up to the computer.

Let’s transform the Sprocket/Cog test above into a real JUnit test.

public class CogAdapterTest extends TestCase {
  public void testAttach(){
    Cog c = CogswellCogFactory.getCog(CogswellCatalog.X43RNT);
    Sprocket s = SpacelySprocketFactory.getSprocket();

    try {
      c.attach(s);
      assertNotNull(c.getAttachedObjects());
      assertEquals(1, c.getAttachedObjects().length());
    } catch(RatioException re){
      assertEquals(c.getRatio(), s.getRatio());
    } catch(JetsonException je){
      fail("Incompetent Manufacturer");
    }
  }
}

No System.out references in sight. Now the computer will only produce output if something goes wrong, and once JUnit is scheduled to run as a part of the build process, your project will never go untested again.

System.out is great at printing data, but it makes a lousy test framework.

Ready to learn more? Check out these resources:

JUnit

Pragmatic Unit Testing in Java with JUnit or in C# with NUnit.

A Dozen Ways to Get the Testing Bug in the New Year by Mike Clark.

Posted in Code, Tutorials | no trackbacks

Graphs and Charts for GOODCalc with SparkliPy

Posted by Tim Freund Mon, 12 Jun 2006 05:43:00 GMT

I haven't worked much on GOODCalc lately, but one of the things that I'd really like to add is a collection of graphs and charts that will help people visualize their debt repayment process.  A sparkline is a type of inline graph that is perfect for this type of application, and there is even an existing library available for Python.  I found it through inspection of the source code to the Ruby sparkline library. Joe's library is nice, but I want better integration with TurboGears than a CGI program will provide, so with a spare moment here and there I have started to piece together a new sparkline library called SparkliPy.  I hope to get it working well, documented well, and then well integrated with GOODCalc very soon, but you can dig in now if you like playing with brand new stuff.

Go to the SparkliPy Project Page

Posted in Code, Open Source

Prototype for the Proto-programmer

Posted by Tim Freund Tue, 06 Jun 2006 04:05:00 GMT

So you’re a web designer afraid of a little code?  Or maybe you really are a developer, but there are some static pages that need a little dynamic boost.  The Prototype JavaScript library enables dynamic behavior in pages created by those of you who don’t think you can code as well as those of you who aren’t allowed to code.  Perhaps a case study is in order:

Zack runs a website for a shelter that finds homes for older dogs dogs.  Dogs whose owners are "moving", or who ran away from home, and even the seldom seen blind seeing eye dog.  Don’t worry, the blind seeing eye dogs are obviously retired.  It is tough to place these animals in a caring home since so many people would prefer to spend hundreds of dollars on a pedigreed animal, complete with a lifetime of bad hips and an increased likelihood of tumors, but Zack and the crew at Red Rover Rescue carry on.  At least once a week they take an assortment of animals out to local events and pet stores, hoping that some kind soul will take Miss Growlypants home. It is very helpful to have a calendar of events on their home page so that volunteers and potential adopters alike know when and where the dogs will be out.

Zack knows HTML, CSS, and a little bit of JavaScript.  He has access to pre-installed calendar programs, but he doesn’t know how to marry his dynamic calendar with his oh so static shelter web page.

Can we help Zack?  Of course we can, otherwise this would be the lamest tutorial in the world.  There is a popular JavaScript library called Prototype.  Prototype wraps up a lot of inconsistencies among different web browsers and makes accomplishing some rather dynamic stuff with JavaScript quite simple.  Among it’s many tricks, Prototype can load files and call programs on the server into the current page.  That’s just what Zack needs.  Zack does the following to make things work:

1. Downloads the Prototype library and puts the prototype.js file in his javascripts directory
   
2. Inserts the following line into the HEAD section of his HTML file to load the library:
   

<script src="./javascripts/prototype.js" type="text/javascript"></script>

The Prototype is nice, but it isn’t quite magic.  Zach still needs to write a little bit of JavaScript to glue everything together.  He wants the calendar loaded when the page loads, so he writes the following function:

// the first argument is the name of the div where the results of the remote call will end up
// the second argument is the url to retrieve
// we'll talk about the third argument in a later lesson
function loadFirstTarget(){
  new Ajax.Updater('firsttarget', './lesson01-calendar.html', {asyhnchronous:true, method:"GET"});
}

Noting that the Ajax.Updater will look for a div with the id ‘firsttarget’, Zack adds one to his page where he wants the calendar to appear:

<div class="target" id="firsttarget"></div>

Zack then changes the body element of his page to have an onload attribute:

<body onload="loadFirstTarget();">

There’s one other thing—Zack wants people to be able to click a button to see the dog of the month.  He adds another function to the page:

function loadSecondTarget(){
  new Ajax.Updater('secondtarget', '/lesson01-dotm.html', {asyncrhronous:true, method:"GET"});
}

And then he creates a form with a single button that fires loadSecondTarget() as well as the ‘secondtarget’ div:

<form>
  <input type="button" value="See the Dog of the Month!" onclick="loadSecondTarget();">
</form>

<div class="target" id="secondtarget"></div>

At this point, you probably want to see the whole thing in action. 

To recap, let’s put you in Zack’s shoes.  If the following fits your situation then you just learned a new trick that you can apply to your site today:

1. You have a pretty static website.
2. You have some dynamic web applications like a calendar or some forums.
   
3. You’d like to take some of the content from the dynamic bits and put them in the static bits, but you don’t want to turn the static bits dynamic with a bunch of programming.
4. JavaScript doesn’t scare you.

If that’s the case, you should roll up your sleeves and implement the technique outlined in this tutorial, if you haven’t done so already. 

So what do you think?  Leave a comment to let me know how it works out for you.  You can also contact me directly if you’d like to talk about work that needs to be done on your own site that you’d rather not do yourself.

Posted in Code, Tutorials

Passion + Power, the Proof is in the Product

Posted by Tim Freund Sat, 13 May 2006 15:38:00 GMT

I wrote most of an application last night.  Sure, it’s a simple application with only three database objects and lots of javascript.  It isn’t earth shattering in any way, but it is full of potential for the people who would use the application, and it is full of new technologies and sheer delight for me, the programmer.

From 11:00 PM until 3:30 AM last night I was fueled by passion and power.  The passion flowed from knowing that I was solving a problem voiced by an associate at work, and the power flowed from my weapons of choice, TurboGears and Prototype.  When powerful tools allow programmers to progress by leaps and bounds on interesting problems it becomes quite hard to get us to step away from the computer.  I love that feeling—that’s why I got into this racket in the first place!

People have questioned my obsession with open source and with the newest of open source at that.  The proof is in product.  Going back in time, had I started writing in Java last night, I bet I would have been asleep by 12:30.  I would have stumbled across some problem that I tangled with many times before, each time wondering why I encounter it in the first place.  With my flow disrupted, I would have stumbled lazily down to bed.

More will be published about this mystery application soon—for now I must get back to work on it to see what it becomes.

Posted in Code, Open Source

Typo Themes

Posted by Tim Freund Tue, 02 May 2006 13:02:00 GMT

I just posted screen shots and updated components and milestones in the issue tracker for the four Typo themes that I created.  More exciting to any Typo designers using Apple systems, I uploaded an Automator workflow that I wrote to streamline creating preview images for themes.  I will be very happy to receive comments on any of the above work.

At this point all four themes have preview images and are available for download as gzipped tar file or from the subversion repository. 

Please don't let the invalid SSL cert scare you off from the subversion repository -- we're getting that worked out.

Posted in Code