Posted by Tim Freund
Thu, 19 Feb 2009 03:34:00 GMT
The TurboJson package used in TurboGears can automatically serialize
most SQLAlchemy model objects. Hints can be provided for those objects
that TurboJson (and simplejson) cannot serialize on their own. Provide
a method named __json__(self) in a class definition to override the
default serialization algorithm:
class Invoice(object):
def __json__(self):
return {'id': self.id, 'date': self.date, 'line_items': self.line_items}
This is an especially handy technique if object X has a reference to
object Y and object Y has a reference to object X. The default
serialization algorithm will produce the following message: ValueError:
Circular reference detected.
This is something that I learned, and then forgot, and then spent 15
minutes searching for last night. I finally found the answer by
looking in the TurboJson test suite. This post serves as a reminder
to myself, and hopefully others.
Posted in Python
Posted by Tim Freund
Mon, 02 Feb 2009 00:46:00 GMT
Last time we established that recording audio with Python and
GStreamer is a fairly straightforward process. The next task for my
project was to find the peak recording level, so I turned to the
vumeter.py example to see how to get that information from a live
stream. This program is provided in the examples directory of the
gst-python package. You can download it here.
But it didn't work. The program produced an error message, and that was
all:
So how do we debug problems like this? A very cool feature of the
GStreamer Python bindings is support for debug output configuration on
the command line. Provide an integer from 0 to 5 to the
--gst-debug-level option to get debug output in real time while
the application runs. But don't ask for level 5 if you know what's
good for your terminal -- you will be overwhelmed with debug output.
Level 3 was very helpful in sorting out my problem. There was no recording
channel on the default device. The default device was a USB audio device
that had since been removed, so there really wasn't a default device
at all. The following output pointed me in the right direction:
0:00:00.302811092 27033 0x1039f30 WARN alsa confmisc.c:768:parse_card: alsalib error: cannot find card '0'
0:00:00.302949192 27033 0x1039f30 WARN alsa conf.c:3513:_snd_config_evaluate: alsalib error: function snd_func_card_driver returned error: No such file or directory
0:00:00.303018433 27033 0x1039f30 WARN alsa confmisc.c:392:snd_func_concat: alsalib error: error evaluating strings
0:00:00.303088567 27033 0x1039f30 WARN alsa conf.c:3513:_snd_config_evaluate: alsalib error: function snd_func_concat returned error: No such file or directory
0:00:00.303156031 27033 0x1039f30 WARN alsa confmisc.c:1251:snd_func_refer: alsalib error: error evaluating name
0:00:00.303225321 27033 0x1039f30 WARN alsa conf.c:3513:_snd_config_evaluate: alsalib error: function snd_func_refer returned error: No such file or directory
0:00:00.303295320 27033 0x1039f30 WARN alsa conf.c:3985:snd_config_expand: alsalib error: Evaluate error: No such file or directory
0:00:00.303364659 27033 0x1039f30 WARN alsa pcm.c:2196:snd_pcm_open_noupdate: alsalib error: Unknown PCM default
0:00:00.316686427 27033 0x1039f30 WARN alsa gstalsasrc.c:635:gst_alsasrc_open:<alsasrc0> error: Could not open audio device for recording.
0:00:00.316703039 27033 0x1039f30 WARN alsa gstalsasrc.c:635:gst_alsasrc_open:<alsasrc0> error: Recording open error on device 'default': No such file or directory
0:00:00.316743785 27033 0x1039f30 INFO GST_ERROR_SYSTEM gstelement.c:1675:gst_element_message_full:<alsasrc0> posting message: Could not open audio device for recording.
0:00:00.316788221 27033 0x1039f30 INFO GST_ERROR_SYSTEM gstelement.c:1698:gst_element_message_full:<alsasrc0> posted error message: Could not open audio device for recording.
For the record, I was able to get this output (and a lot more) with the following
invocation:
python vumeter.py --gst-debug-level=3
The vumeter.py program started to work once the GStreamer pipeline
string was changed to specific an Alsa device:
s = 'alsasrc device=hw:1 ! level message=true ! fakesink'
We covered how to get an Alsa device ID in the last installment. Check it out
to remember how that was done.
Posted in Python
Posted by Tim Freund
Thu, 29 Jan 2009 03:45:00 GMT
I have a project idea that involves recording audio from USB powered
microphones, and I wanted to create a quick sketch of a program to
judge the difficulty of doing that with Python. Short answer? It's
really easy. If you're impatient, you can download this sample
script.
For those of you still reading, I'll explain the process in a little
more detail. The GStreamer library handles all sorts of media input and
output, and there are bindings written for Python. This seemed like an
excellent avenue for my project, but I had some trouble getting started.
I discovered Jokosher after a bit of searching. It is a multi-track
recording application written with Python and GStreamer. I was able
to use the debug output of Jokosher to figure out what an appropriate
GStreamer pipeline would look like to capture audio from my microphone,
and I was able to see how D-Bus can be used to determine the ALSA card
identifier for any given sound device.
This web cam tutorial was also helpful, as it demonstrated how to properly
close a GStreamer pipeline.
Of course, after my proof of concept was done I stumbled across this
documentation for GStreamer's Python bindings.
I will excerpt the interesting bits of the script here, but don't forget
that you can download the entire thing.
List Sound Cards and Get Their ALSA ID
def list_capture_devices():
bus = dbus.SystemBus()
hal_manager = bus.get_object("org.freedesktop.Hal", "/org/freedesktop/Hal/Manager")
hal_manager = dbus.Interface(hal_manager, "org.freedesktop.Hal.Manager")
devices = hal_manager.FindDeviceStringMatch("alsa.type", "capture")
identifiers = []
for dev in devices:
device = bus.get_object("org.freedesktop.Hal", dev)
card = device.GetAllProperties(dbus_interface="org.freedesktop.Hal.Device")
if card["alsa.card"] not in identifiers:
print "%d. %s" % (card["alsa.card"], card["alsa.card_id"])
identifiers.append(card["alsa.card"])
return identifiers
Start (and Stop) a GStreamer Pipeline
def record(device_id, capture_path):
pipeline = gst.parse_launch("""alsasrc device=hw:%d ! audioconvert ! level name=recordlevel interval=10000000 ! audioconvert ! flacenc ! filesink location=%s""" % (device_id, capture_path))
pipeline.set_state(gst.STATE_PLAYING)
print "recording, press enter to stop"
sys.stdin.readline()
pipeline.set_state(gst.STATE_NULL)
time.sleep(5)
With as easy as this seems, maybe we really will import antigravity one day.
Posted in Open Source, Python
Posted by Tim Freund
Tue, 13 Jan 2009 05:47:00 GMT
There's nothing more relaxing than to sit down and write an
implementation of a commonly available network service in your
favorite scripting language. OK, so maybe writing an SFTP server in
Python wasn't the most relaxing thing I could have done this evening,
but the Paramiko library made it pretty darn easy, and I had a lot of
fun because the work moved so quickly.
Did I mention how quickly the development went? 284 lines, including
the license and a demo configuration file. And people wonder why
I enjoy Python programming so much.
But Why?
Most code is written to scratch an itch, and this code was no
different. We write a number of scheduled jobs at work that upload or
download files from SFTP servers. We needed a way to test the jobs,
but we didn't want to set up operating system accounts on our servers
to run the tests.
My minimal implementation of an SFTP service allows us to test many
job configurations with one SFTP service running as an unprivileged
user. The user accounts are optionally chroot'ed into their "home"
directory, so it appears as if they are connecting to a server that
has the exact same file hierarchy that is present on the production
file servers.
To play around with the code yourself, you can install Paramiko and
clone the Simple SFTP Server Mercurial repository. If you don't
have Mercurial installed, you can always download the script straight
out of the repository.
The code works, but it is minimally functional: it only does what we
needed it to do at work. No worries, though. It is licensed under
the MIT license, so you are free to do with it what you please. Patches
are accepted if you are feeling generous.
Posted in Open Source, Python
Posted by Tim Freund
Tue, 21 Oct 2008 03:35:00 GMT
Your TurboGears 2 application is not perfect. Just for a second,
let's pretend like it is perfect. Even with all of its perfection,
your application will need to deal with bad incoming links and
malformed data. Is it ready? It is only a matter of time before your
users receive a 404 or 500 response from your most wonderful
application. Whether the source of trouble is a bug in the code or a
bad incoming link, why leave users lost in the dark? Custom error
pages can put them back on track when something goes wrong, and they
are very easy to implement. We will create one in the following few
paragraphs.
We will use Turtle Goals to demonstrate the techniques in this
tutorial. It is open source and fairly simple. Please feel free to
follow along in the Turtle Goals source, or work along in your
own TurboGears 2 project.
Look in your controllers directory. See that file named error.py?
That's the key to a custom error page. The Routes package does a
bit of work behind the scenes to send any request that generates an
error through the ErrorController, so by customizing the
ErrorController, we can customize the resulting 404 and 500 pages.
The default document method produces a standard Pylons error page.
It looks nice, but it probably doesn't look right compared to the rest
of your project.
TurboGears projects default to the Genshi template engine, and
that is the engine used by Turtle Goals. Let's create a new template,
error.html, and save it in the templates directory.
So now all we need to do is add the expose decorator to the
document method, and return a dictionary of appropriate values.
Done, with enough time to check Reddit before your next meeting,
right? Well, almost. Look closely at the ErrorController
definition, and you will see that it is not a standard TurboGears
controller. It extends a Pylons controller class, WSGIController,
and that causes it to behave differently from our other controllers.
At least it should extend WSGIController according to a post on the
mailing list. Apparently there is a bug in the quickstart
template, and you you may need to chage the ErrorController definition
to extend WSGIController. I was easy to convince: as soon as I made
the suggested change, my error page started working. Back to the
point: the expose decorator will do you no good inside of the
WSGIController. It is up to us to render the template to a string and
return that string. Fortunately TurboGears provides a method to do
just that:
from tg import render
rendered_template = render.render_genshi("error.html", {})
Of course there are also methods like render.render_mako and
render.render_jinja if you prefer those other template engines.
Here's the full listing of our modified error.py:
There's one other small matter to deal with: configuration. There are
two relevant configuration values in your application: debug and
full_stack. To use your custom 404 error but still get the
interactive debug page for 500 errors, try the following:
debug = true
full_stack = true
The interactive debug page is inappropriate for production
environments. When your application is deployed into production, use
these settings instead:
debug = false
full_stack = true
If you have both set to false, you will get generic error pages that
end users will run from, screaming. All done, for real this time. And
you still have time to check Reddit, but you may want to check out
these links instead:
What is your strategy for designing and implementing custom 404 and 500
error pages?
Posted in Python, TurboGears, Tutorials
Posted by Tim Freund
Wed, 24 Sep 2008 06:06:00 GMT
My mom always reminded me to cover my nose when I sneezed. Now I take
it a step further, and I cover my nose when I test.
Nose and the nosetest command are used to run unit tests for
Pylons and TurboGears 2 applications, as well as a multitude of other
Python applications and libraries. Although nose is great at running
tests and reporting back issues, it doesn't natively show developers
what isn't being tested. For that, we need a code coverage tool.
Just because nose doesn't handle code coverage reports natively
doesn't mean this will be a difficult task. Ned Batchelder's coverage package provides exactly those reports, and nose ships
with a plugin to enable it. To install coverage and invoke the
reports, you could do something like this:
$ easy_install coverage
$ nosetests --with-coverage
Name Stmts Exec Cover Missing
--------------------------------------------------------------------------
_strptime 228 149 65% 23, 80, 84-89, 155, 169-170, 175, 189, 237, 280-294, 303-304, 306, 314-323, 329, 332, 353-360, 366, 368, 374-388, 393-427, 431-432, 443-446
encodings.ascii 19 0 0% 9-42
ez_setup 103 11 10% 53-62, 80-104, 117-151, 156-190, 197-222, 226-229
fixture 10 9 90% 38
fixture.base 124 33 26% 10-19, 25-28, 48-49, 56, 60, 64, 98, 103-104, 122-217
fixture.dataset 301 225 74% 41, 51-52, 55-59, 65, 76-77, 80, 98-110, 141, 144, 225-229, 232-239, 243-248, 277, 285-289, 323-325, 447-450, 456-457, 461, 468, 479, 483, 529-530, 560-564, 571-574, 580, 631, 720-722, 725-737, 740-744, 752-753
... lots of lines cut to keep you from going blind ...
zope.interface.ro 22 22 100%
zope.interface.verify 45 29 64% 46, 51, 56-61, 66, 70, 75-79, 88, 93, 104, 107, 109, 111
--------------------------------------------------------------------------
TOTAL 17257 7646 44%
----------------------------------------------------------------------
Good Gravy, man! That's a big coverage report!?
As cool as it is to know that we can get coverage reports for the
entire Pylons/TG2 stack, how about we focus for just a minute on only
our project. That's the problem with kids today, no focus.
$ nosetests --with-coverage --cover-package=YOUR_PACKAGE_NAME
OK, that's better. Now the report only shows the coverage for code
inside of our package. But, seriously, do we really need to type those
arguments every time?
Of course not. Sitting there in the root of your Pylons or TurboGears
2 application is a file named setup.cfg. If I were a betting man, I'd
say you've never opened it. Ever. Let's knock the dust off and take
a look at it in any decent text editor. We're looking for a section
named [nosetests], and since you've never changed the file, it is
probably at line 8 and looks just like this:
[nosetests]
with-pylons=test.ini
We can add any additional options for nose in this section. Now is
your chance to spring into action. Add the following lines to
with-coverage=true
cover-package=YOUR_PACKAGE_NAME
Save setup.cfg and run your tests. You will see a code coverage
report at the end of the test run, and the last two columns will be
the most interesting. They show the percentage of the code that was
covered and the lines of code that were not covered, respectively.
Depending on the outcome of your coverage report, you may be feeling
rather smug right now. Stop it, we're not Rails developers, and there
is still work to do.
On the other hand, if your code coverage report leaves you feeling a
little ashamed of just how much code is uncovered, don't despair.
Just knowing what code coverage is and caring about the results of a
coverage report already puts you into a minority of all programmers.
Pick a block of untested code and write a test. All of the sudden,
your numbers are better and you'll start to leave those
fly-by-the-seat-of-their-pants programmers in the dust.
Remember that this setup.cfg trick will work with any python
application or library that is testable with nose. We happen to be
focusing on Pylons and TurboGears because that's what I've been busy
using lately. How do you use nose to help you write better code?
Posted in Python, TurboGears, Tutorials
Posted by Tim Freund
Tue, 12 Aug 2008 03:42:00 GMT
Python is known as a batteries included language, but sometimes it can
feel like a home improvement project gone bad. You know, the kind of
project where there seems to be too many parts for the work that
remains? I had that feeling as I tried to figure out which
Jabber/XMPP library would suit my application best. Like any
programmer worth their salt, my first tool for research is a visit to
Google. Two searches, python jabber and python xmpp, seemed
to turn up three viable projects.
There’s also a reference to Twisted’s support for XMPP in the first
page of search results, but we’ll save Twisted for another day. I
haven’t grasped Twisted enough to feel comfortable writing about it,
and the application that I am working on doesn’t use Twisted in any
way. Keep an eye on these guys for more information regarding
Twisted’s XMPP support.
All three of the options seem like reasonable choices, so we will
review each of the packages for the following criteria:
-
Installation
-
Ease of Use
-
Support Options
Installation
The trial begins with an empty virtualenv environment. We have a
blank slate into which we can install the projects. We will first
attempt easy_install $PROJECT_NAME, and then easy_install
$DOWNLOAD_URL. Any library that won’t easily install with
easy_install is going to be a source of grief form any team or project
that depends on it.
jabber.py
(xmpp)tim@prime:~/src/xmpp$ easy_install jabber.py
...
error: Could not find suitable distribution for Requirement.parse('jabber.py')
(xmpp)tim@prime:~/src/xmpp$ easy_install "http://downloads.sourceforge.net/jabberpy/jabberpy-0.5-0.tar.gz?modtime=1075826815&big_mirror=0"
...
Finished processing dependencies for jabber.py==0.3-1
It appears that jabber.py isn’t in the PyPI index at all, and when we
attempt to download and install jabber.py version 0.5-0, we are told
that we have just installed version 0.3-1. I downloaded
jabberpy-0.5-0.tar.gz and confirmed that setup.py reports the
distribution as version 0.3-1. One other slightly confusing issue:
the README file reports jabber.py as a GPL project, but the home page
and setup.py files report it as LGPL. The xmpppy project also refers
to jabber.py as LGPL. Best 3 out of 4 licensing?
The good news: jabber.py installation is painless on Windows, OS X,
and Linux, as long as a download URL is provided.
PyXMPP
(xmpp)tim@prime:~/src/xmpp$ easy_install PyXMPP
...
Reading http://pypi.python.org/simple/
Reading http://pypi.python.org/simple/pyxmpp/
Reading http://pyxmpp.jabberstudio.org/
No local packages or download links found for PyXMPP
error: Could not find suitable distribution for Requirement.parse('PyXMPP')
(xmpp)tim@prime:~/src/xmpp$ easy_install http://pyxmpp.jajcus.net/downloads/pyxmpp-1.0.0.tar.gz
...
Finished processing dependencies for pyxmpp==1.0.0
PyXMPP is in the PyPI index, but the download URL is out of date and
the download URL is not automatically discovered from the page at
http://pyxmpp.jajcus.net. The package installs without issues once
the download URL is passed to the easy_install program, assuming you
are not using Windows. Windows users will likely see something like
“Python was build with Visual Studio version 7.1, and extensions need
to be build with the same version of the compiler, but it isn’t
installed.” And with that, we’ve exceeded the scope of this tutorial.
Before we go further, are you running OS X? I know you OS X users
were laughing at the Windows guys in the last paragraph because their
inferior system left them hanging. Sure, PyXMPP installed just fine on
your Mac Book Pro, but open an iTerm session right now and try to run
a PyXMPP application. You will probably see ImportError: No module
named libxml2, and if you don’t, you’ve run into this problem before
and fixed it yourself. Let this be a lesson, karma always finds a
way. Google may hold the answer to this problem, however, I would
guess that Cuil does not.
So now we have, what, maybe a handful of Linux users still chomping at
the bit to try PyXMPP? Let’s get going. One more package to install,
and we’re off to the races.
(xmpp)tim@prime:~/src/xmpp$ easy_install dnspython
...
Finished processing dependencies for dnspython
xmpppy
Hey, you Windows and OS X users, you can come back now. This next one
is a piece of cake for everybody:
(xmpp)tim@prime:~/src/xmpp$ easy_install xmpppy
...
Installed /home/tim/pyenvs/xmpp/lib/python2.5/site-packages/xmpppy-0.4.1-py2.5.egg
Processing dependencies for xmpppy
Finished processing dependencies for xmpppy
The latest version of the xmpppy project installs automatically with easy_install.
Ease of Use
We are about to write some code, and this is where any head to head
competition can turn subjective. For the sake of this comparison, I
have written a small base class that does the following:
-
Create a chat bot
-
Send an arbitrary message to an arbitrary user on a supplied list every 10 seconds
-
Accept messages from users: print the message to the console, and thank the user.
Although the messages sent in this demonstration are fairly silly, the
concept can easily be applied to real world applications. For
instance, bug trackers could send notifications to developers, and
developers could request status changes with the help of a chat bot.
We will create an implementation for each of the libraries under
review. Before we begin, you may want to review the base class below. You
can also check out the whole package from subversion.
jabber.py
Let’s dive in with jabber.py because it comes first alphabetically.
That’s the kind of rigorous science we’re doing here, we alphabetize.
The JabberPyBot subclass proceeds in a pretty straightforward way that
matches my experiences with writing Jabber client software in Java.
We create a Client, connect, authenticate, and prepare to send and
receive messages in the __init__ method.
I was able to send messages in very little time, but
receiving messages was a little tricky. We registered a message
handler when we created our Client object, and I thought that
registering the handler would cause a glitter-laden jabber fairy to
fly in on gossamer wings and sprinkle a little “we’ll handle these
incoming messages automagically” dust over the code.
I ran the code a second time. Surely those dropped incoming messages
were a fluke, but running unchanged code a second time produced
exactly the same results. I was shocked. Shocked! Truthfully, I was
just delaying the inevitable RTM moment. After reading through the
example a second time, I saw the process method. That’s the
ticket, when called it processes queued messages and fires off the
appropriate handlers. We wire the process method into a receiver
method that will run in its own thread, and we’re in business.
PyXMPP
The PyXMPP client was the last of the three clients written. So much
for rigorous, alphabetized, science, eh? This code was a nice change
of pace since the jabber.py and xmpppy clients were so similar, as you
will soon find out.
The code for the PyXMPP implementation is just as short as our other
two implementations, but something feels a little funny about it.
Notice how we need to keep digging into our connection
(JabberClient) object and grabbing its stream object to get stuff
done? I don’t know about you, but that feels a little dirty. Matter
of fact, there’s even a name for the “rule” that we are breaking.
Take note, and during your next code review at the office when you see
something similar you can say “I don’t know about that, Barry, the
AbstractDatabaseManagerManagerFactory doesn’t adhere to the Law of
Demeter very well, and I think it will be a maintenance nightmare
down the road. Are you going to support this stuff when the API
changes in the future?” But who are we kidding? Barry is such an old
crank that it’s easier to fix his stuff after it’s been deployed than
argue with him about his crazy design right now at 10:55. It is so
close to lunch, and we can’t be late for that. Man, I’m hungry. A
gyro sounds really good.
What were we talking about? Oh, right, our PyxmppBot and the code
that talks to more than just its immediate friends. I feel like we’re
giving the library a quick brush off just because it doesn’t adhere to
the world view that we’ve adopted in the other two client examples.
Let’s take another look at the PyXMPP example. We have formed
the opinion that our JabberBot “has-a” connection/client
object and interacts with it, but the PyXMPP programmers seem to think
that a bot “is-a” client/connection object. The have been doing
this stuff longer than we have, so let’s give it one more try.
That feels a little bit cleaner. If Demeter is happy, I’m happy.
xmpppy
The xmpppy client was shockingly similar to the jabber.py client, but
the similarities were explained away upon reading the xmpppy home page
where it mentions that some of the code and API decisions came
directly from jabber.py. That just goes to show, when in doubt, read
the docs.
One thing I didn’t quite understand about the xmpppy API was the
choice of method names. Why are some methods given capitalized names,
while others are lowercase? This is probably a silly complaint on my
part, but I just know I will spend a lot of time second guessing the
capitalization of method names if I use this library in my
applications.
Reading the xtalk.py source provided most of the pointers that I
needed to get our client up and running, with only one minor issue
that stood in the way. The example calls a method named
SendInitPresence which was since changed to sendInitPresence. Again,
this made me wonder about the naming conventions in use.
Support and Growth Options
The example we worked through today was pretty basic. Where do these
packages leave us as our skill and requirements grow?
jabber.py
Jabber.py is dead, dead, dead, and they are kind enough to tell us
that right on the front page. If that’s the case, why did we look
into it? Well, it works, and it isn’t licensed under the GPL. The
working bit matters to everyone, and the GPL bit matters to some. If
you have fairly simple needs and a list of requirements that includes
“NO GPL!”, then this may just be the project for you.
PyXMPP
PyXMPP under active development, and it is a capable performer that
seems to have a lot of power available for developers. For instance,
as your bots grow, you can transfer your efforts into server side
Jabber components, and the PyXMPP API will support your efforts. Of
course, the fact that a large majority of developers will need to jump
through a few hoops to install the library will be a source of
frustration.
xmpppy
The xmpppy project’s last release was in December of 2007, and the
mailing list remains active. It installs easily on Windows, OS X, and
Linux, and it is a fairly easy to grasp library that could be quickly
integrated into existing applications. The fact that it is GPL
licensed could cause heartache for some projects or corporations, so
proceed with caution if you are working in such a situation.
Conclusion
Sometimes the best way to pick a library is to sit down and write a
little bit with each of the contenders. A little bit of code, and a
little bit of reading goes a very long way toward a sound design
decision. Sometimes these experiments lead us to question our initial
assumptions. In this case, I am reconsidering Twisted to handle my
XMPP needs since it is mature, under active development, and installs
quickly on all of my potential target platforms.
What did I do right? What did I get wrong? What did I miss? Your
comments are appreciated!
Submit to programming.reddit.com
Posted in Code, Python, Tutorials
Posted by Tim Freund
Mon, 10 Dec 2007 06:56:00 GMT
Earlier today I posted a note in the Amazon Developer Forums about FPyS, my attempt at a Python library for the Amazon Flexible Payment Service. It's rough, very rough, but it is a decent start at a fully featured FPS library. The library provides enough functionality to run FPeS, but not much beyond that.
For the curious, the demo application is a TurboGears application that allows people to select and purchase full sized images from a thumbnail gallery. The application is configured to process payments through the Amazon FPS Sandbox, so any completed transactions are drawn from imaginary credit cards. With that in mind, I'd like to welcome anybody to attempt a sale. The more the merrier, as it will cost us nothing and help to find bugs in the library.
Posted in Open Source, Python, TurboGears
Posted by Tim Freund
Mon, 06 Aug 2007 00:42:00 GMT
I learned about the importance of schema migrations the hard way. At my previous job, I helped a team upgrade a Java web application. The upgrade involved schema changes, and I had the forethought to script the upgrade and thoroughly test it on the development database. Even with that preparation, the night of the upgrade would teach me two important questions that new developers should always ask of their team. Does the development database schema match the production database schema? And do you know how to restore the database from backups should anything go wrong? The answers to both questions on that night were no and oh no. Sometime after midnight things started working.
A better way existed. I first learned of schema migrations a few months earlier when exposed to ActiveRecord::Migrations. After using them on several projects, I was itching to have the same capability in my Python and Java projects. The Pythonic answer came in the form of Migrate, a schema migration library for SQLAlchemy, and direct support for TurboGears was added with TGMigrate. Having migrate integrated with my projects greatly reduces my blood pressure on deployment days. Of course, I still make sure that my database backups are working.
Those interested in integrating Migrate with a TurboGears project might enjoy the screencast I just completed on the topic. If you use SQLAlchemy but avoid Migrate, I would be interested to hear what is holding you back.
Schema refactoring and migration was one of three topics at the last DotNext Kansas City Tech Coffee meeting. Notes on the schema migration talk were posted to Squidoo.
My apologies to those with small screens. I will record my next screencast in a smaller window. Your comments on today's screencast are appreciated.
Posted in Code, Python, TurboGears, Tutorials
Posted by Tim Freund
Thu, 14 Jun 2007 03:15:00 GMT
Identity
defines who we are. Identity is made up of all the little
distinguishing traits that differentiate one person from another.
We've all changed our identity throughout our lives. We change from
student to graduate, single to married, dogless to dogged, and more,
but that's not what we're talking about today. Identity is the
authentication and authorization framework for TurboGears, and it is
easy to extend.
At the core of the Identity framework is an IdentityProvider.
The Identity Provider interfaces with an authentication and
authorization repository to determine two things: are you who you
say you are, and do you belong where you are trying to go. The
framework comes with two providers, one each for SQLObject and
SqlAlchemy.
We will customize an IdentityProvider to authenticate
against an IMAP server in the few steps that follow. This would be
helpful for writing a web mail application, and the concept can be
applied to other authentication mechanisms as well, including LDAP,
Radius, and others.
Action Plan
- Quickstart a project
- Create an identity provider
- Define an entry point for the identity provider
- Configure the application for the new provider
- Finish the identity provider
- Test
- Relax
The code for this tutorial is available
from subversion
or as
a tar.gz
file. It is released under the MIT license. No TurboGears
installation? Install it like
so.
Step 1: Quickstart a Project
If you don't have an existing TurboGears project to experiment
with, now would be a great time to start one.
tim@iris ~/src $ tg-admin quickstart -s -i iddemo
...
tim@iris ~/src/ $ cd iddemo
tim@iris ~/src/iddemo $ tg-admin sql create
Note the -s and -i flags. This is a
project with support for SqlAlchemy and Identity.
Step 2: Create an Identity Provider
Any object can be an identity provider as long as it supplies the
following methods: validate_identity, validate_password,
load_identity, anonymous_identity, authenticated_identity, but
it isn't always necessary to write one from scratch. Extending an
existing provider often gets an application authenticating as required
without much trouble. We will extend
the SqlAlchemyIdentityProvider in this example to
authenticate against an IMAP server.
iddemo/identity.py
from turbogears.identity.saprovider import SqlAlchemyIdentityProvider
class ImapSqlAlchemyIdentityProvider(SqlAlchemyIdentityProvider):
pass
Step 3: Define an Entry Point
The Identity Framework uses an entry point
named turbogears.identity.provider to decide what class
to use when authenticating users. We are about to define a new option
for this entry point, but further reading on the subject of entry
points is recommended. Scroll to the bottom of this entry for a couple
of relevant links. It's OK, we have the time.
This entry point step won't be necessary in the
future, thanks to
this patch, but entry points are a powerful tool and worth
learning, regardless.
setup.py
setup(
name="iddemo",
... (more setup stuff) ...
entry_points="""
[turbogears.identity.provider]
imapsa = iddemo.identity:ImapSqlAlchemyIdentityProvider
""",
... (more setup stuff) ...
)
To let the setuptools system know about this new identity provider, run the following:
tim@iris ~/src/iddemo $ python setup.py develop
Step 4: Configure the Application
With our imapsa option defined for
the turbogears.identity.provider entry point, we can now
configure the application to call the new provider. There is a value
named identity.provider in app.cfg. We will replace the
existing value with imapsa. While app.cfg open is,
add the other three lines in the following example. They will be
explained in the next step.
iddemo/config/app.cfg
...
identity.provider='imapsa'
identity.imapprovider.imap_authoritative=True
identity.imapprovider.server="localhost"
identity.imapprovider.port=143
...
The application is now ready to run with the new identity provider.
Restart the application if it is currently running so that the
configuration change will take effect.
Step 5: Finish the Identity Provider
Now let's dig in and implement the new authentication behavior.
iddemo/identity.py
...
class ImapSqlAlchemyIdentityProvider(SqlAlchemyIdentityProvider):
def __init__(self):
SqlAlchemyIdentityProvider.__init__(self)
# These three lines get the configuration parameters we set in app.cfg
self.imap_authoritative = get("identity.imapprovider.imap_authoritative", False)
self.server = get("identity.imapprovider.server", "localhost")
self.port = get("identity.imapprovider.port", 143)
# These four lines make the user and visit classes available for
# later use
user_class_path = get("identity.saprovider.model.user", None)
self.user_class = load_class(user_class_path)
visit_class_path = get("identity.saprovider.model.visit", None)
self.visit_class = load_class(visit_class_path)
def validate_identity(self, user_name, password, visit_key):
if self.validate_password(None, user_name, password):
user = session.query(self.user_class).get_by(user_name=user_name)
if not user:
if self.imap_authoritative:
user = self.user_class()
user.user_name = user_name
user.save()
session.flush()
else:
return None
link = session.query(self.visit_class).get_by(visit_key=visit_key)
if not link:
link = self.visit_class(visit_key=visit_key, user_id=user.user_id)
session.save(link)
else:
link.user_id = user.user_id
session.flush()
return SqlAlchemyIdentity(visit_key, user)
return None
def validate_password(self, user, user_name, password):
rc = False
try:
imapcon = imaplib.IMAP4(self.server, self.port)
except:
log.error("Could not establish connection to server at %s:%d" % (self.server, self.port))
return rc
try:
if imapcon.login(user_name, password)[0] == 'OK':
rc = True
except:
# Probably threw an error for invalid username/password
log.info("Passwords don't match for user: %s", user_name)
imapcon.shutdown()
return rc
Take a look at each method to figure out what they accomplish.
__init__ invokes the constructor of the
SqlAlchemyIdentityProvider and then collects the configuration
parameters required for authentication.
validate_identity first
invokes validate_password. If the password validate
succeeds, the user is selected from the database. If the user does
not exist in the database, the provider will create the user when
the imap_authoritative option is set. Should you not
require this capability, you can remove this method entirely and rely
upon the implementation in SqlAlchemyIdentityProvider.
Finally, this method links the user with the
current visit_key.
validate_password handles all IMAP access. It is the
perfect method to override if all you need to do is change the
identity authentication mechanism.
Step 6: Test
Open controllers.py and change the identity decorator to require an
authenticated user when accessing the index page.
iddemo/controllers.py
class Root(controllers.RootController):
@expose(template="iddemo.templates.welcome")
@identity.require(identity.not_anonymous())
def index(self):
import time
flash("Your application is now running")
return dict(now=time.ctime())
The moment of truth has arrived. Point a browser at the
application. You should see a login page, and a valid IMAP
username/password should provide access to the application.
Step 7: Relax
We're done here. Go tell your friends and family how cool you are.
Any feedback is appreciated. Leave a comment here, post a response
on your own blog, or send an email to tim -at- achievewith -dot-
us. You can also usually find me lurking in #turbogears on
irc.freenode.net as timphnode.
For more information, follow these links:
Posted in Code, Python, TurboGears, Tutorials
