PHP: One Year On

I try to learn a new programming language each year. In recent years, my bias has been towards a more functional nature with Erlang and Clojure being two very enjoyable and intellectually satisfying choices. I didn’t make much fanfare of my new language for 2011, mainly because it was born of necessity rather than a way to challenge myself. Attendees to the London Python Code Dojo will probably have heard a few things from me, not always in a flattering light.

It’s roughly a year on from when I first started doing serious things with PHP and I thought it might be an excellent time for a quick bit of reflection. One problem with Erlang and Clojure has been that I’ve not directly had to use either in a production environment, or deal with a legacy codebase written in said languages. I say directly, because I’ve been working with RabbitMQ, which is written in Erlang… but that doesn’t really count.

So, first a disclaimer and some background. It’ll seem very negative, so any fans of PHP please bear with me…

I’m not sure when I first encountered PHP, but it was still called Personal Home Page at the time. Wikipedia tells me it was called Personal Home Page Tools, but anyway. I was writing Perl at the time and it didn’t appeal to me. I rediscovered it in 2000 when I spent a day updating someone’s website that was using PHP 3. At the time I was writing Python and Zope code, and it felt horribly backward and just plain wrong. (I said this would seem very negative, bear with me…)

From then on, PHP didn’t really register with me much. Generally, my encounters with PHP were via poorly written and insecure websites, written by people who really didn’t know how to program. This tainted my view of PHP and the associated community quite considerably.

So, with that background in mind, I joined a new company at the end of 2010 which had a very complex legacy web application written in… PHP. I was thrilled, obviously. But the good news was that I was recruited to help rewrite the application from scratch, replacing the considerable backend functionality with Python and using PHP and JavaScript for the front-end. For various reasons, I ended up taking over maintenance of the existing website for a few months, which gave me a rather urgent requirement to get down and dirty with PHP.

Now, working with the legacy code base, and given my prejudice already, this didn’t initially endear me to PHP. There seemed to be a lot of workarounds and convoluted functions to replicate what comes for free in Python. The old bracing wars of C were back, with different past developers having their own unique style of formatting – and not always lining things up nicely, which makes Pythonistas like me cry. Don’t even get me started on all the includes being pulled in causing all kinds of chaos, the warnings, the functions with lines of the code measured in four digits (I wish I was joking). A true maintenance nightmare.

But nestled amongst the mess were signs of a language that had changed radically from PHP 3, and was actually quite nice once you got to know it. Ever since I discovered Perl, I’ve always preferred dynamic languages because they don’t get in the way of solving problems. They’re quicker to write, easier to test, more adaptable to change, and very powerful. You lose a minor few things in the process, but gain a heck of a lot more. I still feel PHP has a little way to go, but on the whole it doesn’t get in my way too much these days – a little quirky in places, but people find Python and Clojure quirky too. And many people are totally freaked out when they see Erlang.

The PHPUnit testing framework is pretty good. Easy to convert to from Python’s unittest (they’re both part of the xUnit family), and comes with things like mocking and database testing tools as standard. Pretty cool. Reflection was a little clumsy to deal with at first, a little inelegant, but it works fine once you understand it. TDD wasn’t initially part of the company’s workflow but PHPUnit made it very easy to introduce, which was great.

There are three web frameworks that I’ve used/experienced. Symfony is being moved away from at work and I must admit I’m not overly fond of it – but this could just be down to the way it’s been used by bits of the legacy code base. Zend Framework is the main one we’re adopting, and it’s proving pretty good. We’ve had a a few niggles and frustrations, but nothing major except that the documentation seems to be great for introductions, but not very good for the detail. There have been moments when something that should be easy to figure out wasn’t because of the docs – but the great thing about open source is that you can always wade through the code to find the information you need. I think I’ve been spoilt by Django.

Finally, honourable mention goes to the third framework, which we use for some smaller client sites: Kohana. It’s an elegant, lightweight framework, although the documentation could do with some improvements (I feel a little guilty because I was going to find time to contribute some but never did). Worth a look, so giving it some free PR.

Although Python will remain my main language of choice, and I’ll be sticking to Django and Flask for my own projects, I can’t be too dismissive of PHP and PHP developers these days (although seriously guys, who thought adding goto to PHP 5.3 was a good idea?*). The language has matured into a respectable dynamic language, and frameworks like Kohana and Zend deserve particular mention. There seems to be a lot of support and encouragement within the community for good practices like automated testing and better structuring of code which is great to see.

* Although the docs do warn you of Velociraptors: http://php.net/manual/en/control-structures.goto.php (scroll down a bit)

So there you go: PHP ain’t that bad after all.

As for my language for 2012, I was trying to decide between ones like Lua, Squeak, Haskell, Java and er… something else (must’ve been good!). In the end, I decided not to decide and see what happens this year. Meanwhile, I’m going to keep working on improving my PHP skills, update my JavaScript knowledge (part of a general “things have changed on the client-side since 2004″ approach) and continue learning Clojure.

PyCon UK 2011

I attended PyCon UK 2011 at the weekend, which was loosely run as an unconference. As I was about to start writing about the conference, I realised I haven’t written anything about PyCon AU last month in Sydney, which I also had the pleasure to attend. Two conferences in
two months! I’ll try to mention my Australian experiences in another blogpost, but will mention some items of relevance in this post…

Once again, John Pinner and his trusty team put on an excellent event. The TechnoCentre in Coventry provided a superb venue, and the on-site catering was very good. You couldn’t go anywhere without finding a water cooler, fine teas, coffee and other beverages. Delegates were fuelled up in the mornings with breakfast baps and pastries, offered a decent lunch (especially on the Sunday) and served a fine conference dinner on the Saturday evening.

It’s been a good year for keynotes for me: PyCon AU had Audrey Roy, Mary Gardiner and Raymond Hettinger. Audrey’s talk on diversity was particularly important, emphasising that diversity isn’t just about encouraging more women into computing. Worthy though that goal is.

PyCon UK had keynotes from Allison Randal, Laura Creighton and Lorna Jane Mitchell. Allison kicked things off with “The Fallacy of the Zero-Sum Game”, discussing why free software is the future of technology because the values of cooperation and collaboration provide a better environment for improving technology. Even though I was clutching my freedom-hating MacBook, free/open source software is something I believe in for the same reasons Allison does.

Laura Creighton is always a joy to watch in action. “Reflections on the work of Sociologist Charles Perrow, and what he can teach software developers” did seem to leave a few people in the audience scratching their heads, but I really enjoyed it. The general idea is understanding the lessons learnt by Charles Perrow on analysing “risky systems” and applying to software development. He knows a bit about this through his study of accidents such as Three Mile Island. I’ve discoverd that simple things are more complicated than I first thought, and that tightly-coupled, complex and unpredictable systems need to conform to two mutually incompatible states to be controlled and understood. Which is why they are both very risky and lead to catastrophic accidents when things go wrong: aircraft crashes, spacecraft breakups, nuclear reactor explosions. The sort of thing you don’t want to be involved in.

It’s one of the nice things about the Python community that PHP developer Lorna Jane Mitchell can be welcomed wholeheartedly. I don’t have her talk title to hand, but she conducted a wonderful, slide-free talk about contributing to open source projects and the benefits outside the project that can result: job offers, recognition, personal improvement. It was well-timed, as I was about to undertake my first conference talk…

I took the stage for the Testing Workshop alongside Michael Foord. I wasn’t quite sure what level to pitch the workshop at, or even what to cover. I’d abandoned the tutorial style I’d originally thought about, although there was a bit of legacy from that era in some of my slides. It was on researching the subject that I’d realised a fundamental issue with testing: no one really agrees on the terminology, and most books and training material for developers don’t really cover it in much breadth. I took a gamble and decided to start from basics, to cover the widest possible audience and to try and get people thinking about the types of testing and the terminology used. I also opted to cover the very basics of Python testing, ditching some material on TDD and dealing with legacy code.

As I was leading up to the presentation, I was beginning to think it might’ve been the wrong decision when I looked out at a very large audience and spotted some very knowledge people in the Python community looking back at me. Uh oh. I did give a clear warning, but no one left which was a good sign. Also, despite preparing my set-up in advance, there were some additional technical hurdles to overcome – notably, coordinating with the live streaming! At least the internets don’t heckle.

My first section covered what testing is and the types of testing. The second section was really more of an overview of testing with Python and so most of the audience knew the options already – but important to ensure everyone is aware. However, it lead to some interesting questions and responses from the audience, which was good. I then handed over to Michael who covered his Mock library and served the audience with some much-needed in-depth material. Jonathan Hartley followed-up with a talk on modifying the Django test runner, a modification I wish I’d had earlier this year while debugging a nasty issue with test fixtures and multiple databases.

Afterwards, I read some very nice tweets in support of the subject matter being presented, and I had some good bits of feedback post-talk. I’ve spoken a couple of times at the London Python Code Dojo, but have had the advantages of a smaller audience, smaller venue and a subject of my choice – not much different to when I stand up in front of a small group of people and teach them to snowboard. The venue was definitely a lot bigger, the audience a lot bigger and strangely the podium and lectern made me feel the most awkward. The first five minutes my throat was the driest it has probably ever been and I then worried I’d spill my bottle of water over the laptop and other electronics surrounding me!

In the end, I rather enjoyed it. To use a phrase from Russel Winder, I’ve picked up the bug. Even while I thought I was failing badly, I would look out across the audience and see someone smiling, nodding their head in agreement or looking like they’d just discovered something new. Which is all that matters. If you attended, I’d like to say thank you for your support and hope you came away with something of use.

Next year, I’m toying with either a talk on beginning Test Driven Development or something a bit more technical: common patterns for testing legacy code. Although I’ve been dealing with PHP legacy code recently, I’ll aim to keep it in Python. The principles are fairly similar across the dynamic languages.

Turning to other matters, teaching school kids to program was a recurring theme during the weekend. The BBC Micro featured prominently in discussions, as did the new Raspberry Pi – which is a wonderful piece of kit scheduled to cost just £25.

I discovered programming through the BBC Micro at school. I’m not sure the exact age I was, but I was probably about 5 or 6 and the Beeb had just been made available in schools. When we weren’t playing educational games, we could fire up the BASIC interpreter and do some very simple things. However, it wasn’t until my parents bought me a Commodore Plus/4 a couple of years later that things really took off for me. Doing it as a career wasn’t something that occurred to me until university, or just before. It was always a fun hobby.

Logo on the RM Nimbus came a bit later, and I remember returning to my old primary school a few years later and teaching a group of very excitable kids to draw on the screen using Logo. Kids of all abilities loved it, and my hope is that one of those kids ended up programming.

Back in my day (now I sound really old), computing wasn’t on the syllabus because computers were new, a bit of a novelty and the teachers had little or no experience of them. That was a reasonable excuse in the 80s. We live in the 21st century now, and I was horrified to discover recently that computing skills in the classroom consist of learning how to use Word or Powerpoint. The computer is this magic box that comes with games, Facebook and something to type out homework with. If anything, we’ve actually regressed.

At PyCon AU, I attended a talk on the NCSS Challenge which teaches school kids across Australia to program. It even caters to different levels of ability, from complete beginners to those who already have programming skills. You can watch the talk on the technical side from the PyCon AU YouTube channel.

That kind of scheme is just wonderful, and it makes me sad that the UK has nothing similar. Instead, we’re raising a new generation of kids who think a computer is indistinguishable from magic, the software just appears, and everything has a nice, neat Microsoft brand emblazoned across it. We’re not teaching kids to fully explore technology at all levels, to create applications themselves and become producers not consumers. It makes me wonder how many great programmers of the future are not discovering their calling earlier in life?

Anyway, thank you to all the crew at PyCon UK – the organisers, the venue staff, and my fellow delegates. See you all next year!

Fluid In Your Ear: When FluidDB Meets Heavy Metal

(This is based on a 15 minute talk for the London Python Code Dojo – slides available from SlideShare)

My interest in FluidDB began earlier this year when I attended a talk by Nicholas Tollervey at the London Clojure Dojo. I was expecting yet another talk about yet another non-relational database, but what I discovered was something different. The idea of a shared database storing “things” which anyone could tag with data seemed to be a rather powerful concept, yet simple and elegant. I thought it was a very cool and interesting idea.

But there was a problem.

How could people actually explore the “Fluidverse”? While people using FluidDB are building up conventions, such as the naming and content of tags, and there are tools out there to drill down through the hierarchies of tags and namespaces… there had to be an easier way to find the tags that were of interest to me. I decided I needed data in order to begin finding ways to explore… but what data?

So, while pondering the idea one evening, I was listening to the band Napalm Death and realised I had the answer. One of the things I love to do is find new bands, particularly extreme metal ones, and one way I do this is follow links between bands on Wikipedia. Wikipedia is a great source of band biographies, the content is under a Creative Commons license, and the band biographies often have lists of related bands and genres.

This seemed like a really good starting point to take data I’m interested in, build relationships between the data and give me something to start exploring with. I hacked together a scraper which used Napalm Death as a starting point and branched outwards in a “six degrees of separation” way, initially dumping the information directly into the FluidDB sandbox.

After a few runs, it made more sense to scrape to an intermediate file, and load that instead – allowing me to clean up typos, adjust names, amend tags and also allow me to regenerate the data in FluidDB’s sandbox without having to keep hitting Wikipedia. An example of the output format is as follows:

band:Burzum
  metaljoe/music/band_name = Burzum
  metaljoe/music/source_url = http://en.wikipedia.org/wiki/Burzum
  metaljoe/music/genre/black_metal -> Black metal
  metaljoe/music/genre/dark_ambient -> Dark ambient
  metaljoe/music/related_bands = ['Darkthrone', 'Mayhem', 'Old Funeral']

I wasn’t planning to release the source code, but have had some interest in it so I’ve decided to release it under the MIT license. You can find the code on my BitBucket account: http://bitbucket.org/metaljoe/fluidinyourear-scraper – note this is just the scraper code, not the loader.

With the data in place, I then needed to build something for exploring the relationships in the data. Enter “Fluid In Your Ear”, a very simple web application built around Python, Django and the excellent FOM (Fluid Object Manager) created by Ali Afshar. Given the nature of the bands, there is also a liberal application of Heavy Metal Umlauts – the power of which, courtesy of a particular Black Metal band, managed to crash the FluidDB sandbox a few times by exposing a unicode bug.

The application is deliberately very simple. I’m not a graphics genius (painting with real acrylic paints is my field), and at the moment it’s a basic core – you can browse genres and bands, and explore relationships between the two. I’ve already discovered some new bands through following the links, and re-discovered some older ones.

Due to the six-degrees nature, there is quite a lot that doesn’t fit into a metal or punk category which is quite cool. I’ve encountered a jazz musician called John Zorn who has crossed into hardcore punk and grindcore, to produce some outstanding music I would probably not have found before.

The source code is pretty grotty and the first casualty was a lack of tests. Shocking. In order to improve my confidence in the code and make it easier to refactor, I added unit tests using Django’s test harness and some functional testing using the Twill web testing framework. An example of the Twill test code is as follows:

# test missing genre
go http://127.0.0.1:8000/genre/progressive_vegetarian_grindcore
code 404

# test with trailing slash
go http://127.0.0.1:8000/genre/jazz/
code 200

# test without trailing slash
go http://127.0.0.1:8000/genre/jazz
code 200

# check page contents
find '<h2>Jazz</h2>'
find '<div id="related_bands">'
find '<li><a href="/band/Frank%20Zappa">Frank Zappa</a></li>'

So where next?

Well, first off is to get the application online so my plan was to port to Google App Engine. Unfortunately, I hit a few snags with the fact my app runs Django 1.2 and App Engine is using 1.1. I considered bundling Django in the app, but it became obvious that I’m not really using much of Django’s functionality – some URL routing and templates. The creator of FOM introduced me to Flask, a lightweight web framework, and it looks perfect for my needs. So I’m going to port to Flask and Google App Engine at the same time.

Another thing I want to do is add a JavaScript “social” layer over the top, allowing some of the richness of FluidDB to shine through and allow the addition of functionality not originally envisaged. I’m also hoping people will tag bands with ratings, annotations and the like with a hope to making recommendations possible.

In a similar way, I want the application code to be reusable and reskinnable so people can customise and create their own starting point. Maybe someone will produce a Classical In Your Ear in the future?

Source code is available from BitBucket, if you fancy a giggle at the clumsy bits: http://bitbucket.org/metaljoe/fluidinyourear – released under the GNU Affero GPL.

Twill: The Command Line Web Browser

I’ve had Twill on my list of technologies to look at for a couple of years now. I finally had the chance to use it today and have been mightily impressed so want to share my thoughts.

For those not aware, Twill is a scripting language implemented in Python for web browsing. Using the command line or scripts, you can interactively or programmatically access websites.

Now, my particular circumstances were that I wanted to start writing functional tests for a web application I am developing in Django with a FluidDB backend. Although Django provides superb additions to the standard Python unittest, you’re still writing code that is relatively low-level. I wanted to step back from the Python implementation and think of tests at a higher level, thinking in terms of user functionality rather than objects, methods and functions. I wanted to express my tests in a more high-level way, a more domain-specific way.

Twill’s core language is simple and readable, which is just what I wanted. It can be easily hooked into Python and unittest if I desired that functionality in the future.

Let’s say I have a Django development server running on my local machine and I want Twill to go to a particular URL and check I have an HTTP status returned of 200:

go http://127.0.0.1:8000/search/genre
code 200

That’s it: clear and readable. If the status isn’t 200, Twill will raise an error and report the code it actually received.

One thing I need to do with my tests is check for the presence of certain HTML elements, to ensure things like forms and headings are being set correctly:

find '<h2 class="heading">Jazz</h2>'

Here I’m checking that I have the heading Jazz present in the current URL loaded. Again, if this condition is not met, Twill will raise an appropriate error message.

Am I sure a valid form submission is working correctly and redirecting me to the right URL? If I set the genre field of the form named search_form to the value “grindcore”, and then submit the form I can then verify the returned URL is what I expect it to be:

formvalue search_form genre grindcore
submit
code 200
url http://127.0.0.1:8000/genre/grindcore

The 200 is possibly superfluous, but it adds a little extra confidence that things are working okay.

I’m not one for things which bring in lots of dependencies, provide a high barrier to entry or get in the way of the task at hand. Twill is painless to install, quick to learn and enables readable web tests to be written quickly and easily using just a plain old text editor. It might lack the functionality of tools like Selenium, but the Python API provides an easy way to extend its capabilities if desired.

Go take a look: http://twill.idyll.org/

Bit Counts in Python, Erlang and Clojure

The subject came up yesterday about counting bits. I remembered the subject from university in the context of error checking, but I couldn’t remember the specifics during the conversation. It was only later that I recalled the subject of Hamming values or weights, and from there I tracked down an algorithm by Brian Kernighan (or Peter Wegner, or Derrick Lehmer). For fun, I translated the C source to Python to yield the following:

def bit_count( v ):
    c = 0
    while v > 0:
        v &= v - 1
        c += 1
    return c

For a bit of practice, I then opted to convert into Erlang:

bitcount(V) ->
  bitcount(V, 0).
bitcount(0, C) ->
  C;
bitcount(V, C) ->
  bitcount( (V band (V - 1)), (C + 1)).

I could probably roll the two arity 2 clauses into one, but this is a minor habit from my old Prolog days and I always feel it looks a bit cleaner than opting for a case statement. Feel free to disagree.

Since Clojure is my “language for 2010″, and I’ve been rather slack on keeping up with it lately:

(defn bit-count
  ([v]
    (bit-count v 0) )
  ([v c]
    (if (zero? v)
      c
      (recur (bit-and v (dec v)) (inc c)))))

This feels like it could be expressed more functionally, dropping the two argument version for a compact one-argument version. This might be an interesting exercise for when I have a spare moment, unless someone out there already has a solution?

Just thought I’d share….

EuroPython 2010

Well, I’m back from another great EuroPython – my thanks to John Pinner and his EuroPython crew, the presenters, Josette from O’Reilly, and all my fellow delegates. Regrettably, I might not be able to make the next EuroPython in the wonderful city of Florence, but I might be able to make it to PyCon AU in Sydney next year. Fingers crossed on that one – I might even pluck up the courage to do a talk!

Monday

Monday’s talks kicked off with Raymond Hettinger’s “Idiomatic Python”. He guaranteed everyone would learn something new and he definitely delivered. I walked away with some interesting gems of Pythonic knowledge, from basic stuff like enumerate() through to more advancing knowledge like the surprising behaviour of super().

Ezio Melotti followed after the break with a useful overview of the Python development process, which inspired me to join the Python Core sprint on Friday. I’ve already contributed my first couple of patches to Python 3.2 – the first of which has been committed to trunk already.

“How Import Works” was a run through by Brett Cannon on how, you guessed it, Python imports code. Although you’ll probably never need to modify the standard import process, it’s fascinating to discover how it all works. Best of all, new utilities in Python 3 make it much easier to customise things safely if you ever need to.

“PostgreSQL’s Python Soup” was a little disappointing, with a good chunk of the talk being a general one about relational databases. The interesting stuff about connecting to PostgreSQL came quite near the end when we were introduced to the many different connectors. And I mean many.

Monstrum is a new HTTP functional testing framework for Python 3, and was detailed in “Testing HTTP Apps with Python3″. It looks very cool, and the logo would make a great t-shirt! Continuing the web testing theme, Raymond Hettinger followed up with “Python & Selenium Testing”. Alongside an overview of Selenium, Raymond introduced Sauce Labs, a company providing commercial support for Selenium, the web testing framework, via their cloud service.

To round off the day, I attended two talks by Richard Watts and Tony Ibbs of Kynesim who presented Muddle, their open source build system which looks very cool, and KBUS which is an elegant and lightweight messaging system implemented as a Linux kernel extension.

Tuesday

The day’s talks began with a rather packed presentation by Guido on AppStats, an AppEngine monitoring tool. I must admit that I did partially go to hear him talk, as did probably a lot of other people, but also because I was interested to learn more about AppEngine. I picked up some interesting bits of information about AppEngine internals, and the tool looks fantastic. However, I did feel a bit out of my depth.

Bart Demeulenaere’s “Pyradiso Rapid Software Development” was a call to arms, a request and discussion on how to create a rapid application development framework in Python that supports the multicore world out-of-the-box.

Engaging both my computer science and art historical sides, I then attended Richard Barrett-Small’s “The Trojan Snake”. Richard works for the Victoria & Albert Museum, and has been instrumental in introducing Python and Django as a way to clean-up and replace a mess of bespoke PHP, ASP.NET and Java. An interesting look at how an organisation faced issues with bespoke applications in a variety of technologies, and found Python to be a flexible and effective solution. It’s a shame more people didn’t attend.

I was an avid reader of Michele Simionato’s blog posts, covering his experiences with Scheme, so it was great to hear him speak at EuroPython with his talk entitled “Intro to Functional Programming”. Functional programming is receiving a surge of interest amongst programmers, thanks to the search for better ways to deal with concurrency, so it was good for someone to cover other reasons to start exploring FP.

“Tickery, Pyjamas & FluidDB” by Terry Jones was a run through of Tickery, a FluidDB application for analysing Twitter friend sets. Regular readers of this blog will know that I’ve been interested in FluidDB since attending an introductory talk by Nicholas Tollervey. Definitely worth a look.

Rounding out the end of the day were Andrew Godwin’s “Fun with Databases and Django” and Russel Winder’s “Pythonic Parallelism with CSP”. Andrew’s talk provided a general overview of some of the more advanced features of the Django ORM, including Django’s interaction with non-relational databases such as MongoDB and CouchDB. Meanwhile, Russel Winder is one of the key players in pushing Python support for concurrent and parallel programming. CSP has been around for over 30 years, and now Python programmers can begin to take advantage of the ideas thanks to projects like Python-CSP and PyCSP.

Wednesday

Wednesday kicked off with Nicholas Tollervey’s “Organise a Python Code Dojo!”. Nicholas shared his experiences running the London Code Dojo – the joys, the successes… and the things that didn’t quite go to plan.

“HotPy – A comparison” was Mark Shannon’s comparison between three different VMs for Python: PyPy, Unladen Swallow and his own HotPy project which runs on top of the GVMT.

Henrik Vendelbo gave two short talks in succession. The first was “Real Time Websites with Python” which gave a simple example of the Tornado web server, released as open source by FriendFeed. The second was “Custom Runtimes with PyPy”, a very interesting talk on using PyPy to provide an easy way to bundle up a custom Python application as a self-contained app.

qooxdoo is a full-featured JavaScript framework for developing Rich Internet Applications. It’s quite an impressive piece of work, not only because of the sheer size of the code base. Managing large code bases can be a headache, especially if you also need to process that source to produce more compact, optimised versions for delivery to specific browsers. In “Python for Javascript Apps”, Thomas Herchenröder explained how they use a set of Python tools to make dealing with the code a much more pleasant experience.

“A Python’s DNA” by Erik Groeneveld illustrated an issue I’ve hit myself: how to configure a complex component-based system in a simple, maintainable way. The answer Erik came up with for Meresco is not to use XML or similar to express configurations, but use Python instead to provide a rich, flexible, and elegant way to define dependencies, data flows and configurations.

It’s not often a talk begins by telling you that the subject of the talk is now officially dead, and a new project has taken over. Kay Schlühr’s “The Trails of EasyExtend” quickly changed to “The Trails of LangScape”. LangScape evolved from a toolkit for writing language extensions to Python, but takes a much broader view with the aim of allowing easy ways to extend many different languages.

Thursday

The morning began with “Building a python web app”. Anthony Theocharis and Nathan Wright took us through their journey to find the right Python web framework to implement MediaCore. They discussed Django, TurboGears and Pylons, illustrating their experiences with each and why they settled on Pylons. The different approaches to database access and templating languages were also examined in brief, followed by how they open sourced the project. An entertaining and interesting talk, refreshing by the fact they apparently came from a non-Python background so lacked any preconceptions.

Next up, Denis Bilenko introduced gevent in his talk “gevent network library”. gevent is a network library for handling large numbers of connections efficiently and, more importantly, elegantly. Optional, seamless monkey patching of standard library modules makes it very easy to work with existing code and libraries. I also discovered Gunicorn a Python implementation of Ruby’s Unicorn webserver.

Finally, my selection of conference talks closed with “Arduino and Python” by Michael Sparks. The talk should probably have been renamed “How not to blow up your computer” as Michael crammed in a quick intro to electricity and electronics into a few slides to dispel concerns about inexpensive pieces of electronics destroying your computer. The talk was illustrated by his prototype of a rather interesting TV remote control developed at a BBC R&D workshop, built with an Arduino… and Python of course.

Friday

Big thanks to Brett Cannon and Richard Jones for putting up with me all day while I got to grips with downloading, building, testing and patching Python 3.2. It was a really enjoyable day, and very satisfying to have a patch committed to the Python source code – no matter how small.

FluidDB Objects in 10 Minutes

When looking at FluidDB‘s tags (see FluidDB Tags in 10 Minutes), we briefly touched upon FOM‘s Object class (not to be confused with Python’s native object class). I instantiated an object representing the planet Mars:

>>> from fom.mapping import Object
>>> mars = Object(about="planet:Mars")
>>> mars
<Object planet:Mars>

The thing to note is that I supplied the about keyword. By instantiating this way, I have no way of knowing if it existed before or not – the appropriate FluidDB object is created if one doesn’t exist already. In the big scheme of things it probably doesn’t matter, but it’s important to be aware.

The about keyword sets the value of a useful tag that is often assigned to objects when created: fluiddb/about. The about tag is used to label an object with something descriptive. It can be anything, but certain conventions are slowly evolving amongst FluidDB developers and users.

For example, an application I am developing uses the convention “band:<band name>” to label an object as representing a particular musical band. Nicholas J Radcliffe has written more on the subject on his About Tag blog which is a recommended read.

Let’s take a look at that tag:

>>> mars.get("fluiddb/about")
('planet:Mars', 'application/vnd.fluiddb.value+json')

An optional about tag is very useful but not all objects have, or indeed need, one. Fortunately, we can also refer to any object by its unique GUID which is guaranteed to exist:

>>> mars.uid
'276da99e-c8d9-42c9-99ae-3db69a5e9ef0'

If I wanted to load an object by its GUID, I simply supply the uid parameter (it’s a good job I know the ID!):

>>> red_planet = Object( uid='276da99e-c8d9-42c9-99ae-3db69a5e9ef0' )
>>> red_planet
<Object planet:Mars>
>>> red_planet.get("fluiddb/about")
('planet:Mars', 'application/vnd.fluiddb.value+json')

Tags form the real power of FluidDB, so it’s important to be able to determine what tags have been added to an object. Let’s take our Mars example first:

>>> red_planet.tag_paths
['metaljoe/foo', 'fluiddb/about']
>>> red_planet.tags
[<fom.mapping.Tag object at 0x57edb0>, <fom.mapping.Tag object at 0x57fd10>]

The first attribute stores the tag names, while the second returns these tags instantiated as FOM Tag objects. We can check they are indeed proper Tag instances by peeking at the descriptions:

>>> [ tag.description for tag in red_planet.tags ]
['behold: the foo tag', 'A description of what an object is about.']

I can also test for the presence of a tag by using the has() method:

>>> red_planet.has( "metaljoe/foo" )
True
>>> red_planet.has( "the_mekon/likes" )
False

URLs were briefly mentioned in an earlier post about tags – each tag has a corresponding URL. FluidDB’s RESTful API means each object in the system also has a unique URL. For example, on the FluidDB sandbox instance our planet:Mars object can be referenced by the URL http://sandbox.fluidinfo.com/objects/276da99e-c8d9-42c9-99ae-3db69a5e9ef0 – pointing your browser to that URL returns a little bit of JSON:

{"tagPaths": ["metaljoe\/foo", "fluiddb\/about"]}

An object’s tags can also be referenced via URLs. For example, the fluiddb/about tag is referenced by the following URL:

http://sandbox.fluidinfo.com/objects/276da99e-c8d9-42c9-99ae-3db69a5e9ef0/fluiddb/about

Which yields the contents of the tag for that object:

"planet:Mars"

Note that a browser will be making GET requests to these resources. Being RESTful, we can make full use of different HTTP methods (GET, PUT, POST, DELETE) to perform reading, creating, updating and deleting of our FluidDB resources, but that’s a subject for another time.

EuroPython 2010 – Revised Talks of Interest

With about a week to go until EuroPython 2010, I realised the talks timetable has shifted around a bit so some clashes have been resolved, others created, and some talks have suddenly grabbed my interest.

I’m currently looking at the following:

Monday 19th July

• Idiomatic Python – Raymond Hettinger
• The Development Process of Python – Ezio Melotti
• How Import Works – Brett Cannon
• PostgreSQL’s Python soup – Hannu Krosing
• Django and the Semantic Web – Zachary Voase
• Python & Selenium Testing – Raymond Hettinger
• Muddle, inventing a build system – Richard Watts
• KBUS: A simple messaging system – Tony Ibbs

Tuesday 20th July

• Appstats – Guido van Rossum
• A case for accessibility – Stuart Bowness
• The Trojan Snake – Richard Barrett-Small
• Intro to Functional Programming – Michele Simionato
• Tickery, Pyjamas & FluidDB – Terry Jones
• Visualizing Software Quality – Mark Fink (clashes with Andrew’s talk)
• Fun with Databases and Django – Andrew Godwin (clashes with Mark’s talk)
• Pythonic Parallelism with CSP – Russel Winder

Wednesday 21st July

• Taming Twisted with Generators – Raymond Hettinger
• Adapting libraries to Twisted – Esteve Fernandez (clash!)
• Deferred Gratification – Terry Jones (clash!)
• Real Time Websites with Python – Henrik Vendelbo
• Custom Runtimes with PyPy – Henrik Vendelbo
• Python for Javascript Apps – Thomas Herchenröder
• A Python’s DNA – Erik Groeneveld
• The Trails of EasyExtend – Kay Schlühr

Thursday 22nd July

• Building a python web app – Stuart Bowness
• gevent network library – Denis Bilenko
• Arduino and Python – Michael Sparks

Look forward to seeing everyone there, new faces and old, and I’m in the market for some new Python challenges.

FluidDB Tags in 10 Minutes

So, with your namespaces structured it’s now time to add tags.

I’ll assume you already have a FluidDB session open. To add a tag to your current namespace:

>>> namespace.tag_names
[]
>>> namespace.create_tag( "foo", "behold: the foo tag", False )
<FluidResponse (201, 'application/json', None, {'id': 'cac795ce-777f-4d64-94cc-6df5356eb651', 'URI': 'http://fluiddb.fluidinfo.com/tags/metaljoe/foo'})>
>>> namespace.tag_names
['foo']

Okay, let’s take a look at that in more detail. The create_tag() method has three arguments: the tag name, the tag description and a mysterious boolean value at the end. The last value specifies whether the tag should be indexed – in this case, I have left it unindexed. At present, I’m not sure how the indexing works or if it gives any noticeable advantages at present. I should really make an effort to find out….

When my tag has been created, FOM returns a response from FluidDB. For those not fluent in HTTP, 201 is the Created response status: my tag was successfully created. You can also see that FluidDB has assigned my tag a unique GUID, and that the URI of my tag is http://fluiddb.fluidinfo.com/tags/metaljoe/foo – everything in FluidDB can be referenced by a URL, FluidDB’s RESTful API, and everything has a GUID.

>>> namespace.tag_paths
['metaljoe/foo']

Yup, that looks pretty conclusive.

Having a tag defined is one thing, the power of that tag comes when you apply it to objects in the system. Let’s find a suitable object first:

>>> from fom.mapping import Object
>>> mars = Object(about="planet:Mars")
>>> mars
<Object planet:Mars>

Tagging the object is trivial:

>>> mars.set( "metaljoe/foo", None )

As is retrieving the value of that tag:

>>> mars.get( "metaljoe/foo" )
(None, 'application/vnd.fluiddb.value+json')

Now, setting the value to None is pretty dull. How about we set it to something more interesting?

>>> mars.set( "metaljoe/foo", "The red planet" )
>>> mars.get( "metaljoe/foo" )
('The red planet', 'application/vnd.fluiddb.value+json')

Actually, we might not want the value set in the system as a FluidDB JSON value. Let’s try setting it to the text/plain MIME type:

>>> mars.set( "metaljoe/foo", "The red planet", "text/plain" )
>>> mars.get( "metaljoe/foo" )
('The red planet', 'text/plain')

Python lists can also be stored easily enough:

>>> mars.set( "metaljoe/foo", ["The red planet", "My favourite planet"] )
>>> mars.get( "metaljoe/foo" )
(['My favourite planet', 'The red planet'], 'application/vnd.fluiddb.value+json')

Finally, tags have their own class in FOM:

>>> from fom.mapping import Tag
>>> tag = Tag( "metaljoe/foo" )
>>> tag
<fom.mapping.Tag object at 0x5b62d0>
>>> tag.description
'behold: the foo tag'

FluidDB Namespaces in 10 Minutes

I’ve been working on a pet project with FluidDB recently and thought I’d post a few observations.

FluidDB is the core technology being developed by a company called FluidInfo. It’s in alpha test and the general idea is a central (I’m loathe to use the word “cloud”) database populated by objects. That’s objects as in things rather than the object oriented programming concept of object. Objects are tagged by users, and those tags define information and relationships between objects.

Each user can build up and tear down their selection of available tags, arranging these tags in a series of namespaces. For example, I have a namespace defined for an application I’m building:

metaljoe/music

That namespace consists of my root namespace (me, my user name) with a child namespace entitled music. There are no rules on how people structure their namespaces – I could easily call it metaljoe/aural or metaljoe/rhubarb if I wanted. As FluidDB evolves, conventions will naturally arise as people, hopefully, find certain naming styles preferable or popular. For the moment, anything goes.

First, I need to instantiate and bind my proxy to FluidDB. Using The excellent FOM library for Python: (please note that I am using the FluidDB sandbox rather than the live database)

>>> from fom.session import Fluid
>>> fluid = Fluid( "http://sandbox.fluidinfo.com" )
>>> fluid.db.client.login( "metaljoe", <password> )
>>> fluid.bind()

I then retrieve my user’s namespace:

>>> from fom.mapping import Namespace
>>> namespace = Namespace( u"metaljoe" )

To retrieve the list of child namespaces as either “leaf” names or full paths:

>>> namespace.namespace_names
['music']
>>> namespace.namespace_paths
[u'metaljoe/music']

If I want to create a new namespace under the current one, I can do this…

>>> namespace.create_namespace( u"test", u"A test namespace" )
<fom.mapping.Namespace object at 0x743670>

…or if I decide to remove it:

>>> namespace.namespace_names
['music', 'test']
>>> namespace = Namespace( u"metaljoe/test" )
>>> namespace.delete()
<FluidResponse (204, 'text/html', None, '')>
>>> namespace = Namespace( u"metaljoe" )
>>> namespace.namespace_names
['music']

You’ll notice a little bit of FluidDB’s RESTful API shows through here, because we obtain a 204 (No Content) response code on a successful deletion.

As well as child namespaces, a namespace can have child tags:

>>> namespace = Namespace( u"metaljoe" )
>>> namespace.tag_names
[]

Hmm, okay, let’s find a better example:

>>> namespace = Namespace( u"metaljoe/music" )
>>> namespace.tag_names
['related_bands', 'source_url', 'band_name']

Or if I want the full path for the tags:

>>> namespace.tag_paths
[u'metaljoe/music/related_bands', u'metaljoe/music/source_url', u'metaljoe/music/band_name']

I’ll cover tags another time.

So there we have it, a real quick tour of FluidDB namespaces – add, delete and view namespaces, and find out what child namespaces and tags are present.