Pondering Patterns

It’s been something like 11 or 12 years since I first heard of the book “Design Patterns: Elements of Reusable Object-Oriented Software”. It was around the time I was beginning to realise there might be something to this idea of object-oriented software development.

I’ve been pondering patterns ever since.

For those not aware of patterns (where have you been?), the idea stems from architecture of the physical building kind. People realised that there was a set of principles that had formed based on common solutions to common problems. You don’t stick a door there, you don’t put a window here, you need X number of columns for this, you need Y whenever you have a Z. You can probably tell why I never became an architect… but you get the idea. It encapsulated and documented a collective knowledge and experience of the process of architecture.

Humans have been building houses, fortresses and temples for thousands of years. That’s a lot of experience and knowledge we’ve accumulated about what we should and shouldn’t do. But when it comes to building software, we’ve only got a few decades under our belts. Much as we’d love to think of it as a mature and sophisticated process, it isn’t. When I left university, I was convinced software development was an engineering process – but the real world taught me that it just doesn’t fit well. It’s not a science either, despite my computer science degree. As a trained artist, I’d like to say it’s an art but that’s a simplification. And then someone mentioned a craft, and that is currently the best way to sum up what I do for a living. But even that isn’t a perfect analogy.

Anyway, I’ve diverged a bit. The “Gang of Four” made an attempt to encapsulate a set of knowledge and experience from software development in the same way architects had done, and that knowledge is present in their book “Design Patterns”. It’s a fascinating book to have on your book shelf and, in my mind, it presents two very important bits of information to programmers: a selection of solved problems and a vocabulary for programmers.

Solved problems are good. There’s no point wasting time re-inventing the wheel unless you can build a much, much better one. The Internet has given us a rich, supplemental way to seek out solutions to problems or advice on how best to approach common situations. The growth of free and open source software gives developers access to the real shared knowledge and literature of programming: the source code. We lead busy lives and the business wants results yesterday, so knowledge of what is a solved problem is good to have because it’s one less thing to worry about and we can concentrate on the other stuff.

The vocabulary aspect is one that often gets overlooked by people talking about patterns. Even before I discovered the ideas behind domain driven design, I’d been interested in the idea of a “ubiquitous language”, a shared glossary between developers, domain experts and users that would allow all three to communicate effectively. Too many software projects fail because of the lack of communication or, perhaps worse, misinterpreted communication. While patterns might not hold to all three types of people mentioned, they do provide a shared terminology for developers to talk to each other about common solutions. At least your developers all understand each other, right?

So patterns are a good thing then? Well, yes and no. “Design Patterns” is a fascinating book to have on your shelf, and that’s ultimately where it resides. I actually read it all the way through just after I bought it and went “great!”. I dipped into it maybe two or three times in the space of a few months afterwards and that was it. It now sits there nestled between “Patterns of Enterprise Application Architecture” and “Refactoring”. Don’t ask about my elaborate book ordering system.

So what’s wrong with patterns? Well, nothing intrinsically. As I said earlier, it provides a repository of shared knowledge/experience in solved problems, and it encourages a shared language between developers. Both of which are good. Trust me.

Let’s take the shared language idea first as it’s a bit of a flimsy argument. Most, but definitely not all, developers I’ve worked with either haven’t heard of patterns or don’t know them off by heart. I know I don’t remember all the names. That’s mainly an education issue, rather than a problem with patterns, but it puts a barrier between people – it can even be taken as a mild form of elitism. You can shove your Flyweight Pattern up your Chain of Responsibility, they say. Maybe.

Even amongst those of us “in the know”, a problem is rarely a pure example of one of the patterns so we end up adding extensions to what we’re saying. In fact, it’s often more comfortable slipping from pure Patternspeak to Domainspeak. It’s not a Command pattern, it’s an Order Request or a Unicorn Disposal. Or we go low-level and discuss language features. The Patternspeak abstraction sits in a middle ground between domain language (understandable by the business) and implementation language (understandable by geeks and compilers).

Defining language and process has an unusual side-effect in that it sets an, often arbitrary, limit on creativity. Much as there is still a school of thought that programming can be defined without the need for creativity, that is nonsense – it’s problem-solving, and problem-solving often cannot be described by a rigid formula or process. I laugh when I hear about ideas such as executable UML, because computer-assisted software engineering was The Next Big Thing when I was at university and we’re still waiting for it to deliver.

The risk with patterns is that developers can become restricted by them – they follow the pattern precisely, or refuse to see options beyond the documented patterns. Again, that’s not a problem with patterns as such, but the way they are used. Actually, it’s a limitation of the human mind – we tend to stay within our comfort zone and often stick to what we know, even when that knowledge doesn’t seem to be beneficial in the circumstances. We become blinkered.

Recently I heard mention of “refactoring to patterns”. I think it was on an episode of DotNetRocks, but I might be wrong. You write your code free from preconceived ideas. Strictly speaking, you write code confined by the limitations of your own experiences and knowledge (or that of your team). Actually, you write code based on the real problem at hand. Then you step back and realise you have a Facade, Memento, Syntax-Directed Translator, Service Stub or whatever, which may or may not be a help when refactoring your work to be cleaner, more elegant or more understandable. Or when discussing the problem with an “in-the-know” colleague.

The other thing I heard a couple of years ago was that patterns are language-dependent – and I’ll chip in the phrase “paradigm-dependent” too. Interesting idea. The clue was the Gang of Four’s subtitle “Elements of Reusable Object-Oriented Software”. Object-oriented development might have the mindshare these days, but C is still one of the most widespread languages and functional languages like Haskell, Erlang, Clojure and F# are rapidly becoming the cool kids on the block. The Gang’s patterns are based on their knowledge and experience of statically-typed, object-oriented languages.

Something that could be a pattern in C++ or Java, might be a simple bit of syntax in something like Python or Ruby. How do object-oriented patterns map to functional or procedural languages? What about for dynamically typed languages? Of course, they have their own patterns – I’ve seen Erlang patterns crop up on the mailing lists, decades of Lisp and Scheme pattern seep into Clojure, and even a minority of fellow Pythonistas claiming patterns don’t exist in Python. Wishful thinking guys – we just have different patterns. When put like that, the shared vocabulary and knowledge provided by patterns becomes more disperse – with maybe a tiny shared core somewhere in the murk.

In conclusion, reviewing my meandering and rambling, patterns are still a good thing because of the shared language and wisdom of solved problems. But they can be troublesome too, if used inappropriately. My copy of the book has “An Invitation” at the end which essentially says patterns are something you build up yourself, with those in the book as a starting point. It was something I’d forgotten about until I dusted off the book to sit beside me as I wrote this. I wonder how many others have forgotten this important piece of advice – or even missed it completely?

Think about the patterns in your own domain, your own choice of technology. Then think about the language you use to describe things to your fellow developers, your domain experts and your customers – are you all understanding each other? Really?

Things to look at in 2011

Before I start on my 2011 list, it’s time for a quick review. My list for 2010 was:

• Clojure (http://clojure.org/)
• CouchDB (http://couchdb.apache.org/)
• Natural Language Toolkit (NLTK) (http://www.nltk.org/)
• PyGame (http://www.pygame.org/news.html)
• Twisted (http://twistedmatrix.com/trac/)
• XMPP (http://xmpp.org/)

I spent the year going to the London Clojure Dojo and I must admit that I do like the language and will continue to learn it. I’ve not had much reason to use it outside of the dojo, but I’m sure something will come up this year.

CouchDB is very impressive and CouchApp makes it pretty easy to develop self-hosted apps in CouchDB. I have an idea for a CouchDB app which I want to work on.

On the PyGame front, I did a fair bit of development early in the year, turning a game idea into… well, a mess. I started off a little too ambitiously and the unfinished result needs a serious bit of refactoring in order to progress further. I’ve got a simpler idea in the same style that I want to work on this year, which should give me a better idea about how to rework and finish the original game.

I spectacularly failed to look at XMPP (other than read the O’Reilly book), Twisted or NLTK (again, other than read the book).

So, for this year:

• PHP (http://www.php.net/)
• Python 3 (http://www.python.org/)
• Celery / RabbitMQ (http://celeryproject.org/ / http://www.rabbitmq.com/)
• XMPP (http://xmpp.org/)
• jQuery (http://jquery.com/)
• Android SDK (http://www.android.com/)

So, PHP is my new language for the year. What gives? Well, the company I work for uses a lot of PHP code. Even though I’ve been recruited for helping migrate PHP and Perl on the back end to Python, the front end  PHP code is not going away and it would be useful to be able to roll up my sleeves and help with the maintenance and development work. Unlike Erlang or Clojure, this is just a straight case of learning the syntax, and not really the same challenge of those languages.

Ah, Python 3. The company roadmap is to finish 2011 using Python 3 in production. This is, admittedly, a bit ambitious because not all the Python code we use will be Python 3 compatible (Django springs to mind). I’ve made the personal decision that the final release of Python 3.2 will signal the transition point for my home projects, at least taking into account library support.

I’ve already begun to look at Celery from a very basic point-of-view, but this is a technology we are going to be using more heavily at work this year. It’s actually one of the many things that attracted me to joining the company late last year – because I was never going to get a chance to use this commercially at my last place.

XMPP is still on the list and the London Python Dojo will be using it this year for the inter-dojo game challenge. Looking forward to it!

jQuery is something that was hidden on my 2010 list. I’m actually seeking to dust-off and improve my JavaScript knowledge this year and jQuery would be a useful addition to my toolbelt.

Finally, time to upgrade my ancient phone to one of these new-fangled “smart” phones. I had a company iPhone for a few months, but never really used it. It’s a nice enough piece of hardware and the UI is slick as you would expect… but it did nothing for me. The Android OS is open source and seems more in tune with what I might use for a phone OS. Although I would rather like an Objective-C SDK for it Actually, it looks like Clojure and Jython are unofficially supported, albeit rather slow.

As well as the new skills above, I’ll be continuing with Clojure, PyGame and CouchDB. I’ll also be getting some commercial Django experience, continuing to explore FluidDB (FluidInYourEar might finally get a public release this year!), the Flask web framework, and hopefully refresh my Erlang skills.

Here’s to 2011 – what technologies are you planning to look at this year?

 

EuroPython 2010

Well, I’m back from another great EuroPython – my thanks to John Pinner and his EuroPython crew, the presenters, Josette from O’Reilly, and all my fellow delegates. Regrettably, I might not be able to make the next EuroPython in the wonderful city of Florence, but I might be able to make it to PyCon AU in Sydney next year. Fingers crossed on that one – I might even pluck up the courage to do a talk!

Monday

Monday’s talks kicked off with Raymond Hettinger’s “Idiomatic Python”. He guaranteed everyone would learn something new and he definitely delivered. I walked away with some interesting gems of Pythonic knowledge, from basic stuff like enumerate() through to more advancing knowledge like the surprising behaviour of super().

Ezio Melotti followed after the break with a useful overview of the Python development process, which inspired me to join the Python Core sprint on Friday. I’ve already contributed my first couple of patches to Python 3.2 – the first of which has been committed to trunk already.

“How Import Works” was a run through by Brett Cannon on how, you guessed it, Python imports code. Although you’ll probably never need to modify the standard import process, it’s fascinating to discover how it all works. Best of all, new utilities in Python 3 make it much easier to customise things safely if you ever need to.

“PostgreSQL’s Python Soup” was a little disappointing, with a good chunk of the talk being a general one about relational databases. The interesting stuff about connecting to PostgreSQL came quite near the end when we were introduced to the many different connectors. And I mean many.

Monstrum is a new HTTP functional testing framework for Python 3, and was detailed in “Testing HTTP Apps with Python3″. It looks very cool, and the logo would make a great t-shirt! Continuing the web testing theme, Raymond Hettinger followed up with “Python & Selenium Testing”. Alongside an overview of Selenium, Raymond introduced Sauce Labs, a company providing commercial support for Selenium, the web testing framework, via their cloud service.

To round off the day, I attended two talks by Richard Watts and Tony Ibbs of Kynesim who presented Muddle, their open source build system which looks very cool, and KBUS which is an elegant and lightweight messaging system implemented as a Linux kernel extension.

Tuesday

The day’s talks began with a rather packed presentation by Guido on AppStats, an AppEngine monitoring tool. I must admit that I did partially go to hear him talk, as did probably a lot of other people, but also because I was interested to learn more about AppEngine. I picked up some interesting bits of information about AppEngine internals, and the tool looks fantastic. However, I did feel a bit out of my depth.

Bart Demeulenaere’s “Pyradiso Rapid Software Development” was a call to arms, a request and discussion on how to create a rapid application development framework in Python that supports the multicore world out-of-the-box.

Engaging both my computer science and art historical sides, I then attended Richard Barrett-Small’s “The Trojan Snake”. Richard works for the Victoria & Albert Museum, and has been instrumental in introducing Python and Django as a way to clean-up and replace a mess of bespoke PHP, ASP.NET and Java. An interesting look at how an organisation faced issues with bespoke applications in a variety of technologies, and found Python to be a flexible and effective solution. It’s a shame more people didn’t attend.

I was an avid reader of Michele Simionato’s blog posts, covering his experiences with Scheme, so it was great to hear him speak at EuroPython with his talk entitled “Intro to Functional Programming”. Functional programming is receiving a surge of interest amongst programmers, thanks to the search for better ways to deal with concurrency, so it was good for someone to cover other reasons to start exploring FP.

“Tickery, Pyjamas & FluidDB” by Terry Jones was a run through of Tickery, a FluidDB application for analysing Twitter friend sets. Regular readers of this blog will know that I’ve been interested in FluidDB since attending an introductory talk by Nicholas Tollervey. Definitely worth a look.

Rounding out the end of the day were Andrew Godwin’s “Fun with Databases and Django” and Russel Winder’s “Pythonic Parallelism with CSP”. Andrew’s talk provided a general overview of some of the more advanced features of the Django ORM, including Django’s interaction with non-relational databases such as MongoDB and CouchDB. Meanwhile, Russel Winder is one of the key players in pushing Python support for concurrent and parallel programming. CSP has been around for over 30 years, and now Python programmers can begin to take advantage of the ideas thanks to projects like Python-CSP and PyCSP.

Wednesday

Wednesday kicked off with Nicholas Tollervey’s “Organise a Python Code Dojo!”. Nicholas shared his experiences running the London Code Dojo – the joys, the successes… and the things that didn’t quite go to plan.

“HotPy – A comparison” was Mark Shannon’s comparison between three different VMs for Python: PyPy, Unladen Swallow and his own HotPy project which runs on top of the GVMT.

Henrik Vendelbo gave two short talks in succession. The first was “Real Time Websites with Python” which gave a simple example of the Tornado web server, released as open source by FriendFeed. The second was “Custom Runtimes with PyPy”, a very interesting talk on using PyPy to provide an easy way to bundle up a custom Python application as a self-contained app.

qooxdoo is a full-featured JavaScript framework for developing Rich Internet Applications. It’s quite an impressive piece of work, not only because of the sheer size of the code base. Managing large code bases can be a headache, especially if you also need to process that source to produce more compact, optimised versions for delivery to specific browsers. In “Python for Javascript Apps”, Thomas Herchenröder explained how they use a set of Python tools to make dealing with the code a much more pleasant experience.

“A Python’s DNA” by Erik Groeneveld illustrated an issue I’ve hit myself: how to configure a complex component-based system in a simple, maintainable way. The answer Erik came up with for Meresco is not to use XML or similar to express configurations, but use Python instead to provide a rich, flexible, and elegant way to define dependencies, data flows and configurations.

It’s not often a talk begins by telling you that the subject of the talk is now officially dead, and a new project has taken over. Kay Schlühr’s “The Trails of EasyExtend” quickly changed to “The Trails of LangScape”. LangScape evolved from a toolkit for writing language extensions to Python, but takes a much broader view with the aim of allowing easy ways to extend many different languages.

Thursday

The morning began with “Building a python web app”. Anthony Theocharis and Nathan Wright took us through their journey to find the right Python web framework to implement MediaCore. They discussed Django, TurboGears and Pylons, illustrating their experiences with each and why they settled on Pylons. The different approaches to database access and templating languages were also examined in brief, followed by how they open sourced the project. An entertaining and interesting talk, refreshing by the fact they apparently came from a non-Python background so lacked any preconceptions.

Next up, Denis Bilenko introduced gevent in his talk “gevent network library”. gevent is a network library for handling large numbers of connections efficiently and, more importantly, elegantly. Optional, seamless monkey patching of standard library modules makes it very easy to work with existing code and libraries. I also discovered Gunicorn a Python implementation of Ruby’s Unicorn webserver.

Finally, my selection of conference talks closed with “Arduino and Python” by Michael Sparks. The talk should probably have been renamed “How not to blow up your computer” as Michael crammed in a quick intro to electricity and electronics into a few slides to dispel concerns about inexpensive pieces of electronics destroying your computer. The talk was illustrated by his prototype of a rather interesting TV remote control developed at a BBC R&D workshop, built with an Arduino… and Python of course.

Friday

Big thanks to Brett Cannon and Richard Jones for putting up with me all day while I got to grips with downloading, building, testing and patching Python 3.2. It was a really enjoyable day, and very satisfying to have a patch committed to the Python source code – no matter how small.

FluidDB Objects in 10 Minutes

When looking at FluidDB‘s tags (see FluidDB Tags in 10 Minutes), we briefly touched upon FOM‘s Object class (not to be confused with Python’s native object class). I instantiated an object representing the planet Mars:

>>> from fom.mapping import Object
>>> mars = Object(about="planet:Mars")
>>> mars
<Object planet:Mars>

The thing to note is that I supplied the about keyword. By instantiating this way, I have no way of knowing if it existed before or not – the appropriate FluidDB object is created if one doesn’t exist already. In the big scheme of things it probably doesn’t matter, but it’s important to be aware.

The about keyword sets the value of a useful tag that is often assigned to objects when created: fluiddb/about. The about tag is used to label an object with something descriptive. It can be anything, but certain conventions are slowly evolving amongst FluidDB developers and users.

For example, an application I am developing uses the convention “band:<band name>” to label an object as representing a particular musical band. Nicholas J Radcliffe has written more on the subject on his About Tag blog which is a recommended read.

Let’s take a look at that tag:

>>> mars.get("fluiddb/about")
('planet:Mars', 'application/vnd.fluiddb.value+json')

An optional about tag is very useful but not all objects have, or indeed need, one. Fortunately, we can also refer to any object by its unique GUID which is guaranteed to exist:

>>> mars.uid
'276da99e-c8d9-42c9-99ae-3db69a5e9ef0'

If I wanted to load an object by its GUID, I simply supply the uid parameter (it’s a good job I know the ID!):

>>> red_planet = Object( uid='276da99e-c8d9-42c9-99ae-3db69a5e9ef0' )
>>> red_planet
<Object planet:Mars>
>>> red_planet.get("fluiddb/about")
('planet:Mars', 'application/vnd.fluiddb.value+json')

Tags form the real power of FluidDB, so it’s important to be able to determine what tags have been added to an object. Let’s take our Mars example first:

>>> red_planet.tag_paths
['metaljoe/foo', 'fluiddb/about']
>>> red_planet.tags
[<fom.mapping.Tag object at 0x57edb0>, <fom.mapping.Tag object at 0x57fd10>]

The first attribute stores the tag names, while the second returns these tags instantiated as FOM Tag objects. We can check they are indeed proper Tag instances by peeking at the descriptions:

>>> [ tag.description for tag in red_planet.tags ]
['behold: the foo tag', 'A description of what an object is about.']

I can also test for the presence of a tag by using the has() method:

>>> red_planet.has( "metaljoe/foo" )
True
>>> red_planet.has( "the_mekon/likes" )
False

URLs were briefly mentioned in an earlier post about tags – each tag has a corresponding URL. FluidDB’s RESTful API means each object in the system also has a unique URL. For example, on the FluidDB sandbox instance our planet:Mars object can be referenced by the URL http://sandbox.fluidinfo.com/objects/276da99e-c8d9-42c9-99ae-3db69a5e9ef0 – pointing your browser to that URL returns a little bit of JSON:

{"tagPaths": ["metaljoe\/foo", "fluiddb\/about"]}

An object’s tags can also be referenced via URLs. For example, the fluiddb/about tag is referenced by the following URL:

http://sandbox.fluidinfo.com/objects/276da99e-c8d9-42c9-99ae-3db69a5e9ef0/fluiddb/about

Which yields the contents of the tag for that object:

"planet:Mars"

Note that a browser will be making GET requests to these resources. Being RESTful, we can make full use of different HTTP methods (GET, PUT, POST, DELETE) to perform reading, creating, updating and deleting of our FluidDB resources, but that’s a subject for another time.

An Agile Reading List

I love books. Proper, paper books. I’m always recommending titles to people, so thought it might be a good idea to pluck ten books from my bookshelves that I regularly recommend to anyone who wants to learn about Agile development, or improve their skills.

Now, please note that my background is with XP as that is the process that has worked the best for myself and for the teams I have worked with. As such, recommendation number one is:

Extreme Programming Explained (2000) – Kent Beck

This is the book that kicked it all off for me back in 2000, and it still remains the most concise and readable introduction to XP. It’s also an important text for anyone wishing to understand what it means to be “Agile”.

User Stories Applied (2004) – Mike Cohn

Human society has used stories for a very long time to convey information, yet writing user stories is often not as easy as it should be. This is one of those books that I ummed and ahhed about buying: what could a book on such a “small” subject really offer me? The answer was quite a lot, because there’s a great deal involved in learning to write effective stories. More than that, writing stories is just the first step – what we do with those stories is an important part often glossed over.

Agile Estimating and Planning (2006) – Mike Cohn

In many ways, this leads on from User Stories Applied. How do we turn a stack of stories into a tangible piece of software that delivers value to the customer? The whole spectrum of estimation, prioritisation and planning is given coverage, explaining not just how we do things, but why we do things.

Domain-Driven Design (2004) – Eric Evans

Anyone who says that Agile doesn’t do design is talking nonsense. Unfortunately, the lack of books on the subject of Agile design doesn’t help to dispel this myth. Domain-Driven Design is a much needed addition to the Agile library. Two core concepts, the use of common language and a strong model, are presented and elaborated upon to form a simple, elegant, yet powerful framework for designing large systems. My only negative point is that the book could probably be condensed into one half the size without losing the clarity of vision.

Refactoring (1999) – Martin Fowler

I picked up this book not long after reading Extreme Programming Explained. This book was very influential on me, because it made me realise we don’t have to be satisfied with the final implemented solution to a problem – people who haven’t read the book often raise an eyebrow when I mention things like “code smells”. Designs and code go stale more often than we like to admit, and this book provides a readable, in-depth guide to tackling the overhaul of code in order to improve the design, readability and maintainability. Those of us who use dynamic languages in our large-scale projects might find some of the refactorings unimportant, but the concepts are still invaluable.

Test-Driven Development (2003) – David Astels

To refactor code safely, you need to have an excellent test suite to ensure you don’t break anything or introduce (or even fix!) unexpected side-effects. This book is an excellent introduction to the subject – very clear and concise, yet often overlooked in favour of Kent Beck’s book. The bulk of the book uses examples with Java and JUnit, but covers other xUnit implementations for a variety of languages. The choice of language is unimportant as the concepts are equally valid.

Agile Testing (2009) – Lisa Crispin / Janet Gregory

An impulse purchase from Amazon that ended up being an essential book in my library. Although aimed more towards a QA team nervously wondering if they’re obsolete in the New Agile Order, this book is an incredibly good resource for anyone involved in a software project. Every aspect of testing is covering in a very approachable and comprehensive style, including the different roles testing plays and how testers can incorporate their role as part of an Agile team. Developers needs to remember that there’s more to testing than their own TDD test suites, and that’s where an Agile QA team comes into play.

Continuous Integration (2007) – Paul Duvall / Steve Matyas / Andrew Glover

There are quite a few good books on continuous integration, but this one manages to provide both a decent introduction to the subject and a good balance between theory and practice. Although I’d used CI long before I’d read this book, I must admit I’d never really needed more than a superficial knowledge. This book helped fill in some massive gaps in my knowledge and understanding.

Agile Retrospectives (2006) – Esther Derby / Diane Larsen

This book covers a much-neglected part of the Agile process: reflection and self-improvement of teams. Regular retrospectives to give teams time to identify where things are going well, where things are not going well and seek out ways to continually improve themselves. The book introduces the concepts behind retrospectives, gives a framework for running them, and presents various supporting exercises that can be used.

Principles of Software Engineering Management (1988) – Tom Gilb

This might seem an unusual one to finish with. Why is it even on an Agile reading list? Because it’s easy to forget that “being Agile” is not new, or a fad, or sprung up suddenly. This book covers evolutionary delivery, estimating and dealing with risk, planning and code inspection. Although written for a “traditional” software engineering audience it reads in places more like a proto-Agile manifesto, a genuine reflection on the discipline and where it was heading at the end of the 80s. I really wish I’d discovered this book at university, rather than ten years later, because it would’ve changed my approach to software engineering outside of academia. Even reading twenty years after it was published, it still offered a very relevant text.

Game Progress: 1

I’ve mentioned in a couple of previous posts about a wargame idea I’ve had swishing around my brain for a while now. The good news is that I finally made a start.

The main problem with getting started was, simply, getting started. No matter how many times I sat down to code, I ended up getting distracted or hitting coder’s block and staring at a blank source file in Aquamacs. Evenings aren’t my best time for programming, especially after a tiring day, which doesn’t help much. The other problem was that the overall task looked both large and hazy. A bit like most projects, except this time the customer is me and the deadline is “whenever”.

The game has three main aspects: historical and non-historical unit availability and deployment, unit combat and movement, political opinion and events. The first area was already the subject of an earlier prototype, of which the only thing I have retained are the unit availability data files, and seemed like a good starting point.

Given the coder’s block, I went back to basics and wrote out a series of bullet points on things needed for this part of the game. After struggling to get a couple of bullets down, things began to flow and I racked up quite a good list of functionality. Some of the ideas have been with me since the very first time I thought about writing this game (a couple of years ago!), while others were fresh. Although I reviewed the list to weed out any weak ideas, I ended up sticking with the list as-is – give or take a few tweaks of terminology to keep things consistent.

Now, the obvious starting point would’ve been to start writing some PyGame code straight away. Much as I’m a visual person, my lack of experience of PyGame encouraged me opt to hold back and take a different approach. My main worry was that I would bog myself down in PyGame, without really considering what I was going to achieve. Looking at my list of tasks, I started to model some of the data required and began to pick and implement some basic functionality using test-driven development. By isolating these models from any user presentation, and from writing any UI code before I had gained some ground learning PyGame, I found myself gaining a much better appreciation for what I was aiming to implement overall.

As I wrote little bits of functionality, I refined my existing task list and design until I reached the critical point: I needed to start on a UI. My first thought was that I should prompt the user for a screen resolution and so on, but then I realised I could hard-code it to my MacBook’s screen dimensions. There was a tinge of guilt, but it was the simplest option at the time to get something working and I’m the only customer so far, okay? After a few iterations of work, I now have a screen displaying three different lists (deployed units, units available for deployment, and units that are likely to become available soon). I can increment the game turn, each one representing a week of real time, and can watch units switch between the various views as time progresses and forces arrive and depart the theatre of operation.

As a game, it currently sucks: it’s not so much a game as a rolling chronology of the order of battle. But that’s okay because it’s barely a fraction of the final vision and the current process is geared at building a suitable foundation for the rest of the game. I’m not thinking “game” just yet, because most of the real gameplay happens in the other two aspects. The current UI itself is temporary, because the different lists will not be presented in their current format later on. The deployed units, for example, will be shown on a game map as NATO-style unit symbols, not a list.

The great thing about tackling little chunks is the motivational factor. Never has the term “low hanging fruit” been more apt – by searching the to do list for small tasks, I can fit coding into spare moments. The sense of progress by chipping away at the tasks is a great motivator. Periodically I refactor the code to keep things fresh, to remove code smells and ugliness that I feel guilty about, and generally improve the design based on my experience of the project so far.

Anyway, that’s my little reflection on the work so far as I’m too tired to write any Python tonight. After hitting some more internal milestones, I want to have a public milestone of some screenshots of the first game aspect. Since I’m the customer and don’t have a deadline, I can’t say when that will be… but hopefully in the next few weeks.

Game Idea and Motivation

I mentioned in a previous post about exploring PyGame because I want to write a wargame in Python. I’ve had the idea floating around for a while and even wrote a whole load of story cards and some quick and dirty Python code to make a start. The main issues have been time and motivation. Time was, and still is, limited and my motivation was poor when I made my initial stab.

One big problem I have is that I’m very much a a daytime coder. When I get back home in the evening, I’m not a late night coder – it’s not my peak mental time and code I’ve written late at night often shows. I’d rather get up early morning and write code. I always used to joke that my peak time was 8:30am to 10:30am, but it’s partially true. I just feel fresher and more focused during the day. At weekends, other commitments eat heavily into my time which takes its toll on when I can code and how motivated I am.

But I really need to make progress.

After the hassles of a recent house move and the festive season, and a relentless deluge of projects at work, I really need to do something fun and interesting. I really want to get my game idea coded and released as an open source project, in the hope it is of use and interest to others. But I need to do at least the bulk of it myself first, for my own personal reasons.

I’ve decided to start from scratch, well mostly. None of that horrible old code, none of those story cards. A fresh start, using the ideas I’ve had sloshing around my brain for the last couple of years. The lack of time and frequent lack of energy is going to be focused into a positive thing: an incentive to keep things simple, concentrate on little tasks, set regular milestones to keep things progressing.

I’m not going to go into too much detail at the moment, but maybe I will in future posts. It’ll probably be easy to guess though. The setting is a historical conflict of the 20th Century, and the game will follow a broadly historical chronology with potential for divergence. It’s not intended to be an accurate simulation, but give a flavour of various aspects of the conflict. It will be aimed at solitaire play and the game should be hackable and, hopefully, flexible enough for people to adapt.

There are a few broad areas of the gameplay to deal with. The historical arrival and departure of forces, which will be affected by player decisions and external influences, will form my first major goal – since I have already assembled some data and even wrote a prototype last year. Unit deployment and movement will form the next major area and follows on from the arrival and departure of forces. This will include rendering of the game map, units and fog-of-war. After that is the handling of combat and other actions, including the effects of morale, experience, fatigue and supply. Finally will be the effects of politics, public opinion and events – both historical and non-historical. You get the idea.

I’ll post up my experiences as I code the game: to consolidate my own learning experience, gather feedback and hopefully offer some hints and tips for others working on similar projects.

Why I’m Learning Clojure

Recently, I posed the question of which language I should learn in 2010. The options were Squeak, Common Lisp, Clojure, Groovy… or to make a second attempt with Ruby. I must admit I was veering towards Lisp when I wrote it, but all options were still possibilities. Out of the two Lisp dialects, Common Lisp was probably my preference over Clojure but in the end I decided I would give the latter a shot.

So why Clojure?

Well, the first reason is down to it being a new language. There’s something quite interesting about trying a language in the early days. Since I’m not going to be writing production code with it for the foreseeable future, having a moving target isn’t really an issue. The community surrounding the language is still forming and there’s a chance to watch and interact with a forming and evolving Clojure ecosystem. London also has a Clojure code dojo running, aimed at introducing people to Clojure, which offers an excellent opportunity to help with my learning – and best of all, there’s an overlap with some members of the London Python community. Very handy.

Secondly, Clojure is a Lisp dialect. Lisp has been on my list of languages to learn for a long time, but I’ve always ended up looking elsewhere and my only real experiences with the Lisp world have been trivial at best, and never with “the real thing”. In some ways I guess I’m still following that path, though Clojure feels more like a proper Lisp because at heart it is.

Clojure’s primary environment is of the JVM, an area I’ve only limited experience of. There was a time when Java was appealing to me, but every time I take a second/third/umpteenth look at the language it seems to have accumulated more baggage and bloat. It’s a huge and complex language and ecosystem to get into these days, though very mature and powerful. Clojure would give me an opportunity to delve into the JVM world a bit more.

The language is also making progress in supporting the .NET world. Much as I’m definitely not a Microsoft fan, .NET has grown on me and there’s some interesting work happening with it and the open source Mono implementation. I actually rather like C#, though wish someone would take the core language and move it out of the .NET world – probably a weird idea to some. Anyway, Clojure on the CLR is an interesting idea, so adds a little to the interest because I have to mix things up with .NET developers at work.

So that’s all the fluffy stuff out of the way – what are the other reasons?

Well, Clojure is a functional programming language, which is a current interest of mine. Spurred on by my adventures with Erlang, functional programming is growing on me. It’s also dynamically typed, which has been one of the big revelations during my career, perhaps even more than object-oriented development. It compiles to JVM bytecode; being Lisp, it has read/eval/print which makes language exploration and prototyping a joy; and I can delve into a language with a proper, powerful macro capability.

Clojure is also geared up for the big issue of the now: concurrency. Erlang interested me because of the way it tackled concurrency, something that all programmers are going to have to face up to sooner or later as we live in a multicore world. I like Erlang’s approach a lot, but there are other options out there and I feel it is my duty to examine a few of the options. In particular, the use of Software Transactional Memory (STM) holds an appeal to me. Having used databases the last ten years, transactions feel quite a natural approach for many problems, but definitely not all, and incorporating them into other software domains can only be a good idea. I was considering Haskell at one point to investigate STM, but I must admit to not being a massive fan of parts of Haskell’s syntax so never pursued the idea. With Clojure, I can at last take the chance to delve into STM.

I’m not going to stop learning and exploring Erlang, far from it, but I’m looking forward to adding Clojure to my programming toolkit.

Language for 2010

I’ve been a computer language geek for a long time. Even before I read the excellent book “The Pragmatic Programmer”, I would try and learn a new language every year or two – or at least look at new languages to get ideas. This year I added C# to my repertoire and made a return to learning Erlang, which was supposed to be my original choice for the year until the demands of needing to deal with .NET forced a more pragmatic change.

Even if you never use a new language long-term, just taking a look helps sharpen skills, bring about new insights or even create a renewed appreciation for your current language of choice.

My choice for 2010 is quite tricky, so I need some help in deciding from you, dear reader. The short list is as follows:

Squeak – http://www.squeak.org/

My Smalltalk knowledge is all read and no write, so time I actually learnt the language properly and wrote some code because it looks like such a simple and elegant language. I don’t think knowing and enjoying some Objective-C quite counts, somehow!

Squeak feels like the right way for me to learn Smalltalk. I even downloaded the software a while back, but am ashamed to say I just haven’t installed it yet.

Common LISP – http://clisp.cons.org/

LISP has been on my list (no pun intended) for years. I’ve done a tiny bit of EMACS LISP programming and the Amiga Installer uses a LISP dialect, but neither are really sufficient for me to say “I’ve done LISP”. Eric Raymond (I believe) said LISP was probably the best programming language to learn that you will never use, and I think he’s right. LISP featured things like garbage collection and dynamic typing back in 1958, features that some might believe are relatively new additions to programming languages.

I will, however, say that I don’t agree with people like author Neal Stephenson who insist LISP is the only programming language that can be described as beautiful.

Clojure – http://clojure.org/
Groovy – http://groovy.codehaus.org/

I made a brief look at Java (1.1) in 1996 thanks to a foresighted university lecturer, and again in 2005 when I was unemployed after returning from a season snowboarding. The JVM is pretty cool, but Java has changed a lot since the early days and not always for the better in my opinion. Apart from those two little forays, I’ve avoided the Java ecosystem as an area of interest. Now, however, things are changing. Non-Java languages on the JVM are gaining ground, including Jython and JRuby, but also new languages like Clojure and Groovy are appearing and revitalising the Java world.

Groovy seems to have quite a bit of interest from people in the Python community and, from what I’ve seen, I can understand why. It simplifies a lot of the complexity / boilerplate of Java, adds the power of dynamic typing that smug Pythonistas such as myself take for granted, and the name is suitably non-Enterprisey for me to take notice.

Clojure, on the other hand, is a dialect of LISP for the JVM. This would allow me to look into the JVM ecosystem while learning LISP. I also note there is CLR / .NET support as well, which is interesting – though I doubt I will be able to use it to interact with legacy .NET code at work.

Ruby – http://www.ruby-lang.org/en/

Not strictly a new language for me, but I briefly looked at Ruby on two occasions (notice a Java-like pattern here?). The first time was in 2005 for a possible Ruby developer job. They couldn’t find Ruby programmers so were looking for people with Perl and Python experience – I hadn’t heard of Ruby at the time, but it sounded interesting so I had a quick play with it. The second time was in 2007 when I read the “RESTful Web Services” book.

I must admit, I found Ruby terribly disappointing on both occasions. Please don’t flame me! A language that blends elements of Perl and Python, with some LISP-like tweaks, should be pretty cool, but it just felt syntactically awkward in places. Despite my disappointment, I can see why some people like it and it’s making more of an appearance in places I wouldn’t expect a dynamic language to be mentioned. I should really give it another go.

Now your turn…

So, what language from the list do you suggest I look at and why?

Software Development… Inspired by Flat Pack Furniture?

I’ve recently moved into the first place of my own, so I’ve become well-versed in the culture of Ikea furniture and the art of flat-pack assemblies. Even though the home DIY genes do not flow through my family line, and my attempts at woodwork at school were poor to say the least, I must admit to getting a certain pleasure from building self-assembly furniture and the results are, in my humble opinion, pretty damn good considering.

So what has this to do with software development? Well, I’m always looking outside the IT world for programmer inspiration and I had a moment of bizarre clarity while screwing together the parts of a dining room chair. As you do.

We, as developers, are always (hopefully) trying to find ways to make our work easier and of better quality. In an ideal world, we seek elegance, simplicity and usefulness. Things should be repeatable consistently, they should do the task at hand well, and should be easy for our users to use. And our ivory towers should be furnished by inexpensive yet stylish items.

Documentation

My moment of clarity began with the Ikea instructions for the chair. They offer clear details on how to go through the instructions, what you will need and what to do if it goes wrong. So what? Well, they do this without words: clear, simple, multi-lingual instructions. From a set of simple pictures, I know what additional tools I will need, the parts I should have in the box, I know to take it slow, remain calm, and give my local store a call if things are missing or it all goes horribly wrong. All from pictures. Can you imagine having to a) write down all those steps and b) repeat in dozens of languages? Can you imagine how many more pages you would find in those instructions?

Each of the build steps are equally clear and pictorial. The minimal amount of information is used to illustrate precisely what is required at every step of the process. When writing documentation, particularly user documentation, we can learn a lot from this. It’s so easy to write a big blob of text, slap together some full-size screenshots and call that a doc. It probably makes sense to us, since we wrote it – but does it make sense to someone else? Can we strip down the information to the essential minimum? Can we make the document clear and unambiguous? Can we show rather than tell? Are we repeating ourselves? Think about these things next time you write user documentation, detail a build process or comment your code.

Make It Easy

Ease-of-use isn’t just something we apply to anything user-facing. We don’t have lots of spare time these days and much as it’s an interesting intellectual exercise to work through a lot of hassles to get something to build or install, I’m a huge fan of things that Just Work. There are few things more tedious than having to wade through shark-infested waters or find specific versions of libraries just to run a simple application or fix a bug in the source code of a third-party tool.

Ikea’s furniture makes it a doddle to assemble. Using a minimal set of tools, anyone who can grasp a screwdriver and bang in a nail (straight is optional) can build decent looking furniture. I know, I’ve been there. The barrier for entry is low, and they’ve designed the items in such a way that it’s difficult to go wrong: the clear instructions are coupled with parts that have obvious ways to fit. If you find yourself forcing parts in with brute force, or a screw is not tightening up correctly, or bits are mis-aligned it’s likely down to user error. Complicated parts, like the back of the aforementioned chair, come ready assembled so a DIY dolt like me doesn’t have to worry: someone else did the hard work.

Good Enough

Ikea keeps costs down by operating the Good Enough principle. I see similar bits cropping up in various products (good reuse), hidden parts have little or no finish compared to visible parts (why waste paint or varnish?), the materials are often good enough for the job at hand, and there’s a minimal theme to parts anyway. My coffee table cost 8 quid and has a mere five bits to it, yet looks great and is fully functional. It’s a classic example of a design that meets the minimal requirement to get the job done.

Professional pride often means we take longer than we should in order to do “a really good job” when really we should aim to be good enough. I’m not expecting that table to be handed down to future generations, and the designer already knew that. As their customer I wanted something cheap and available to rest a mug of drink on in the evening. I didn’t want to wait several months for a master craftsman to build me a one-of-a-kind at great expense, especially if it might not actually be what I really wanted.

When writing code, take a good look at what you’re implementing. Is it a prototype? A demo? Is it going to last a year? A decade? Are people okay if it is rough around the edges or is it a life-critical system that has to work no matter what? Use the appropriate amount of time and effort to meet the requirements, and no more. Don’t add features you or the customer may not need – the cost of my table would’ve gone up a lot if the designer had started adding drinks holders, a cactus pot plinth, customisable colour panels and so on just in case I might need them.

Support

Finally, comes support. No matter how busy and understaffed the employees were (sounds like my office), there was always someone available to offer a hand where possible. You could interrupt them shifting crates or obviously fretting about dealing with lots of problems, but they made the time to help where they could. No matter how much of an irritating or clueless customer I could be. The warehouse could be on fire or buried under an avalanche, but that’s not the customer’s problem – the customer is trying to find his five-part coffee table and that’s the only thing happening in their world right now. No matter how overworked we are, the people using our software remain important – we’re creating code to scratch their own itches, solve their problems, create software that is of use and interest to others. Without them, we’re writing for one and that’s not so fun. Well, to me maybe. Don’t forget to make time for them.

And Enjoy

I’m sure there are other things to take away from my furniture-driven experience, but you get the idea(s). So the next time you look around at something outside the programming world, have a think about what lessons you can apply from the experience to your software development. I certainly will never look at flat pack furniture in quite the same way again.