Software Team Leadership: Safety, Enjoyment, Learning

Back in 2004, I took a sabbatical from software development to pursue my passion for snowboarding. It was a decision that lead to a parallel “career” as a snowboard, and later ski, instructor. Aside from a few communication skills, I never really thought the two would intertwine. However, from instructing came an interest in coaching and mentoring, and from those came helpful ideas when I found myself more and more involved with a leadership role.

When I undertook my first training course, in the mountains of New South Wales, there was an important phrase I learnt: Safety, Fun and Learning. In the UK, we use the variation of Safety, Enjoyment and Learning – which gives the acronym SEL (“sell”). We’re selling a suitable framework for a pleasant and productive lesson experience. Each word leads from the next: it’s difficult to enjoy something if you’re not safe, it’s tricky to learn something if you’re not enjoying it.

Safety, Enjoyment, Learning can be applied to many things other than snowboarding or skiing. As a team lead, senior developer or software coach, we also have to be mindful of these three words whether we realise it or not. All three are required to make sure our team has a pleasant and productive experience developing software.

Let’s look at safety first. The risk of injury, avalanches, sunburn or hypothermia is pretty low in software development – if it isn’t, I’d strongly recommend moving to a different office. But safety is more than just about physical danger. I’ve worked with teams where there’s a heavy fear of failure, an atmosphere where making a mistake is a heinous crime. The problem is, people become cautious – they stay inside their comfort zone, they don’t learn or progress, their work remains static. It’s not productive at all – for the team or the customer.

Mistakes are okay. We all make them. The important thing is that people feel they can own up to making a mistake, without retribution, and then take responsibility for correcting that mistake. Mistakes are learning experiences – do them once, try not to make them again. People need to know they can put their hand up and admit they did something wrong safely, and that the team will pull together to help that person take responsibility and correct the problem.

Leading on from this is the knowledge that someone can always ask for help, without feeling they will lose respect or be ridiculed. We all have a moment of stupidity and forget something obvious, we all stare at a problem for hours because our brain has masked the critical (and obvious) cause of the problem. Asking for help is often one of the most difficult things for people to do, so we need to ensure people feel safe enough that they can stick up a hand without fear.

Enjoyment is next. I’ve been very fortunate during my working career that I’ve nearly always worked with teams where we can laugh, joke and gossip – yet still get our work done. On the rare occasions I’ve been in a team that doesn’t have that environment, you certainly notice the difference in productivity – no one really gets anything done, and certainly not to a good standard. A former boss of mine used to say that he didn’t like a quiet office, because it meant no one was working. The social interaction feeds the mood, raises the spirits even when tackling a Project From Hell, and leads to better communication and thus a higher productivity.

Work shouldn’t be a miserable experience, because it becomes a waste of time for everyone – if you don’t enjoy your job, dust off your CV and start looking for one you will enjoy. It was that kind of decision that resulted in me ending up on a mountain on the other side of the world, a decision that both changed my life in a big way. It allowed me to discover a whole new set of skills, but also help me rediscover my love of programming.

And finally learning. If you’re not learning new things each week in your job, something is wrong. It doesn’t have to be big things like a new programming language or framework – it could be a tiny optimisation, a better way of structuring your code, a new bit of business logic, an unusual hobby of the developer sat next to you. As human beings we need to keep growing and learning or we stagnate, and when we stagnate we become less productive. I know far too many programmers who know one programming language, one framework, one operating system. Some even get quite angry that they should learn something new! After all, BASIC and 6502 assembly should be all you could ever need…

I try to learn a new language every year or two: Erlang, PHP, C#, Clojure. Even if you don’t use them for any real world work, sometimes you can gain a new insight or a better way of doing things with your existing toolset. Erlang helped me with concurrency and parallelism, Clojure taught me to write more functionally, PHP makes me appreciate Python more than ever, I wish C# wasn’t tied to .NET/Mono. The definition of “learn” changes each time: I mainly aim to get a feel for the nuances and pros/cons of the language before deciding to stop or carry on.

Code dojos are great fun. Code katas provide useful exercises for developers. Regular retrospectives encourage developers to sit back and think about what they have or haven’t achieved and find new ways to do things. Pair programming provides knowledge transfer. All that gossip and laughter I mentioned earlier can lead to sudden revelations and new ideas. Teams should be mindful of new things, new techniques, making progress in their own skills and knowledge. Programmers should be encouraged to try new things, push their comfort zone wider and improve themselves.

Safety, enjoyment, learning.

Next time you’re with your team, take a good look and ask yourself if these three concepts are being employed successfully. If not, perhaps it’s time you started making some changes?

Have fun!

Pondering Patterns

It’s been something like 11 or 12 years since I first heard of the book “Design Patterns: Elements of Reusable Object-Oriented Software”. It was around the time I was beginning to realise there might be something to this idea of object-oriented software development.

I’ve been pondering patterns ever since.

For those not aware of patterns (where have you been?), the idea stems from architecture of the physical building kind. People realised that there was a set of principles that had formed based on common solutions to common problems. You don’t stick a door there, you don’t put a window here, you need X number of columns for this, you need Y whenever you have a Z. You can probably tell why I never became an architect… but you get the idea. It encapsulated and documented a collective knowledge and experience of the process of architecture.

Humans have been building houses, fortresses and temples for thousands of years. That’s a lot of experience and knowledge we’ve accumulated about what we should and shouldn’t do. But when it comes to building software, we’ve only got a few decades under our belts. Much as we’d love to think of it as a mature and sophisticated process, it isn’t. When I left university, I was convinced software development was an engineering process – but the real world taught me that it just doesn’t fit well. It’s not a science either, despite my computer science degree. As a trained artist, I’d like to say it’s an art but that’s a simplification. And then someone mentioned a craft, and that is currently the best way to sum up what I do for a living. But even that isn’t a perfect analogy.

Anyway, I’ve diverged a bit. The “Gang of Four” made an attempt to encapsulate a set of knowledge and experience from software development in the same way architects had done, and that knowledge is present in their book “Design Patterns”. It’s a fascinating book to have on your book shelf and, in my mind, it presents two very important bits of information to programmers: a selection of solved problems and a vocabulary for programmers.

Solved problems are good. There’s no point wasting time re-inventing the wheel unless you can build a much, much better one. The Internet has given us a rich, supplemental way to seek out solutions to problems or advice on how best to approach common situations. The growth of free and open source software gives developers access to the real shared knowledge and literature of programming: the source code. We lead busy lives and the business wants results yesterday, so knowledge of what is a solved problem is good to have because it’s one less thing to worry about and we can concentrate on the other stuff.

The vocabulary aspect is one that often gets overlooked by people talking about patterns. Even before I discovered the ideas behind domain driven design, I’d been interested in the idea of a “ubiquitous language”, a shared glossary between developers, domain experts and users that would allow all three to communicate effectively. Too many software projects fail because of the lack of communication or, perhaps worse, misinterpreted communication. While patterns might not hold to all three types of people mentioned, they do provide a shared terminology for developers to talk to each other about common solutions. At least your developers all understand each other, right?

So patterns are a good thing then? Well, yes and no. “Design Patterns” is a fascinating book to have on your shelf, and that’s ultimately where it resides. I actually read it all the way through just after I bought it and went “great!”. I dipped into it maybe two or three times in the space of a few months afterwards and that was it. It now sits there nestled between “Patterns of Enterprise Application Architecture” and “Refactoring”. Don’t ask about my elaborate book ordering system.

So what’s wrong with patterns? Well, nothing intrinsically. As I said earlier, it provides a repository of shared knowledge/experience in solved problems, and it encourages a shared language between developers. Both of which are good. Trust me.

Let’s take the shared language idea first as it’s a bit of a flimsy argument. Most, but definitely not all, developers I’ve worked with either haven’t heard of patterns or don’t know them off by heart. I know I don’t remember all the names. That’s mainly an education issue, rather than a problem with patterns, but it puts a barrier between people – it can even be taken as a mild form of elitism. You can shove your Flyweight Pattern up your Chain of Responsibility, they say. Maybe.

Even amongst those of us “in the know”, a problem is rarely a pure example of one of the patterns so we end up adding extensions to what we’re saying. In fact, it’s often more comfortable slipping from pure Patternspeak to Domainspeak. It’s not a Command pattern, it’s an Order Request or a Unicorn Disposal. Or we go low-level and discuss language features. The Patternspeak abstraction sits in a middle ground between domain language (understandable by the business) and implementation language (understandable by geeks and compilers).

Defining language and process has an unusual side-effect in that it sets an, often arbitrary, limit on creativity. Much as there is still a school of thought that programming can be defined without the need for creativity, that is nonsense – it’s problem-solving, and problem-solving often cannot be described by a rigid formula or process. I laugh when I hear about ideas such as executable UML, because computer-assisted software engineering was The Next Big Thing when I was at university and we’re still waiting for it to deliver.

The risk with patterns is that developers can become restricted by them – they follow the pattern precisely, or refuse to see options beyond the documented patterns. Again, that’s not a problem with patterns as such, but the way they are used. Actually, it’s a limitation of the human mind – we tend to stay within our comfort zone and often stick to what we know, even when that knowledge doesn’t seem to be beneficial in the circumstances. We become blinkered.

Recently I heard mention of “refactoring to patterns”. I think it was on an episode of DotNetRocks, but I might be wrong. You write your code free from preconceived ideas. Strictly speaking, you write code confined by the limitations of your own experiences and knowledge (or that of your team). Actually, you write code based on the real problem at hand. Then you step back and realise you have a Facade, Memento, Syntax-Directed Translator, Service Stub or whatever, which may or may not be a help when refactoring your work to be cleaner, more elegant or more understandable. Or when discussing the problem with an “in-the-know” colleague.

The other thing I heard a couple of years ago was that patterns are language-dependent – and I’ll chip in the phrase “paradigm-dependent” too. Interesting idea. The clue was the Gang of Four’s subtitle “Elements of Reusable Object-Oriented Software”. Object-oriented development might have the mindshare these days, but C is still one of the most widespread languages and functional languages like Haskell, Erlang, Clojure and F# are rapidly becoming the cool kids on the block. The Gang’s patterns are based on their knowledge and experience of statically-typed, object-oriented languages.

Something that could be a pattern in C++ or Java, might be a simple bit of syntax in something like Python or Ruby. How do object-oriented patterns map to functional or procedural languages? What about for dynamically typed languages? Of course, they have their own patterns – I’ve seen Erlang patterns crop up on the mailing lists, decades of Lisp and Scheme pattern seep into Clojure, and even a minority of fellow Pythonistas claiming patterns don’t exist in Python. Wishful thinking guys – we just have different patterns. When put like that, the shared vocabulary and knowledge provided by patterns becomes more disperse – with maybe a tiny shared core somewhere in the murk.

In conclusion, reviewing my meandering and rambling, patterns are still a good thing because of the shared language and wisdom of solved problems. But they can be troublesome too, if used inappropriately. My copy of the book has “An Invitation” at the end which essentially says patterns are something you build up yourself, with those in the book as a starting point. It was something I’d forgotten about until I dusted off the book to sit beside me as I wrote this. I wonder how many others have forgotten this important piece of advice – or even missed it completely?

Think about the patterns in your own domain, your own choice of technology. Then think about the language you use to describe things to your fellow developers, your domain experts and your customers – are you all understanding each other? Really?

Things to look at in 2011

Before I start on my 2011 list, it’s time for a quick review. My list for 2010 was:

• Clojure (http://clojure.org/)
• CouchDB (http://couchdb.apache.org/)
• Natural Language Toolkit (NLTK) (http://www.nltk.org/)
• PyGame (http://www.pygame.org/news.html)
• Twisted (http://twistedmatrix.com/trac/)
• XMPP (http://xmpp.org/)

I spent the year going to the London Clojure Dojo and I must admit that I do like the language and will continue to learn it. I’ve not had much reason to use it outside of the dojo, but I’m sure something will come up this year.

CouchDB is very impressive and CouchApp makes it pretty easy to develop self-hosted apps in CouchDB. I have an idea for a CouchDB app which I want to work on.

On the PyGame front, I did a fair bit of development early in the year, turning a game idea into… well, a mess. I started off a little too ambitiously and the unfinished result needs a serious bit of refactoring in order to progress further. I’ve got a simpler idea in the same style that I want to work on this year, which should give me a better idea about how to rework and finish the original game.

I spectacularly failed to look at XMPP (other than read the O’Reilly book), Twisted or NLTK (again, other than read the book).

So, for this year:

• PHP (http://www.php.net/)
• Python 3 (http://www.python.org/)
• Celery / RabbitMQ (http://celeryproject.org/ / http://www.rabbitmq.com/)
• XMPP (http://xmpp.org/)
• jQuery (http://jquery.com/)
• Android SDK (http://www.android.com/)

So, PHP is my new language for the year. What gives? Well, the company I work for uses a lot of PHP code. Even though I’ve been recruited for helping migrate PHP and Perl on the back end to Python, the front end  PHP code is not going away and it would be useful to be able to roll up my sleeves and help with the maintenance and development work. Unlike Erlang or Clojure, this is just a straight case of learning the syntax, and not really the same challenge of those languages.

Ah, Python 3. The company roadmap is to finish 2011 using Python 3 in production. This is, admittedly, a bit ambitious because not all the Python code we use will be Python 3 compatible (Django springs to mind). I’ve made the personal decision that the final release of Python 3.2 will signal the transition point for my home projects, at least taking into account library support.

I’ve already begun to look at Celery from a very basic point-of-view, but this is a technology we are going to be using more heavily at work this year. It’s actually one of the many things that attracted me to joining the company late last year – because I was never going to get a chance to use this commercially at my last place.

XMPP is still on the list and the London Python Dojo will be using it this year for the inter-dojo game challenge. Looking forward to it!

jQuery is something that was hidden on my 2010 list. I’m actually seeking to dust-off and improve my JavaScript knowledge this year and jQuery would be a useful addition to my toolbelt.

Finally, time to upgrade my ancient phone to one of these new-fangled “smart” phones. I had a company iPhone for a few months, but never really used it. It’s a nice enough piece of hardware and the UI is slick as you would expect… but it did nothing for me. The Android OS is open source and seems more in tune with what I might use for a phone OS. Although I would rather like an Objective-C SDK for it Actually, it looks like Clojure and Jython are unofficially supported, albeit rather slow.

As well as the new skills above, I’ll be continuing with Clojure, PyGame and CouchDB. I’ll also be getting some commercial Django experience, continuing to explore FluidDB (FluidInYourEar might finally get a public release this year!), the Flask web framework, and hopefully refresh my Erlang skills.

Here’s to 2011 – what technologies are you planning to look at this year?

 

Out of the Comfort Zone

One of the best ways to learn new skills is to be out of our comfort zone. When we’re out of our comfort zone, the mind and body attempts to deal with the situation – we become more alert and adaptable, we have to think a bit more, we have to pay attention to what’s going on. After all, we can’t rely on the situation being safe and predictable because we just don’t know what might happen.

When I’m not programming, I love snowboarding. I’m fortunate in that I’m just a few minutes away from an indoor snow slope. It might not be a proper mountain, but I can go ride any day of the year. I’ve been snowboarding for years and am pretty good because I’ve taken the plunge and put myself out of my comfort zone on many occasions. I’ve progressed up the instructor levels, I’ve been on performance courses with top pros, I’ve done a bit of proper off-piste, I’ve even raced boardercross (despite being an injury wuss).

One notable occasion was enrolling on an instructor course in Australia a few years ago. I had no intention to teach, just to improve my technique. In the end, I found myself having to improve my skills in front of people who were much better than me, and do something I feared most: public speaking.

All I wanted was the ground to swallow me up, but I didn’t have that luxury – I had to push myself harder than everyone else, and I had to just do it. Stand up in front of a group of strangers and teach them to snowboard. Keep up with people who had many more days on snow than me. Some had been instructors before. Nothing makes you learn fast like trying to match people way above your skill level.

I still get nervous before every lesson, but I found I enjoyed it a lot. It feels like it exercises different parts of my brain to those I use when programming or even snowboarding for my own enjoyment. The brain specialists might disagree with me, but you get what I mean.

However, there is a limit to how far we can go outside our comfort zone before the benefits disappear. And they can disappear very quickly. I’d never dream of taking a complete beginner to the top of a black run. They would be so far out of the zone that fear would overwhelm any opportunity to learn – it would also be incredibly dangerous and probably put them off ever going on a mountain again.

One of the things I love about code dojos, like the Python and Clojure dojos I attend each month in London, is that they provide a safe environment to move out of the comfort zone and learn new skills. Mistakes are all part of the learning process, yet we often don’t embrace them. I’m a firm believer that people learn most when things go wrong, because we’re taken out of that nice, comfortable, step-by-step process. When we make a mistake, we have to start thinking again to correct the mistake.

The trouble is that people don’t like to admit they’ve made a mistake because they feel they’ve lost credibility against their peers. But their peers are in the same situation. If no one wants to make a mistake, no one learns because of that fear of public failure and embarrassment. Dojos are great because everyone is there to learn new things and make mistakes in the process. Any mistakes you make stay in the dojo – it’s not production code, nor is it intended to be.

So where am I going with this? Well, my next out-of-comfort-zone situation is taking the skill I learnt as an instructor and bringing it into my full-time career: public speaking. Teaching beginner snowboarders is, relatively speaking, easy because they mostly don’t know anything about the subject. If I make a mistake, no one will notice. I get a big fear teaching other instructors because they know the subject, they know when you make a mistake.

But that’s a good thing right? They can let you know the mistake and help you fix it. It also forces you to practice and improve so you don’t make those mistakes – pushing that comfort zone further out.

I want to do the same thing with technical talks. The idea of standing up in front of other programmers and talking tech fills me with dread. Yet I know it’s exactly the same situation I was in when I first stood on an Australian mountain and tried to explain part of a lesson to a group of experienced snowboarders. After a few such experiences, I ended up loving it because it taught something new to my audience and because it helped me to improve my own understanding of the subject.

So my intention is to try to present a few talks over the coming months at events like the dojos, where I can make mistakes in safety. My goal is to present at one of the Python conferences. I was talking to one of the organisers of PyCon AU when I was at EuroPython this year, and he mentioned about presenting, which is where the whole process was set in motion. I’d like to attend PyCon AU next year, combining it with a visit to my brother in Sydney. Presenting would be both great fun and terrifyingly scary at the same time, but if the Aussies in the audience are as supportive of a nervous Pom as those Aussie instructors-to-be were, I should be just fine.

Tips, tricks and encouragement would be very welcome.

Coverage.py

I finally had a chance to try coverage.py, a Python code coverage tool which is maintained by Ned Batchelder. I’m liking it a lot: it’s minimal fuss to set up and use, yet very useful indeed.

When you’ve installed coverage, it should be on your path. Taking my work-in-progress PyGame code as an example, I ran coverage on the test suite:

$ coverage run run_tests.py

The test suite runs as normal:

$ coverage run run_tests.py
.................................................
----------------------------------------------------------------------
Ran 49 tests in 0.006s

OK
$

Coverage writes the results to a .coverage file. To view this as a plain text report:

$ coverage report
Name                      Stmts   Exec  Cover
---------------------------------------------
constants                     6      6   100%
domain/__init__               1      1   100%
domain/deployment            27     23    85%
domain/game_date             48     48   100%
domain/unit                  18     14    77%
infrastructure/__init__       1      1   100%
infrastructure/storage       12     12   100%
run_tests                     4      4   100%
tests/__init__                3      3   100%
tests/test_deployment        94     94   100%
tests/test_game_date         64     64   100%
tests/test_unit              57     57   100%
---------------------------------------------
TOTAL                       335    327    97%
$

But for the best results, you can create an HTML report:

$ coverage html
$

This writes HTML to an htmlcov directory, which can be viewed in a browser. The HTML report provides not only the summary listed in the plain text report, but links to pages for each source file. Each source listing is highlighted to indicate where code coverage is present and where code coverage is missing. Simple, but very, very useful. I’ve used the reports to improve the test suite’s coverage, and even identified code that could be removed. As you can see above, domain/deployment.py and domain/unit.py need better code coverage, which is something I’m aware needs to be addressed next as it will help with some upcoming changes.

One caveat is that coverage can only provide code analysis for code touched by your test suite. There are two areas of the code missing from the above report: the first is the source that binds the game together, since it’s not referenced by the test suite. More importantly, I mentioned I don’t have UI tests yet: since no UI code is called by the test suite, it doesn’t show on the report.

Something else to mention is it only tells you tests have touched sections of code – it’s up to you to ensure your test suite properly exercises that code.

Very cool, and thought I’d share in case you hadn’t seen this tool before.

Coverage can be obtained from http://pypi.python.org/pypi/coverage

Python Code Kata 4

I’ve started working through Dave Thomas’s Code Katas, as previous readers to this blog will be aware. I tackled the previous kata (Code Kata 2: Karate Chop) in Clojure as a way to help learn Clojure. I was planning to attempt the next kata in either Clojure or Erlang, but decided I would implement a solution using my usual language: Python.

The kata is in three parts, and I had to resist the urge not to read ahead and spoil the process. You can find the details of the kata at http://codekata.pragprog.com/2007/01/kata_four_data_.html

Part 1 involved reading values from a weather data file. The data in the file looked like a plain text conversion of an HTML page and had a bit of surrounding fluff that needed to be ignored in order to parse the tabular data. Through sheer laziness, I opted to use a regular expression to match lines with the data I was looking for and scoop the data out.

Part 2 was a similar task, but with football data. In this case, using the regular expression was more of a pain so I decided to split each line on whitespace and use only those rows that could be split into 12 items. All the rows I was looking for matched this criteria and it was quick to do. Unlike the first program, I ended up with more fields than I was planning to need.

Now comes the interesting bit: part 3. The task was to take the two programs and factor out the common parts.

To what extent did the design decisions you made when writing the original programs make it easier or harder to factor out common code?

In both cases, the code was in two functions. The first function parsed the data file, returning a list of valid rows. The second function took the list and identified the row which had the smallest difference of two values. There was a definite duplication of code in both, with differences in logic. The parsing functions iterated over the files the same, but the data extraction differed sufficiently. The other function was much more similar between the two programs.

Take the two file parser functions. For part 1:

def parse_file( filename ):
    data_file = open( filename, "r" )

    data = []
    for line in data_file:
        match = DATA_LINE_RE.match( line )
        if match:
            dy  = int(match.group(1))
            mxt = int(match.group(2))
            mnt = int(match.group(3))

            data.append( (dy, mxt, mnt) )

    return data

and part 2:

def parse_file( filename ):
    data_file = open( filename, "r" )

    data = []
    for line in data_file:
        columns = WHITESPACE_RE.split( line )
        if len(columns) == 12:
            data.append( columns )

    return data

The core of both functions can be distilled down to this:

def parse_file( filename, extraction_function ):
    data_file = open( filename, "r" )

    data = []
    for line in data_file:
        extracted_data = extraction_function( line )
        if extracted_data:
            data.append( extracted_data )

    return data

The domain-specific logic is now encapsulated in a function passed to parse_file. I can easily substitute new logic for future problems by passing in a different function. The extraction function should return either a tuple/list or a value that equates to false if no data could be extracted. I’ve used None in my code but an empty list would also suffice if it made sense for the problem at hand.

The second function in part 1 looks like this:

def print_day_with_smallest_spread( data ):
    smallest_day    = None
    smallest_spread = None

    for dy, mxt, mnt in data:
        spread = mxt - mnt
        if smallest_spread > spread or smallest_spread is None:
            smallest_day = dy
            smallest_spread = spread

    print smallest_day

In part 2 it looks like this:

def print_team_with_smallest_goal_difference( data ):
    smallest_team      = None
    smallest_goal_diff = None

    for row in data:
        goal_diff = abs( int(row[GOALS_FOR]) - int(row[GOALS_AGAINST]) )
        if goal_diff < smallest_goal_diff or smallest_goal_diff is None:
            smallest_team      = row[TEAM_NAME]
            smallest_goal_diff = goal_diff

    print smallest_team

There are some small differences, but the logic is essentially the same for both. The use of absolute values in the goal difference is a notable deviation from that used in the first procedure. However, while above what is required for the weather data it is also valid. I toyed with the idea of leaving this, but the change would mean improved commonality and make the factoring out easier: without changing the result. This was the deciding point and I made the change.

The extraction of values from the data list was the only other differing point, but both were doing the same thing. Some little tweaks resulted in a procedure suitable for both:

def get_item_with_smallest_difference( data, label_index, value1_index, value2_index ):
    smallest_label = None
    smallest_diff  = None

    for row in data:
        diff = abs( int(row[value1_index]) - int(row[value2_index]) )
        if smallest_diff > diff or smallest_diff is None:
            smallest_label = row[label_index]
            smallest_diff  = diff

    return smallest_label

Was the way you wrote the second program influenced by writing the first?

Definitely. It’s difficult not to be influenced by code you’ve written, or seen, before – especially when you have implemented solutions to two similar problems in close proximity. I was about to approach the second part in the same way as the first, using a regular expression to extract the data I needed. However, I realised that this would yield a large and unwieldy expression, so I decided to go for an approach I had briefly considered for the first part: split each line on whitespace to yield a list of values. I ended up with values I wasn’t going to use, but it was a cleaner approach for that particular problem.

Is factoring out as much common code as possible always a good thing? Did the readability of the programs suffer because of this requirement? How about the maintainability?

It’s usually a good thing, but in software development there are no such things as rigid rules. Readability of the refactored programs has some plus/minus points. The core of the problem was made more obvious from the refactoring, but the naming of the function to determine the smallest difference became more generic. That generic name is perfectly fine, but it might’ve made more sense to wrap the call in procedures named better for the specific problem domain.  The idea of constants, to aid readability, in the second solution (to identify fields we’re interested in) was transferred to the first solution which was a useful touch.

Overall, maintainability improved slightly. The duplicated code is factored out and can easily be reused for writing code to solve other problems of a similar nature. If a bug is found in one program, there’s a good chance it’s in the common code and thus can be fixed in both easily. Likewise, I could make performance improvements that would be applied to both easily. By splitting the domain-specific logic out from the generic core of the solution, I made the code easier to test by providing a discrete function for the logic, as well as a way to inject a fake or mock function into the parsing code.

Kata solutions for this and the previous kata are available under the MIT License from BitBucket: http://bitbucket.org/metaljoe/metaljoe_codekata/

Clojure Kata Two: Karate Chop (Pt. 5)

Lying ill in bed probably isn’t the most conducive environment for programming. I’ll spare you the full medical details, but in a strange way it put me in an evil mood… the kind of mood that seemed fitting to give macros a go.

I’ve never really seen the point of macros, but then again my experience of them has been the watered-down variety featured in languages like C. Reading up on macros in Scheme and Clojure more recently, macros come across as predominantly a way to do evil things. I say predominantly, because I accept that there are very rare occasions when a macro makes profound sense. Please note that the fifth solution I am about to present is definitely not an example of a good reason.

Last week, I was re-reading Jon Bentley’s excellent Programming Pearls book. I had been trying to avoid reading anything about binary searches while working on the code katas, so I picked a completely inappropriate book because binary searches are featured quite a bit. With my brain now sullied by a classic binary search algorithm, I went and undertook the fourth solution and tried to be a bit radical. It failed.

For the fifth one, I decided not to come up with my own solution but try a couple of things. The first was to implement Bentley’s algorithm in Clojure, and the second was to try and make it more compact through the use of macros – as a cheeky excuse to give them a try. The algorithm is already pretty elegant anyway, so I wasn’t sure what I could improve.

As ever, the code is on BitBucket at: http://bitbucket.org/metaljoe/metaljoe_codekata/

First thing was to implement the original algorithm in Clojure: (edited from changeset 9b3ef06116fd for clarity)

(defn chop [ item list ]
  (loop [ lower 0
          upper (- (count list) 1) ]
    (let [ mid (quot (+ lower upper) 2) ]
      (if (> lower upper)
        -1
        (if (< (get list mid) item)
          (recur (+ mid 1) upper )
          (if (= (get list mid) item)
            mid
            (recur lower (- mid 1))))))))

Okay, the first thing that stands out is the let, which holds the value of mid. Wouldn’t it be great to get rid of that? However, copying the code for mid back into the references would bloat the code out. I could use a function, but I’d be passing three parameters and it wouldn’t aid readability much. I would probably leave it for production code, but since I have a macro hammer in my hand, this looks like a macro nail waiting to be banged.

(defmacro mid []
  (list 'quot '(+ lower upper) 2) )

This will replace mid with the appropriate code in the macro. I create a list, all Clojure code is a list, and turn off parsing for quot and the lower + upper addition. If I don’t do that, they will both be evaluated in that context and will fail because neither are available in the namespace. I feel bad about these references, but this is just an exercise. I also realised at this point that list already exists in Clojure, and I should’ve used a different name for the argument to chop!

Once I’ve created the macro, I need to adjust the references to mid in the code. The references need to be wrapped in parentheses to avoid an error.

The next observation is that “(get list mid)” is repeated twice. It hasn’t quite reached the three-strikes rule, but it is duplication. I could set a variable to hold the value of the list at mid, or I could write a function to do it. The former would add an extra line to the code, and I would like to reduce the amount of variables if possible, especially as I’ve just gotten rid of one. The latter option wouldn’t offer much advantage because I’d still be passing list and mid to a function, and probably one with a less concise name than “get”. Basically, it’s another contrived situation for me to try a macro – though I would not do this in production code.

(defmacro get-mid []
  (list 'get 'list '(mid)))

This follows the same pattern as the mid macro – and even calls said macro.

(defn chop [ item list ]
  (loop [ lower 0
          upper (- (count list) 1) ]
    (if (> lower upper)
      -1
      (if (< (get-mid) item)
        (recur (+ (mid) 1) upper )
        (if (= (get-mid) item)
          (mid)
          (recur lower (- (mid) 1)))))))

Not a massive improvement to the code readability, and I’m not entirely happy with get-mid as a name. The nicest thing is that the let statement has been removed.

How about dealing with the statement to initialise the upper variable? Shunning a more obvious function…

(defmacro get-upper []
  (list '- '(count list) 1 ))

Okay, time for one more thing… well, two actually. The recur statements are fine, but perhaps we could make them more declarative?

(defmacro look-left []
  (list 'recur 'lower '(- (mid) 1)))

(defmacro look-right []
  (list 'recur '(+ (mid) 1) 'upper ))

This changes the loop code to read:

(loop [ lower 0
        upper (get-upper) ]
  (if (> lower upper)
    -1
    (if (< (get-mid) item)
      (look-right)
      (if (= (get-mid) item)
        (mid)
        (look-left))))))

In fact, while typing this up, it looks odd now because it’s defining two variables that are apparently never updated.

First off, let’s remove the last visible references to lower and upper, using (surprise!) a suitably named macro…

(defmacro search-exhausted []
  (list '> 'lower 'upper))

Now time to deal with that loop statement. Well, my plan was to create a macro like this:

(defmacro do-search [ & body ]
  `(loop [ ~'lower 0
           ~'upper (get-upper) ] ~@body
     {:lower ~'lower :upper ~'upper }))

This macro is based on Stuart Halloway’s evil-bench example from the Programming Clojure book code, which is very evil. But since this kata solution is all about evilness, I only feel slightly guilty.

The aim was to replace the loop with code that looks like this:

(do-search
  (if (search-exhausted)
    -1
    (if (< (get-mid) item)
      (look-right)
      (if (= (get-mid) item)
        (mid)
        (look-left))))))

Unfortunately, this raises the following exception:

Exception in thread "main" java.lang.UnsupportedOperationException: Can only recur from tail position (code_kata_2e.clj:64)

The macroexpand and macroexpand-1 functions weren’t giving me anything useful, probably because they were overwhelmed by nastiness. Oh well, not to matter as I don’t exactly grok macros just yet and I’m happy overall with what I’ve learnt this time. The code is on BitBucket, so if anyone with better macro development skills can point out the problem, please let me know.

Clojure Kata Two: Karate Chop (Pt. 4)

“The programmer builds from pure thought-stuff: concepts and very flexible representations thereof. Because the medium is tractable, we expect few difficulties in implementation; hence our pervasive optimism. Because our ideas are faulty, we have bugs; hence our optimism is unjustified.”

- Frederick Brooks, The Mythical Man-Month

I was on an evening train home when, re-reading The Mythical Man Month for the first time in almost ten years, I came across the above words. It summed up my fourth attempt at the Karate Chop code kata: because our ideas are faulty, we have bugs.

My fourth algorithm idea was faulty, I just didn’t quite want to admit it. My ideas were faulty because of lack of mental preparation. I was tired (I’ve not been sleeping well for a while now), distracted, and in a hurry to get past the Karate Chop kata and move onto another challenge. I didn’t think things through when I started, I just fired up Aquamacs and began to type.

The idea was simple enough – strip the algorithm down of a lot of state and just “bounce” from position to position within the list. Starting from the 0th element, we determine an offset which is equal to half the length of the list. The offset is added to the starting point to determine the position to check.

If we don’t match, we halve the offset and either add or subtract the offset from the current position to yield our new position, depending on whether the item we checked was higher or lower in value than the one we’re searching for.

It sounds simple enough, and plausible enough, so I really should’ve run a quick check in my head or on a sticky note to verify it actually worked. The fact I don’t recall seeing this process before should’ve tipped me off to needing to double-check it.

Thing were fine to begin with, except for one failing test. It was the first off-by-one error I had encountered so far during the Karate Chop implementations, I’m rather proud to say, so it seemed to be quite a simple thing to fix. Oh was I wrong: a simple change resulted in multiple failed tests. Hasty debug statements (I’m guessing I’ll need to learn a Java debugging tool for Clojure development) began to highlight much weirdness with each loop through the search. Values didn’t seem to tally to the expected values, and the search position was often wildly off.

A hasty bit of refactoring to eliminate duplication and side-effects as possible causes didn’t offer much improvement, though I learnt to apply some interesting bits of Clojure in the process. In the end, I had a pretty good idea that the algorithm was flawed, but I kept convincing myself that things like padding the list or adding special cases would massage it into place.

Some days you just have to down tools, stand back, learn from your experiences and move on. The idea started off simple but started to grow… and smell.

In the current form, the algorithm fails on three tests:

(assert (=  0 (chop 1 [1 3 5]) ))
(assert (=  2 (chop 5 [1 3 5]) ))
(assert (=  0 (chop 1 [1 3 5 7]) ))

The bit that stood out straight away was that they were all tests for a value on the boundary of the list. Analysis of the code determined that the search algorithm never reaches the first and last elements in the list. And padding the list doesn’t help either.

In the first test case, our initial offset is 1 since our list consists of three elements, and integer division of three by two yields 1.

Position 1 of the list (3) is not what we’re looking for, so we halve the offset (integer halving of 1 yields 0). 3 is greater than the value we’re looking for, so we subtract 0 from the current position i.e. we stay where we are. The algorithm detects the zero offset, and evaluates to not-found. See the problem?

Searching for 5 in the same list has the same problem, except we try to add the zero offset to our current position with identical result.

What they both should do is try an offset of 1 again, but that wouldn’t involving halving the offset each time.

The four element list has an initial offset of 2, which yields the value 5 at list index 2. We know to check the lower half of the list, so we halve the offset to yield 1 and then subtract that from the current position. Our new position yields the value 3, which means we have to search left again. Alas, our offset of 1 is halved to 0 and the search terminates. We never get to check that first element in the list.

Going through the problem on a sticky note diagnosed the problem really easily, and working through for 5, 6 and 7 element lists made me realise that the maths would need a reworking to cover those edge cases. Typing up my experiences just now, I’ve even had a thought about one way to solve the problem without getting too complex… but I will revisit another time.

All-in-all, this was actually the most interesting experience so far during the kata. We often learn a great deal from our mistakes than our successes, failure can be good for us sometimes – it takes us out of our comfort zone, which is where true learning happens.

As ever, source for this and the previous solutions is available from http://bitbucket.org/metaljoe/metaljoe_codekata/ under the MIT license. If you want to know more about the Karate Chop code kata, head over to http://codekata.pragprog.com/2007/01/kata_two_karate.html and take a look.

Clojure Kata Two: Karate Chop (Pt. 3)

This is my third Clojure-based solution to the Karate Chop kata detailed at http://codekata.pragprog.com/2007/01/kata_two_karate.html

The write up is going to be a little shorter than before, because I’m pretty tired as I type this. As always, source code to the solution is available from BitBucket at http://bitbucket.org/metaljoe/metaljoe_codekata/ under the MIT license and feedback is definitely always welcome.

For this solution, the aim was to produce a concurrent version of the code. Surprisingly, this proved to be quite tricky to write compared to implementing an equivalent concurrent solution in a language like Python or Erlang. The main problem was that the “Programming Clojure” book apparently missed a few important details about Agents (Agents can’t call Agents, shutdown-agents needs to be called for cleaning up). I got rather frustrated with the documentation and examples available, with the code raising assorted semi-cryptic Agent-related exceptions that weren’t adequately explained anywhere.

After a considerable amount of time on Google, and a drastic scaling back of my intended implementation, I finally created a working solution this evening.

The original aim was to keep dividing the list in two, spawning a new Agent/thread to investigate each new list by repeating the splits. This would continue until a match was found or an empty list reached. Not very efficient or elegant, but a way to incorporate an element of concurrency into the kata.

Since Agents can’t call Agents, the first major hurdle I had, I ended up running just two Agents: one to check the left half of the initial list, the other to check the right. The spawning of further Agents was eliminated in favour of an inefficient recursive check of each half of these lists, and so on. I opted to keep this style of checking, to “simulate” what my original implementation would have operated like.

This worked well, but the script would not quit until killed with Control-C. After another search through the book and docs, I found a reference to calling shutdown-agents. This only worked when called at the end of the script, not within the function calling the agents for various reasons.

I also took the opportunity to try and make a little more legible approach, as well as try out the split-at function.

Now I need sleep.

Clojure Kata Two: Karate Chop (Pt. 2)

This is the second attempt at a Clojure-based solution to the Karate Chop kata detailed at http://codekata.pragprog.com/2007/01/kata_two_karate.html

Source code is available from BitBucket at http://bitbucket.org/metaljoe/metaljoe_codekata/ and is offered under the liberal MIT License. Since the aim of a kata is to learn and become a better programmer, feedback is always welcome.

Since the previous solution became a recursive one, my aim for this attempt was to work on an iterative implementation. Concerned about the rather ugly feel to the recursive code, I was also looking to produce something a bit more compact, a bit more elegant.

The code actually came to me pretty quickly compared to the first attempt. One advantage is that the problem domain is clearer on the second attempt, my brain has had time to consider the pitfalls and map out a more concise understanding of the problem. The other factor is that writing my own Clojure code, as opposed to quick REPL exploration or fill-in-the-blank exercises, has become more natural. I have begun to start thinking in Clojure. Note that I said “have begun to” – I still have more work to do on that front.

As far as implementation went, it was pretty smooth until I hit one snag: an infinite loop. It took a while to figure out what was going on, until I spotted a glaring issue. If the slice/partition of the list I’m looking at reaches one element, I check it for a match. If it didn’t match, I would loop around and search it again. An embarrassing newbie error, but a useful mistake to make. I added the appropriate check and all the tests passed.

After completing the implementation I spotted the second mistake when I reviewed the code later on. I was testing for an empty list on each iteration. While it has no impact on the output, it’s an unnecessary check for the implementation – if the list does suddenly become empty, we have bigger problems!

One thing that did intrigue me was that with almost no effort, the solution was much smaller and more readable than the first approach. I usually find the opposite with recursive versus iterative code, but in the case it’s probably because the problem seems more natural expressed iteratively. Or maybe my recursive approach just sucked

Here’s the completed algorithm, with tests removed and the code tidied for clarity. See the Mercurial repository at BitBucket for the original source.

(defn chop [item list]
  (if (empty? list)
    -1
    (loop [ min 0 mid (quot (count list) 2) max (count list)]
      (if (= item (get list mid))
        mid
        (if (or (= mid min) (= mid max))
          -1
          (if (< item (get list mid))
            (recur min (quot (+ mid min) 2) mid)       ; look left
            (recur mid (quot (+ mid max) 2) max))))))) ; look right

If I broke my rule to revisit the code, I’d change a few minor things for readability. For example: why did I put those two comments there, and not elsewhere? The loop initialisation looks a bit of a blur at first glance, the or statement might need to be wrapped in an intention-revealing function name. However, the code is good enough for the task at hand and that’s all that really matters.