Peter Ledbrook

“It’s more of an art than a science”

I’ll be honest, this phrase bothers me. Perhaps it’s because I’m a scientist by training. Perhaps it’s because this seems to be a misuse of the work ‘art’ or a misinterpretation on my part. But whenever I hear it used with reference to software development, I hear: “we use heuristics and guesswork because we don’t have time to do research and there is no body of research from which to draw”. Does that really make the solution to an underlying question or problem an ‘art’ rather than a science?

I of course tried googling the phrase to determine what it’s supposed to mean, but didn’t get very far. The top result from my search was:

It means it is not something which is governed by clearly-defined rules, as a science would be. In science things are either right or wrong; in psychology (or any art) it’s not possible to say what is ‘right’ or ‘wrong’.

This particular answer seems to misunderstand what science is. In essence, it’s a way to understand the way the world works through experimentation and verification. It’s also typically methodical because reproducibility of research results is important.

Perhaps the reference to rules is based on experience with things like mechanics in Physics and Newton’s laws of motion. We can predict the trajectory of projectiles in the air for example. But this is a very limited view of science. I have been watching the programme Horizon on the BBC recently and learned about the science of taste and even creativity. Yes, we’re learning through science about how creativity works!

At the end of the 19th Century and beginning of the 20th Century, we thought that we would soon learn everything there was to learn about the world. Over a century later, there still seems to be no end to the growth of our scientific knowledge. And things that used to be firmly considered “arts” are much less so now.

Consider cooking: more and more chefs are learning the basic science of both taste and cooking. From that base of understanding they can be even more creative in what they do. It allows people like Heston Blumenthal to create bacon and egg ice cream or snail porridge. If you’re interested, McGee on food and cooking is an essential read on the underlying science.

This also highlights an important point: creativity and science are in no way mutually exclusive. In fact, each enables the other. As I mentioned, a scientific base allows for more creativity because of the deeper understanding of how things work, but creativity is also essential in providing insights into how things work.

Coming back to the original point of this post, my ire was recently raised by a discussion on Hacker News where someone wrote

I’m not sure that there is a sure-fire way to quantify what tests are or are not necessary. In my opinion, this is something that comes with experience and is more of an art than a science. But I’m okay with that.

This seems innocuous enough and I wouldn’t be surprised if many people agree with it. But do we really think that it’s not possible to learn through research what a good level of tests is? Software is typically structured and full of patterns, so the pool of possible structures to investigate is limited. In addition, we already have tools to detect cyclomatic complexity and other metrics of software, so would it be so hard to determine which parts of the software are involved with the “critical” features?

I think what bothers me the most is that despite the huge revenue of the industry as a whole, and how much money depends on the successful completion of IT projects, so little research seems to be done to help improve the software development process. Perhaps the research is being done but it’s not widely disseminated. But I would at least have expected to come across research to back up the claims of agile practitioners (as one example). Not that I necessarily disagree with what they say, but it seems that going agile requires more faith than should be necessary.

Does the software development industry and community require a more scientific mindset? What do you think?

Where next for Grails?

A time comes for every open source project when it has to take a step back, reflect on the past and decide where it needs to go next. The world rarely stays the same as when the project was born and in the tech world things change year on year. Back when Grails first came out, Java web frameworks were still a pain to configure and you had to pull together a whole host of libraries to do what you wanted. The use of conventions was rare and ease of use didn’t seem to be a high priority.

Now in 2013 there are probably hundreds of web frameworks for the JVM, most of which have learned lessons from Rails and its ilk. In addition, we have Node.js spawning lookalikes and derivatives (such as Meteor) aiming to provide lightweight servers that can handle hundreds of thousands of concurrent connections through asynchronous IO. In this world, what is Grails’ value proposition and where should it be heading?

This is a pretty long post, so if you want to skip straight to my suggestions, head on down to the conclusion at the end of the article. But I’m hopeful you’ll find value in the rest of the article!

Read the rest of this entry »

On configuring Apple TV

Partly on impulse and partly because I’d been discussing media streaming with my electrician, I bought an Apple TV (3rd generation). For a start, it seems a real steal at £83 and can stream from iTunes, YouTube and Netflix. The fact it doesn’t yet support LoveFilm Instant or Sky Now is a bit of a downer, but this isn’t a review of the Apple TV. Rather, it’s a collection of useful tidbits for getting the thing working better, in particular with AirPlay.

What’s AirPlay?

The Apple TV is a streaming media player. I’ve already mentioned a few sources for that streaming (iTunes, Netflix), but what if you could stream directly from an iPad or iPhone? This is what AirPlay does. When you have an iOS device or even a Mac laptop with Mountain Lion, you can mirror the display to your big-screen TV via the Apple TV box as long as they are on the same network.

This is pretty cool stuff, but there was one big flaw: when I first tried it, the performance was shocking. Completely unusable. Second time I tried it (on a different network), same thing. Being the highly rational person I am, I immediately blamed Apple and their AirPlay implementation. How could they seriously foist this garbage on us?

I didn’t give up though and I had this nagging feeling that the network could be the problem, even though WiFi is supposed to give us up to 54Mb/s. On a whim, I bought a TP-Link Powerline kit (which I see has gone up in price since I bought it) to provide a more reliable network connection between the router and my Apple TV. In one stroke, a significant improvement in the AirPlay performance. It still wasn’t really usable though, but it did highlight WiFi as the likely culprit.

Tuning the WiFi

After tweeting this, someone responded and pointed out that WiFi contention might be an issue. So I started investigating. In the UK, WiFi breaks down into 13 overlapping channels. Your WiFi access point is configured for one of these channels. So are probably a bunch of others in your vicinity. At this point, all I knew was that 15+ WiFi access points were broadcasting in range of my equipment and that many of these were probably interfering with my own network.

The first suggestion I came across was to set the access point to 802.11g only. I’d be surprised if many people have equipment that doesn’t support ‘g’. I certainly don’t, so I duly made the change.

The second suggestion was to pick an uncontested or at least weakly contested channel. To be honest, I didn’t even know what channel my WiFi was configured for. Fortunately, I found a magic incantation for my Mac laptop to discover what channels all the WiFi networks were using:

/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport -s

That’s one ugly command line, but that’s only because the airport utility isn’t on the path by default. Anyway, running that gave me a list of all the WiFi networks and their channels. If you happen to be on Windows or Linux, it shouldn’t be too hard to find an appropriate tool to give you this information.

What I found out was that channels 1, 6, and 11 were the most prominent, with a few networks on 7 and 9. At first glance, it looks like I have plenty of choice for the channel since there are 13 available to me. But it’s not at simple as that. These channels overlap:

(This image linked from http://www.airportal.de)

Any channel that overlaps with another interferes with it. As you can see, a predominance of channels 1, 6 and 11 pretty much mess up all channels! What could I do? I basically deduced that channel 13 was the best bet since the only channel in use it overlapped with was 11, and 13 has a larger stagger over 11 than 12, thus reducing the risk or strength of contention.

I don’t really know whether the above is accurate, but once I picked channel 13 for my WiFi access point and limited it to 802.11g only, AirPlay suddenly started working. And working well. This is great, because I can access things like LoveFilm Instant and Sky Now via my laptop or iPad and stream to my TV, even though Apple TV doesn’t directly support either of them. I also noticed that the Netflix streaming became much more reliable after switching to Powerline.

So if you’ve been having buffering and lagging issues with Apple TV (or any media streamer for that matter), then I hope the information I’ve presented helps you sort them out!

The journey continues

Last Friday (23 Nov 2012) was my last day at VMware. I’ve had a great time there and really enjoyed working with the team. So why leave? The two main ones are:

• to focus on Grails in Action 2nd edition; and
• to do some training and consulting to help teams navigate the big changes happening in the development space.

This doesn’t mean I will be disappearing from the Grails scene – far from it. I’m hoping that I will have a bit more time to work on some of my plugins and maybe some other projects (such as gvm, which I’m contributing to atm). I’ll see how it pans out.

Any switch to consulting won’t happen until later in the new year though as I focus on the book. I’m keen to get it to our patient readers as soon as possible! And preferably before Grails 3 comes out

I’d like to end by thanking VMware for the support they have given the Groovy & Grails projects and for giving me the opportunity to learn some news skills and to continue contributing to the OS community.

DRY JSON and XML with Grails

Have you ever tried to support both JSON and XML in your REST API with Grails? There is the very straightforward:

class MyController {
    def index = {
        def objs = ...
        withFormat {
            json {
                render objs as JSON
            xml {
                render objs as XML
            }
        }
    }
}

It works well and is trivial to implement, but it does suffer from a significant problem: you have no control over the JSON or XML generated. So if you’re serialising domain classes, any change to your internal domain model will be reflected in the public REST API. That is Not A Good Thing.

You might then look into using the JSON and XML builders instead, but you will soon discover that they have different syntax and the structure of JSON and XML is different anyway. So you find yourself writing separate code to render JSON and XML for each action that’s part of your REST API. That’s more work than you really want.

Can you somehow marry the two to get the best of both worlds: minimal coding but control over what goes into the public API? If you don’t mind a few constraints on the output, I think you can. My proposal boils down to:

1. Transform the data in nested maps and lists of maps, filtering out anything you don’t want in the public API
2. Use ‘... as JSON‘ for JSON responses
3. Use a custom method for rendering the same data as XML

Let’s take an example from the grails.org application. I want to generate a list of plugins in both JSON and XML forms so that consumers have a choice of format. The first step is easy: get a list of the plugins I want to render from the database. The next step involves transforming those plugin domain instances into a hierarchical structure based on maps. Here’s the code I came up with:

class PluginController {
    ...
    protected transformPlugins(plugins) {
        return [ pluginList: plugins ?
                plugins.collect { p -> transformPlugin(p) } :
                [] ]
    }

    protected transformPlugin(plugin) {
        def pluginMap = [
                name: plugin.name,
                version: plugin.currentRelease,
                title: plugin.title,
                author: plugin.author,
                authorEmail: plugin.authorEmail,
                description: plugin.summary,
                grailsVersion: plugin.grailsVersion,
                documentation: plugin.documentationUrl,
                file: plugin.downloadUrl,
                rating: plugin.avgRating ]

        if (plugin.issuesUrl) pluginMap.issues = plugin.issuesUrl
        if (plugin.scmUrl) pluginMap.scm = plugin.scmUrl

        return pluginMap
    }
    ...
}

The dynamic nature of Groovy and its expressiveness make the transformation pretty simple to effect. This could probably be simplified even further if we had something similar to the bindData() method that would map all properties of a domain instance to Map entries except those specified in an exclusion list.

Rendering the plugins as JSON then becomes a simple matter of:

def plugins = Plugin.list(...)
render transformPlugins(plugins) as JSON

since JSON maps perfectly to the map structure I’ve created. XML is a trickier proposition because it doesn’t have a straightforward concept of objects and lists. We could use as XML, but then we would end up with a whole bunch of <map> and <list> elements. No, we have to use a different approach.

I plumped for using the render() method in its XML builder form:

class PluginController {
    ...
    protected renderMapAsXml(map, root = "root") {
        render contentType: "application/xml", {
            "${root}" {
                mapAsXml delegate, map
            }
        }
    }

    protected mapAsXml(builder, map) {
        for (entry in map) {
            if (entry.value instanceof Collection) {
                builder."${entry.key}" {
                    for (m in entry.value) {
                        "${entry.key - 'List'}" {
                            mapAsXml builder, m
                        }
                    }
                }
            }
            else {
                builder."${entry.key}"(entry.value, test: "test")
            }
        }
    }
    ...
}

So now rendering the plugins as XML becomes:

def plugins = Plugin.list(...)
renderMapAsXml transformPlugins(plugins), "plugins"

where the second argument is the name of the root element in the generated XML. The great thing is, this method can be applied to _any_ transformed data as it’s not specific to plugins.

One thing to note is that the mapAsXml() method assumes that the name of any map key that has a list as its value consists of ‘<name>List’. The corresponding XML becomes a parent <nameList> element with nested <name> elements for each of the list elements. This convention simplifies the whole process without being excessively onerous.

So there you are: Don’t Repeat Yourself rendering with JSON and XML. It may not be the most efficient approach computationally, but it will save a fair bit of development and maintenance time. And don’t forget that you always have the option of caching the responses.