Pub language discussion in London

For those of you that aren’t members of the London Java Community there has been a pretty active discussion going on through the mailing list about learning a new language (and which one to learn).

The conversation will no doubt be continuing at the monthly social event (which goes out to members of the Clojure, Scala & Groovy communities in London) tomorrow night. Feel free to come down and fight for Groovy’s corner: http://www.meetup.com/Londonjavacommunity/calendar/13588662/

London GGUG, 17th May – DSLs in Groovy & Grails 1.3

I’m pleased to announce that the next London GGUG on the 17th May will have a guest from the US: Peter Bell. He’ll be giving a talk on the practical development of a Groovy DSL with emphasis on the practical side. As he says on his blog, this is for both people that want to go further with DSL development and those that simply would like to see in what ways a DSL can help them.

I’ll be speaking on the new features that come with Grails 1.3, focusing in particular on plugins deployed to Maven-compatible repositories.

You can register for the event on the SkillsMatter website, where you can also see synopses of the talks and the speaker bios. Please register soon and I hope to see you there!

Testing Groovy code – an undesirable burden?

Many of the complaints I’ve heard directed at Groovy and Grails derive from the same issue: the compiler doesn’t pick up type errors. People worry that simple typos will make it into production and that they’ll be less productive due to MissingMethod and MissingProperty exceptions popping up when they run the application.

The standard answer to this is a simple one: write tests for your code. In fact, prefer test-driven (or test-first) development. Yet this doesn’t seem to satisfy some people. “What projects do you know that have 100% test coverage?” One or two that have close to 100%. But are the rest reliable? No, not even the Java-based ones. Any code that isn’t tested has a significantly high chance of being buggy. How can this be a serious argument against dynamic languages?

That’s not to say I think dynamic languages should take over from static languages. Both have their place and deciding which to use for any given task will often come down to personal preference. They both have their strengths and weaknesses, but a discussion of those will have to come another time.

Too much testing!

Another argument I’ve heard is that dynamic languages require more testing because you have to explicitly test for type errors that would otherwise be picked up by the compiler. I think this is frankly rubbish. What do we do when we’re testing? We’re checking that code behaves as expected given certain inputs. If there are any type errors in the code under test, you won’t see the expected behaviour.

Let me demonstrate. Say we have a Grails controller {{MyController}} with a corresponding unit test. The test starts simply:

package org.example

class MyControllerUnitTests extends grails.test.ControllerUnitTestCase {
    void testIndex() {
        controller.index()
    }
}

All we do is invoke the index action on the test controller. Now see what happens when we run the test against this controller code:

package org.example

class MyController {
    def index = {
        def id = parms.id
        render "You have reached ID ${ids}"
    }
}

If you run the test in SpringSource Tool Suite, you’ll see something like this:

As you can probably tell, the test is failing because the variable parms does not exist. Not surprising since this was a deliberate typo. Once you correct that typo, the test fails on the next one, ids. Change that to id and the test is now passing.

So, even though the test simply executes the action and doesn’t even bother checking the results, we get feedback on missing properties and methods. Granted, the feedback isn’t as instantaneous as you get in an IDE with a statically typed language, but in my experience it’s not a huge hit on productivity. And while we’re on IDEs, I know that both STS and IntelliJ IDEA will underline properties and methods it can’t resolve, so you would probably notice that parm and ids were mistyped before running the test.

This doesn’t really do much for my argument yet because I’ve just shown that you need an “extra test” to get the same checks you’d get with a static language. But remember that the aim of a test is to ensure that code behaves as it should, so we should check the action renders the string we expect! That’s easily done:

package org.example

class MyControllerUnitTests extends grails.test.ControllerUnitTestCase {
    void testIndex() {
        controller.params.id = 10
        controller.index()

        assertEquals "You have reached ID 10", mockResponse.contentAsString
    }
}

The test still passes, but it now checks that the action is rendering the appropriate string to the response. The key point I want to make here is that the test is no different than it would be if we were testing Java code, yet it will still pick up typos.

What about type-checking?

I may have justified my position when it comes to missing methods and properties, but what about real type errors, such as using a string where an integer is expected or vice versa? This is an interesting question because in dynamic languages, it’s typically the wrong one to ask. How can you have a type error in a dynamically typed language? You might have some code like this:

def n = 100
n = n.substring(1)

which you may see as a type error (a number is being treated like a string), but as far as the language is concerned all we have is a missing method on the value of n. This is important because you could very well add the method substring() to integers, in which case the code would work:

Integer.metaClass.substring = { start ->
    return delegate.toString().substring(start)
}

def n = 100
n = n.substring(1)

In other words, “type errors” in a dynamic language are simple missing method or property exceptions, which I’ve already demonstrated will be covered by your normal tests (without any extra shenanigans).

I would stop there, but that isn’t the whole story when it comes to Groovy. After all, it has static types! It also frequently interfaces with (statically typed) Java code. How does this affect our testing? Ah, I wish there was a simple answer to this. In some ways, passing the incorrect type to a method will be picked up. Try running this script:

def someMethod(String str) {
    println "Some method: $str"
}

someMethod(100)

Boom! You’ll get a missing method exception because Groovy can’t find an instance of someMethod() that takes an integer. Unit tests can handle cases like this, but in practice such problems raise their head once you start wiring your objects together and they start interacting with each other. At that point, you have to start thinking about integration tests – a topic for another time.

Have I convinced you?

The main thrust of this blog post has been to highlight that your bog standard unit tests, which you can run easily from your IDE, will pick up those typos and “type errors” you’re worried about without you having to do any extra work. All you need to do is focus on writing tests that check the behaviour of your code. It’s not trivial to write thorough tests, but that’s independent of the language you use.

Something else you should bear in mind is that It’s a lot easier to test code written in a dynamic language because you don’t have to always create objects of the correct type – they just have to have the appropriate methods and properties. So dynamic languages both strongly encourage you to adopt best practice and make that best practice easier.

Despite all this, I understand that it can be difficult in many environments to write tests for code, let alone write tests before writing any code. But I think this is a problem of culture and habit. Once you get used to writing tests, and then writing them first, the whole process becomes easier. You then find you have more confidence in your code. I can only strongly recommend that everyone takes any opportunity they can to try this approach out and get used to it. You never know, the managers may even come round to realising it’s a good idea!

GitHub Pages: publish your Grails plugin docs

Currently, the primary site for Grails plugin documentation is the plugin portal on the Grails web site. The trouble with the portal is that it doesn’t readily support multi-page documentation. On top of that, you can use gdoc to write a user guide for your plugin, but where will the generated manual go?

Thanks to Luke Daley, I discovered GitHub Pages as a potential solution for those plugins I have hosted on GitHub. This feature allows you to create simple websites on a per-project basis.

Getting started

Before you can start publishing websites to GitHub, you first have to create a ‘personal’ page. The process is straightforward: create a new GitHub repository, clone it locally, add an index page, and then push your additions to GitHub.

For example, let’s say your GitHub username is ‘franklin’. First, create a new GitHub repository with the name franklin.github.com. This repository will store your personal website. Once that’s ready, clone it and add an index page:

$ git clone git@github.com:franklin/franklin.github.com.git franklin.github.com
$ cd franklin.github.com
$ touch index.html
$ git add .
$ git push origin master

After the above steps, you will have a blank home page for http://franklin.github.com/. Note that it can take a while for your GitHub Pages site to activate. Also, you can access the URL http://<username>.github.com/ before you perform the above steps and you will see some instructions for setting up the various pages.

Now that you have a personal Pages site, you can create project sites with URLs of the form http://<username>.github.com/<projectname>/. The process for this is trickier, so I refer you to the instructions on the GitHub Pages. You can either do it via your local git, or you can got to the project page on GitHub, click on the “admin” button, and select the “Generate project page” link. I prefer the latter approach because it’s easier and involves less typing!

Once you complete the steps from either set of instructions, you will end up with an extra local tracking branch called ‘gh-pages’. This branch contains the website for your project and it’s where we will be storing the plugin documentation.

The plugin documentation

When the grails doc command first appeared, it allowed you to automatically generate API documentation for your projects. With Grails 1.2, its scope was expanded and it can now generate a manual that looks and feels just like the Grails user guide. You don’t have to do anything to get the API documentation, but the manual requires a bit of work.

The basic idea is that you create a src/docs/guide directory and populate it with gdoc files. By prefixing the names of the gdoc files with appropriate numbers, you get a hierarchy of numbered sections. To get some idea of what you need to do, check out the source files that Luke Daley created for the Grails JMS plugin user guide.

Publishing the docs

Generating the documentation leaves you with a docs directory full of HTML files. What do you do with it? The next step is to get it onto your project-specific GitHub Pages site. Doing this manually is a bit of a bind, but I’ve created a Grails script and some template files that make it almost trivially easy (once you have the ‘gh-pages’ branch locally!).

First, copy this script to your $HOME/.grails/scripts directory. Next, switch to your ‘gh-pages’ branch and copy this HTML template to the root directory of the project. Next, create an index page for your site: index.html, index.textile, and index.markdown are all acceptable. Choose whichever format you prefer, but it should start with this section:

---
layout: main
title: My Grails Plugin
---

The three dashes are important! I’ll explain those and the layout/title later. Then,

$ git add main.html.tmpl
$ git add index.textile
$ git commit -m "Adding template layout and index page for GitHub Pages site."

In the above set of commands, replace ‘index.textile’ with the name of the index file you created.

You’re now ready to publish your plugin docs for the first time! Simply switch to the main branch and execute the new Grails command:

$ git checkout master
$ grails publish-github

Once the command successfully completes, you will find yourself back on the ‘gh-pages’ branch with all the documentation staged for a commit. You can then make any changes you want, commit everything, and then push the branch to GitHub. You can even do this in one step with the grails publish-github --push command, or if you want to automatically commit the documentation but not push, use grails publish-github --commit.

A few details

The publish-github command does a few things. First, it generates the plugin’s documentation via Grails’ doccommand. It then copies the generated documentation to the ‘gh-pages’ branch (using an rather ugly technique). Once it’s on the ‘gh-pages’ branch, it generates a new ‘main’ layout based on the main.html.tmpl file. It basically reads the information from the XML plugin descriptor, plugin.xml, and populates the template with things like the plugin version and author. The template is then generated to _layouts/main.html, which you can reference from your individual pages.

It’s up to you to write the site’s pages, but at least you have the option of three different syntax types. I prefer Textile for the moment, because it’s very similar to the gdoc format. The key part of each page is the header block, which you’ve seen already:

---
layout: main
title: My Grails Plugin
---

The ‘layout’ value is used to determine which file under _layouts to inject the page into. Ideally, the index page should use the _layouts/main.html template that the publish-github command generates, hence the value ‘main’. The ‘title’ option is specific to the generated template, which inserts the corresponding value into the title of the page. This is because it’s tough to get a suitable plugin name with correct capitalisation from the plugin descriptor.
Update Somebody has pointed out that the publish-github command doesn’t work if plugin.xml is committed to your main branch (typically master). You could try modifying the script to handle this, but the XML plugin descriptor doesn’t need to be in GitHub at all – it’s generated after all – so I would simply git rm it.

That’s pretty much it. You also have the full power of Jekyll (as described on the GitHub site), so you can build more elaborate sites as you see fit. If you want to see the results of all this jiggery-pokery, then take a look at the GWT plugin pages.

Until next time.

Grails 2.0 #1: Gradle for the win?

Ever since I attended a Gradle course, I’ve been rather obsessive about it. In particular, I’ve been looking at integrating Grails into Gradle builds. The early result was a straightforward Gradle plugin that allows you to build Grails projects. I’ve explained how to use it in a previous post, so I won’t talk about that here. It works, but it bothers me that the plugin delegates to the Grails build system. That interaction is always going to be anaemic and will almost certainly frustrate any Gradle power users. But is there any other approach that can work?

The problem

Grails has a custom build system based on Gant that has evolved over the years. The nature of the build and its interaction with Grails plugins means that it’s very difficult to integrate well with other build systems. It does its own dependency management (although it does delegate the grunt work to Ivy) and it has its own event system.

The trouble is, the build is definitely getting more bloated and complex as features are added. In fact, many of those features, such as dependency management, distinct compile/test/runtime classpaths, and artifact publishing, are built into a tool like Gradle. So we’re basically reinventing the wheel to some extent. And I have to admit that Gradle’s API is far more robust and flexible than the Grails build system.

This got me thinking: would a native Gradle build for Grails projects work? After all, build is not one of Grails’ core competencies. Why not leave it to a dedicated tool if that tool can handle Grails’ special requirements? Graeme and I had thought about doing this some time ago, but decided against the move. Gradle was still young and didn’t seem to offer enough benefits. Times have changed though.

My next thought was this: what is a Grails project if not a Groovy WAR project with a rather unique directory structure? If Gradle can handle that directory structure, why not use its standard compile tasks? Doing so would mean that other Gradle plugins would also work with Grails projects.

The solution

On that Gradle course, I discovered a Gradle feature that really sold me on its potential as a build tool for Grails projects: source sets. These are arbitrary groupings of source files that can pretty much manage any directory structure you can throw at them. Even a Grails project. All I had to do was create a test Grails project with a Gradle build, apply the ‘groovy’ plugin, and configure the source sets. Next thing I knew, I could compile a Grails project. Whoohoo!

If only everything was so simple. Trying to run a Grails application proved to be a far tougher challenge. This required compilation and loading of all installed Grails plugins, including the core plugins. On the bright side, I had an opportunity to create a build-specific plugin manager that didn’t have to worry about runtime stuff. This meant a great simplification of the code.

The result of all this work is another Gradle plugin, although this one hasn’t been published yet. It’s in a very raw state at the moment, and I want to avoid confusion with the existing Gradle plugin I mentioned at the beginning of the post.

About the plugin

So how does this plugin work? The core of it is the grails.build.plugin.GrailsPlugin class, which is the entry point for the plugin. It does several things of note:

• apply the ‘groovy’ and ‘war’ plugins to the project;
• configure the source sets;
• sets up some extra dependency configurations; and
• creates a whole bunch of specialist Grails tasks and creates the dependencies between them.

The most significant tasks are:

• buildData – instantiates BuildSettings and loads the build configuration
• buildPlugins – builds the project’s installed Grails plugins and loads them
• generateWebXml – generates the web descriptor to ./web-app/WEB-INF/web.xml
• generateApplicationContextXml – generates the root application context descriptor to ./web-app/WEB-INF/applicationContext.xml
• packageI18n – copies the i18n resource bundles and converts them to ASCII if required
• run – launches the configured servlet container

As you can see, the plugin already handles Grails plugins, although there is still much that doesn’t work. The fact that the plugin can fully build plugins of various types and add their dependencies and classes to the appropriate classpaths is a big deal though. This just doesn’t happen with the current Grails build system.

Another really neat feature is that Grails plugins can provide their own build files and Gradle plugins. You can see an example in the Jetty plugin that’s packaged in the root of the grails-build-x project. This is incredibly powerful stuff because it means that the Grails plugin can have full access to the Gradle build and do pretty much anything! For Jetty, this means adding the container starter classes to the servletContainer configuration and adding its dependencies to the project’s runtime classpath. But pretty much anything is possible.

The current run task demands some extra requirements of servlet container plugins, so that standard ones won’t work. However, the custom Jetty plugin I referred to earlier can also be used in a normal Grails project with the Grails build system. That means a plugin author can add features that help it integrate with the Gradle plugin without breaking normal Grails projects!

So what does it all mean?

The Grails build system evolved out of a failure on the part of existing build tools to provide a solution to building Grails projects, particularly once plugins were introduced. At the time, I think Gradle was still an idea in the air. Since then, the build system has grown in features and complexity, and it works pretty well.

Now, though, I think it’s time for it to retire. It doesn’t make sense for Grails developers to spend a lot of their time on build stuff when another tool is now available that will do most of the work itself. The build system also doesn’t play particularly well with other build tools, which can make adoption in the enterprise rather tricky. And a clean room implementation means clearing out a lot of cruft and being able to resolve some outstanding and difficult problems.

The great thing about the new Gradle build is that it can be developed in parallel to the Grails 1.x line. And if it makes it into Grails 2.0 as the build system, then hey, the plugin authors have already prepared and tested their plugins for it. Certainly my hope is that Gradle forms the core of the build system for Grails in the future. Fingers crossed!