Peter Ledbrook

GitHub Pages: publish your Grails plugin docs

Currently, the primary site for Grails plugin documentation is the plugin portal on the Grails web site. The trouble with the portal is that it doesn’t readily support multi-page documentation. On top of that, you can use gdoc to write a user guide for your plugin, but where will the generated manual go?

Thanks to Luke Daley, I discovered GitHub Pages as a potential solution for those plugins I have hosted on GitHub. This feature allows you to create simple websites on a per-project basis.

Getting started

Before you can start publishing websites to GitHub, you first have to create a ‘personal’ page. The process is straightforward: create a new GitHub repository, clone it locally, add an index page, and then push your additions to GitHub.

For example, let’s say your GitHub username is ‘franklin’. First, create a new GitHub repository with the name franklin.github.com. This repository will store your personal website. Once that’s ready, clone it and add an index page:

$ git clone git@github.com:franklin/franklin.github.com.git franklin.github.com
$ cd franklin.github.com
$ touch index.html
$ git add .
$ git push origin master

After the above steps, you will have a blank home page for http://franklin.github.com/. Note that it can take a while for your GitHub Pages site to activate. Also, you can access the URL http://<username>.github.com/ before you perform the above steps and you will see some instructions for setting up the various pages.

Now that you have a personal Pages site, you can create project sites with URLs of the form http://<username>.github.com/<projectname>/. The process for this is trickier, so I refer you to the instructions on the GitHub Pages. You can either do it via your local git, or you can got to the project page on GitHub, click on the “admin” button, and select the “Generate project page” link. I prefer the latter approach because it’s easier and involves less typing!

Once you complete the steps from either set of instructions, you will end up with an extra local tracking branch called ‘gh-pages’. This branch contains the website for your project and it’s where we will be storing the plugin documentation.

The plugin documentation

When the grails doc command first appeared, it allowed you to automatically generate API documentation for your projects. With Grails 1.2, its scope was expanded and it can now generate a manual that looks and feels just like the Grails user guide. You don’t have to do anything to get the API documentation, but the manual requires a bit of work.

The basic idea is that you create a src/docs/guide directory and populate it with gdoc files. By prefixing the names of the gdoc files with appropriate numbers, you get a hierarchy of numbered sections. To get some idea of what you need to do, check out the source files that Luke Daley created for the Grails JMS plugin user guide.

Publishing the docs

Generating the documentation leaves you with a docs directory full of HTML files. What do you do with it? The next step is to get it onto your project-specific GitHub Pages site. Doing this manually is a bit of a bind, but I’ve created a Grails script and some template files that make it almost trivially easy (once you have the ‘gh-pages’ branch locally!).

First, copy this script to your $HOME/.grails/scripts directory. Next, switch to your ‘gh-pages’ branch and copy this HTML template to the root directory of the project. Next, create an index page for your site: index.html, index.textile, and index.markdown are all acceptable. Choose whichever format you prefer, but it should start with this section:

---
layout: main
title: My Grails Plugin
---

The three dashes are important! I’ll explain those and the layout/title later. Then,

$ git add main.html.tmpl
$ git add index.textile
$ git commit -m "Adding template layout and index page for GitHub Pages site."

In the above set of commands, replace ‘index.textile’ with the name of the index file you created.

You’re now ready to publish your plugin docs for the first time! Simply switch to the main branch and execute the new Grails command:

$ git checkout master
$ grails publish-github

Once the command successfully completes, you will find yourself back on the ‘gh-pages’ branch with all the documentation staged for a commit. You can then make any changes you want, commit everything, and then push the branch to GitHub. You can even do this in one step with the grails publish-github --push command, or if you want to automatically commit the documentation but not push, use grails publish-github --commit.

A few details

The publish-github command does a few things. First, it generates the plugin’s documentation via Grails’ doccommand. It then copies the generated documentation to the ‘gh-pages’ branch (using an rather ugly technique). Once it’s on the ‘gh-pages’ branch, it generates a new ‘main’ layout based on the main.html.tmpl file. It basically reads the information from the XML plugin descriptor, plugin.xml, and populates the template with things like the plugin version and author. The template is then generated to _layouts/main.html, which you can reference from your individual pages.

It’s up to you to write the site’s pages, but at least you have the option of three different syntax types. I prefer Textile for the moment, because it’s very similar to the gdoc format. The key part of each page is the header block, which you’ve seen already:

---
layout: main
title: My Grails Plugin
---

The ‘layout’ value is used to determine which file under _layouts to inject the page into. Ideally, the index page should use the _layouts/main.html template that the publish-github command generates, hence the value ‘main’. The ‘title’ option is specific to the generated template, which inserts the corresponding value into the title of the page. This is because it’s tough to get a suitable plugin name with correct capitalisation from the plugin descriptor.
Update Somebody has pointed out that the publish-github command doesn’t work if plugin.xml is committed to your main branch (typically master). You could try modifying the script to handle this, but the XML plugin descriptor doesn’t need to be in GitHub at all – it’s generated after all – so I would simply git rm it.

That’s pretty much it. You also have the full power of Jekyll (as described on the GitHub site), so you can build more elaborate sites as you see fit. If you want to see the results of all this jiggery-pokery, then take a look at the GWT plugin pages.

Until next time.

Java’s new keyword

Creating objects in Java is easy with the new keyword. In fact, it’s one of those things that you don’t think about. Need to access a file? Just create a new File instance: new File("build.properties"). For most Java developers, that’s all they need to know. Life becomes more interesting, though, when you start working with multiple class loaders.

Class loaders? Argh! Run away, run away!

That was pretty much my reaction for many a year. I just didn’t want to know about them. They were some kind of black magic and always Somebody Else’s Problem. It’s strange, because class loaders are actually pretty straightforward. Most Java developers know that you compile Java files to these *.class files and that those compiled classes have to be loaded by the JVM somehow. That’s basically what the class loader does. But like threads, the problem is not understanding what they do, but getting them to work together.

How many times have you heard the phrase “it’s a class loader issue?” I’ve certainly heard (and said) it more times than I’d care to admit. As soon as you have more than one class loader in an application, you have to start worrying about which classes can “see” which others. It can easily become a nightmare. But class loader behaviour is perhaps a post for another time. Let’s get back to new.

So, the first time that you create a new object, the JVM has to first load the class. This happens transparently when you use new. The question is, what class loader is used? And why does it matter?

Consider a scenario from Grails. We have a build system based on Gant that loads build scripts and executes them. In one of them, we instantiate a Jetty server and start it. The sequence of object creation goes like this:

In fact, the above is a simplification of what actually happens, but it suits the purpose of this post.

The JARs for the first three classes are all on the classpath of what we will call the build class loader. This loads all the classes used directly by the build. So what about Jetty’s Server class? The most important thing to understand is that the Server class must be loaded by the same class loader that loads the Grails web application. Although you can pass your own class loader to the embedded server, if it’s different to the one that loads Server you’ll run into those dreaded class loader issues.

Bearing that in mind, let’s look at what happens if the RunApp script uses new to create the server instance:

def server = new org.mortbay.jetty.Server()
...
server.start()

Right about now, you should be asking yourself “what class loader was used to load the Server class?” It’s a critical question because it determines what class loader is used to load the entire web application and hence what classpath the application’s runtime dependencies should be on. In this case, the class loader used is whichever one loaded the RunApp script. The new operator effectively delegates to this.getClass().getClassLoader().

What does that mean for our example? It means that the build class loader is used to load the Server class and therefore must also be used to load the web application classes. In other words, all the application’s runtime dependencies must be included in the build class loader! What’s the problem with that, you may ask. There is one potential problem and one actual.

The potential problem is class conflicts. What if the web application depends on a different version of a library that’s already on the build system? It’s a particular problem if any of the Apache XML API libraries are on the classpath. These cause absolute havoc.

The other problem is that the more JARs you have on the classpath, the longer it takes for the JVM to find the class it’s after. That means longer start up times. It’s one of the problems OSGi was designed to solve (he was told by a man in the pub). Why put JARs on the build classpath that the build itself doesn’t need?

The solution is to work out where you want a class loader boundary and use reflection to instantiate your object:

def runtimeClassLoader = new URLClassLoader(...)
def server = runtimeClassLoader.loadClass("org.mortbay.jetty.Server").newInstance()
...
server.start()

This is pretty easy in Groovy because the start() method is evaluated at runtime, but Java needs to know the type at compile-time. You can’t do this:

ClassLoader runtimeClassLoader = new URLClassLoader(...)
Server server = (Server) runtimeClassLoader.loadClass("org.mortbay.jetty.Server").newInstance()
...
server.start()

because you’ll get a ClassCastException on line 2. The declared type of server is loaded by this.getClass().getClassLoader(), whereas the new instance is loaded in a different class loader. Different class loader means different classes. So you have to use reflection to invoke the methods and access the fields you need. Fortunately, you only have to jump through these hoops at class loader boundaries.

As you’ve seen, the new operator is normally something you don’t have to think about, but as soon as you start dealing with multiple class loaders, you have to be aware of and understand its behaviour. The trick is to work out suitable class loader boundaries and then use reflection to load and instantiate classes at those boundaries. It may sound like unnecessary extra work, but you can gain real improvements in application/framework reliability. If you’re lucky, things may even run a bit faster

A Tomcat gotcha

Servlet filters. Straightforward, right? A request comes in, goes through each filter in the chain, hits the servlet and then the response goes the other way. Sort of. What could possibly go wrong?

Not much…unless you throw error handling into the mix. The trouble is, the servlet specification doesn’t lay down many rules on how error pages are invoked. My expectation was for the servlet container to transparently offload the request to the error page and then execute the filters in reverse. In other words, the filters shouldn’t be or need to be aware that the request has gone via the error page. The request should behave like any other request.

Tomcat seriously disabused me. It takes a different approach in which the main request almost completes (all the filters execute in reverse) before the request goes to the error page. The following diagram should make it clear what I mean:

Tomcat & Jetty error handling

Having read the servlet specification, it does make a bit of sense. One of the few requirements is that the servlet container should pass the original, unwrapped request and response to the error page, unless a filter is configured to execute for the ERROR dispatcher. Unfortunately, if you add something to the request in your filter, or attach a variable to the thread, it won’t be available to the error page if your filter also removes that data.

That may all be a bit too abstract, so how about a concrete example? Some of the filters in Apache Shiro bind an object called the security manager to the request’s thread. Any security code can then easily grab it on demand. When the request has finished, the filter removes (unbinds) the security manager from the thread. That’s just good practice. But what happens if the error page wants to access the security manager?

Hopefully it’s obvious from the discussion so far that you need to add Shiro’s filter to the ERROR dispatcher as well as the REQUEST one. This should work. When the servlet container triggers the error page, it first executes the filter. But that’s not what happened when I tried it. Instead, the error page threw an exception saying that no security manager was bound to the current thread.

What had gone wrong? After some debugging, I discovered that a super class of the filter, OncePerRequestFilter, was skipping execution of the filter if it detected that it had already been processed. How did it do that? By checking whether a particular request attribute was set. In this case, the attribute was still there from the main request, but the security manager had been unbound from the request thread.

This problem didn’t exhibit in Jetty at all, which is kind of annoying because you would hope that code that runs on one container would also run on another. Fortunately, there is a simple moral to this story: filters should always clean up after themselves when they finish and never leave stuff in the request, session, or thread.

Grails 2.0 #1: Gradle for the win?

Ever since I attended a Gradle course, I’ve been rather obsessive about it. In particular, I’ve been looking at integrating Grails into Gradle builds. The early result was a straightforward Gradle plugin that allows you to build Grails projects. I’ve explained how to use it in a previous post, so I won’t talk about that here. It works, but it bothers me that the plugin delegates to the Grails build system. That interaction is always going to be anaemic and will almost certainly frustrate any Gradle power users. But is there any other approach that can work?

The problem

Grails has a custom build system based on Gant that has evolved over the years. The nature of the build and its interaction with Grails plugins means that it’s very difficult to integrate well with other build systems. It does its own dependency management (although it does delegate the grunt work to Ivy) and it has its own event system.

The trouble is, the build is definitely getting more bloated and complex as features are added. In fact, many of those features, such as dependency management, distinct compile/test/runtime classpaths, and artifact publishing, are built into a tool like Gradle. So we’re basically reinventing the wheel to some extent. And I have to admit that Gradle’s API is far more robust and flexible than the Grails build system.

This got me thinking: would a native Gradle build for Grails projects work? After all, build is not one of Grails’ core competencies. Why not leave it to a dedicated tool if that tool can handle Grails’ special requirements? Graeme and I had thought about doing this some time ago, but decided against the move. Gradle was still young and didn’t seem to offer enough benefits. Times have changed though.

My next thought was this: what is a Grails project if not a Groovy WAR project with a rather unique directory structure? If Gradle can handle that directory structure, why not use its standard compile tasks? Doing so would mean that other Gradle plugins would also work with Grails projects.

The solution

On that Gradle course, I discovered a Gradle feature that really sold me on its potential as a build tool for Grails projects: source sets. These are arbitrary groupings of source files that can pretty much manage any directory structure you can throw at them. Even a Grails project. All I had to do was create a test Grails project with a Gradle build, apply the ‘groovy’ plugin, and configure the source sets. Next thing I knew, I could compile a Grails project. Whoohoo!

If only everything was so simple. Trying to run a Grails application proved to be a far tougher challenge. This required compilation and loading of all installed Grails plugins, including the core plugins. On the bright side, I had an opportunity to create a build-specific plugin manager that didn’t have to worry about runtime stuff. This meant a great simplification of the code.

The result of all this work is another Gradle plugin, although this one hasn’t been published yet. It’s in a very raw state at the moment, and I want to avoid confusion with the existing Gradle plugin I mentioned at the beginning of the post.

About the plugin

So how does this plugin work? The core of it is the grails.build.plugin.GrailsPlugin class, which is the entry point for the plugin. It does several things of note:

• apply the ‘groovy’ and ‘war’ plugins to the project;
• configure the source sets;
• sets up some extra dependency configurations; and
• creates a whole bunch of specialist Grails tasks and creates the dependencies between them.

The most significant tasks are:

• buildData – instantiates BuildSettings and loads the build configuration
• buildPlugins – builds the project’s installed Grails plugins and loads them
• generateWebXml – generates the web descriptor to ./web-app/WEB-INF/web.xml
• generateApplicationContextXml – generates the root application context descriptor to ./web-app/WEB-INF/applicationContext.xml
• packageI18n – copies the i18n resource bundles and converts them to ASCII if required
• run – launches the configured servlet container

As you can see, the plugin already handles Grails plugins, although there is still much that doesn’t work. The fact that the plugin can fully build plugins of various types and add their dependencies and classes to the appropriate classpaths is a big deal though. This just doesn’t happen with the current Grails build system.

Another really neat feature is that Grails plugins can provide their own build files and Gradle plugins. You can see an example in the Jetty plugin that’s packaged in the root of the grails-build-x project. This is incredibly powerful stuff because it means that the Grails plugin can have full access to the Gradle build and do pretty much anything! For Jetty, this means adding the container starter classes to the servletContainer configuration and adding its dependencies to the project’s runtime classpath. But pretty much anything is possible.

The current run task demands some extra requirements of servlet container plugins, so that standard ones won’t work. However, the custom Jetty plugin I referred to earlier can also be used in a normal Grails project with the Grails build system. That means a plugin author can add features that help it integrate with the Gradle plugin without breaking normal Grails projects!

So what does it all mean?

The Grails build system evolved out of a failure on the part of existing build tools to provide a solution to building Grails projects, particularly once plugins were introduced. At the time, I think Gradle was still an idea in the air. Since then, the build system has grown in features and complexity, and it works pretty well.

Now, though, I think it’s time for it to retire. It doesn’t make sense for Grails developers to spend a lot of their time on build stuff when another tool is now available that will do most of the work itself. The build system also doesn’t play particularly well with other build tools, which can make adoption in the enterprise rather tricky. And a clean room implementation means clearing out a lot of cruft and being able to resolve some outstanding and difficult problems.

The great thing about the new Gradle build is that it can be developed in parallel to the Grails 1.x line. And if it makes it into Grails 2.0 as the build system, then hey, the plugin authors have already prepared and tested their plugins for it. Certainly my hope is that Gradle forms the core of the build system for Grails in the future. Fingers crossed!

Building a Grails project with Gradle

Earlier this week, I attended Hans Dockter’s first ever Gradle course. Everything seemed to click into place and it inspired me to have another crack at writing a Gradle plugin for building Grails projects. A lot of the hard work had already been done to get the Maven plugin working reasonably well, so all I had to do was make use of the helper classes that already existed for bootstrapping the Grails build system.

So how do you use it? Simple. Here’s a basic Gradle build file (build.gradle) that you can put in the root of your Grails project:

buildscript {
    repositories {
        mavenCentral()
        mavenRepo urls: 'http://snapshots.repository.codehaus.org'
        mavenRepo urls: 'http://download.java.net/maven/2/'
    }

    dependencies {
        classpath "org.grails:grails-gradle-plugin:1.0-SNAPSHOT"
    }
}

apply id: "grails"

repositories {
    mavenCentral()
    flatDir dirs: "lib"
}

configurations {
    compile.exclude module: "commons-logging"
}

dependencies {
    compile "org.grails:grails-crud:1.2.0",
            "org.grails:grails-gorm:1.2.0",
            "org.apache.httpcomponents:httpclient:4.0",
            "org.apache.httpcomponents:httpmime:4.0",
            ":apache-solr-solrj:1.4:dev",
            ":pgptools:0.1"

    // Required by the Joda Time plugin.
    compile "org.springframework:spring-web:3.0.0.RELEASE"
    compile "org.springframework:spring-webmvc:3.0.0.RELEASE"

    // Required by the JMS plugin.
    compile "org.springframework:spring-jms:3.0.0.RELEASE"

    runtime "org.slf4j:slf4j-log4j12:1.5.5",
            "hsqldb:hsqldb:1.8.0.5",
            "postgresql:postgresql:8.3-603.jdbc3",
            "org.bouncycastle:bcpg-jdk15:1.44",
            "net.sf.ehcache:ehcache-core:1.7.1"
}

Unbelievably, that’s it! You can now execute any of the Grails commands via Gradle like so:

gradle compile
gradle create-domain-class -Pargs=org.example.Book
gradle run-app
gradle run-app -Pargs=--https -Penv=prod
gradle create-gwt-action -Pargs="--shared-pkg=client org.example cmd ListBooks"

The -Pargs option allows you to pass any arguments you want to the corresponding Grails command, while -Penv allows you to override the default environment. In case you’re wondering, the -P option allows you configure project properties. args and env are simply two that are understood by the Grails plugin.

So what’s happening in this build file? First of all, we use the buildscript {} section to configure the plugin as a dependency of the script. Once the plugin is on the classpath, we can apply it to our build via apply id: 'grails'. Alternatively, you could download the plugin JAR file and put it into $GRADLE_HOME/lib.

The rest of the build file is concerned with setting up the dependencies for our Grails project. Quite a few are declared in this example, but the only requirement is that you declare dependencies on at least one Grails module, a logging implementation, and the JDBC driver(s) for whatever database(s) you’re using. If you’re using the Hibernate second-level cache, you should also include EhCache or OSCache.

The true beauty of this plugin is that *all* the Grails commands are available to you, including those provided by plugins. This is far better than the Maven plugin, which has a set of hard-coded goals for the most common Grails commands and a generic exec goal for executing arbitrary Grails commands. I haven’t tested all of the commands, so feel free to report any problems.

The code for the plugin can be found on GitHub. I think you’ll be surprised at how little code there is. In fact, there are only two files: build.gradle in the root of the project, and GrailsPlugin.groovy under src/groovy. I definitely recommend taking a look at the plugin file to see how Gradle’s dynamic tasks made it so easy to support all the Grails commands.

Give it a go and see what you think!

Update First off, the plugin currently only works with Grails 1.2.0+ and Gradle 0.9 snapshots. Second, I’m aware that the build script doesn’t work at the moment. Unfortunately, there appears to be a Gradle bug preventing the fixed build script from working. Stay tuned.

Update 2 My bad. The version of mavenRepo that takes more than one repository URL expects the first one to contain the POM, while the other ones are for getting hold of the JARs. I have corrected the example build so that it has separate mavenRepo lines for the Codehaus Snapshots repository (for the plugin) and the java.net repository (for GParallelizer),

Update 3 I have uploaded a new version of the plugin that allows you to start from scratch. Create a new directory for your Grails project and add the following build.gradle file to it:

buildscript {
    repositories {
        mavenCentral()
        mavenRepo urls: 'http://snapshots.repository.codehaus.org'
        mavenRepo urls: 'http://download.java.net/maven/2/'
    }

    dependencies {
        classpath "org.grails:grails-gradle-plugin:1.0-SNAPSHOT"
    }
}

apply id: "grails"

version = "1.0-SNAPSHOT"

repositories {
    mavenCentral()
    flatDir dirs: "lib"
}

configurations {
    compile.exclude module: "commons-logging"
}

dependencies {
    compile "org.grails:grails-crud:1.2.0",
            "org.grails:grails-gorm:1.2.0"

    runtime "org.slf4j:slf4j-log4j12:1.5.5",
            "hsqldb:hsqldb:1.8.0.5",
            "net.sf.ehcache:ehcache-core:1.7.1"
}

From the new directory, run gradle init and a fresh Grails project will be created for you.