Google Data API, Google App Engine and Scala/Lift

Recently I decided to bite off this chunk of coding. What followed really emphasized and reinforced the title of this blog. It was quite frustrating at times, frequently interesting and in the end, rewarding. I did finally get most everything to work, so here are my notes...

The objective of the particular web site I did was to have the user enter a handful of values via the web page, run a model calculation based on those values and display the results. Prior to my involvement, my colleague had done all the modeling in Excel and VB. His next step was to export the data to XML files and re-code the calculations in Java.

My first step was to make the site run with Scala and Lift and let the Java code read the XML files. No problem.

Step 2: host the app somewhere. I had read a lot about Google App Engine (GAE) and knew people who had good luck with it. Great idea.

First roadblock: I had read GAE doesn't support static data files, but that's no problem there's the GData API, and all these XML files came from spreadsheets anyway, so how hard could it be?

First sidetrack: instead of reading static files and short of migrating to the GData API, it seemed I should be able to read XML files from the Jar files: ClassLoader.getResource("xmlfile_name"). I spent many hours and never did get this to work!

At this point it became clear - the GAE support for Java seems to be WAY behind the Python support/documentation/functionality/etc.

GoogleData API
It seems there are several variations on this, but the REST API model seems to be quite adequate. There are however a handful of glitchs/gotchas it seems.

1) I created a user which had only view authority on a spreadsheet. Unfortunately, it could *not* read that spreadsheet with the REST api.
2) Everything comes back as a string. This is not too bad, but it seems a more "robust" API would provide a bit more flexibility.
3) The column headings are used as the key to a map of the values in each row. This is a decent strategy, however I would like access to the actual value in the header, not the munged value used in the key. Again, not a complete deal-killer.

Google App Engine
I did get everything running just fine with the Gdata api within those constraints. Now it was time to deploy. I had read several people deployed with no problem and sure enough, there are a handful of apps running with Scala/Lift.

Would mine deploy? Of course not. It generated a couple stack traces indicating problems in the LiftFilter, but seemingly in the configuration of the logging. Might that be just a 'warning'? You would think so, but NO. What the heck? Well, 5 combo boxes aren't that big a deal, so I rewrote them in JSPs and it deployed just fine. As always, appetite grows while dining and the client wanted the drop downs to repopulate based on certain selections. No problem - but wait! That was why I wanted to use Lift in the first place! Ack. The long and the short of it was all I had to do was get a more recent version of Lift (specifically 1.1-SNAPSHOT), and it deployed just fine!

Miscellaneous Glitches
Because it is a proprietary application, I set the security of the main site to be limited to the registered users of a domain I had authorized in Google Apps. This works quite well - the user is redirected to a login page, then to the web site. Cool. Now I added a page to allow an admin user to reload the app. The documentation describes a really easy way to do that. The fine print I didn't digest the first time around is that the members in the admin role are those users defined as admins of the application. Oops. I'm the admin of the application and I created it from my main Google account which is NOT in the aforementioned domain. Oh, I can add admins, so email has been sent inviting another user I created in that domain. Now, where did that email go?

Pleasant Surprise
All the while I was developing the app, I was wondering about releasing new versions after it was in production. I only uploaded code a couple times before I encountered the concept that GAE has the idea of versions of your application. Several versions can be stored on the server and run independently and simultaneously! Well done, GOOG.

Bottom line: Google App Engine and GData API are a good way to go. Better if you're using Python, which is not a bad choice. I just have a lot of legacy Java code, so I'm really liking Scala and Lift. Now I know, those are GAE-compatible also, so life is good.

Another Reason to look at Scala for GUI

It's just a simple thing - a form GUI to edit domain objects stored in XML. We even have a schema to describe those docs pretty well. Certain tags have one of a pre-defined list of values ("restriction" in XML schema terms). To edit this, a drop-down list is a very handy thing. How would you populate that? Well an enum sure looks like an attractive thing and these days in Java, enums have come a long way so that you can give the enum a handful of properties.

So I copied each set of restricted values out of the XSD file into a Java file and created enums - dandy! The first small problem we created for ourselves is that the restricted values have dashes. Gosh, as enum identifiers dashes look a whole lot like minus signs. So now I have to map the XML value to the enum value somehow.

    enum SensorType {
      accelerator,
      altimeter
    }

But it's very easy to populate the combo box:

inputSensorCombo.setModel(new DefaultComboBoxModel(SensorType.values()));

So I have to add the mapper, but while I'm at it I would like to have a different
display value in the combo box that has blanks!

Still not a big problem - create the enum class with a display value and XML tag value:

enum SensorType {
    accelerator("Accelerator", "accelerator"),
    altimiter("Altimiter", "altimiter");
    final String displayValue;
    final String xmlTagValue;
    SensorType(String displayValue, String xmlTagValue) {
       this.displayValue = displayValue;
       this.xmlTagValue = xmlTagValue
    }
    @Override
    public String toString() {
       return displayValue;
    }
}

With the toString override, populating the combo box is still the same. Pretty good, yeah? Now, I have about 4 of those. We're OO so just make a base class, right? Sorry. As 'syntactic sugar' enum already extends Enum, so we're SOL.

Bummer. Let's try Scala.... next post.

Do developers know branding?

I think we learn branding or marketing "spin" at an early age. Here's an amusing post in Toni Epple's blog showing he clearly has: "122 Platform Exercises New & Free - First-come, first-served!"

The evolution of SCM or Why I would like any teams I'm on to use Git

Back in the olden days of Source Code Management (SCM) there was SCCS and the original Unix things. These depended on developers checking out code with a LOCK. Because, after all, if you didn't have a lock and you changed things, badness would follow.

Well, then along came CVS and out went the idea of locking. Why? Merging became easy - in fact CVS would do it for you. Ok sure, sometimes there are conflicts and you have to merge yourself... but in fact, this is rare (at least for small to medium teams I've been on).

So then we upgraded to Subversion - not really substantially different than CVS - still lock-free central repository.

In this world (at least on the projects I've been on) you always check in against the trunk - i.e. the main line of code. Branching the code is still icky and to be avoided.

Now, along comes Git. (N.B. this discussion may apply to Mercurial also, i have no experience with that, but I know it's popular in many Java (i.e. Snoracle) circles). I don't have tons of experience with Git yet, and I may have forgotten the wonderful things the guys from Everlater.com came over to @iVolunteer to teach us (http://tr.im/gitevrl8r) - BUT I'm convinced of the Git-goodness!

There are two basic revolutionary aspects to it, from what I can tell:
1) no central repository. In some sense you use convention over configuration to determine who has the "central repository". Yes, you need to take backups somewhere. Yes, you need to figure out which repository to check out from for doing a build. However, the good news is that this can be *anybody* "in the network". Because everybody has a copy of the whole history (more or less), *anybody* can do the build.
2) local repository. This means several things, not the least of which is that if you're developing code you can check in locally and not break the build. How does that work? You have the whole repository. But what about everybody else's code? You have all the history you've requested from them. If you haven't "pulled" from their local repository, yes, you're hosed. So pull it and then do a build. Or have them pull from you and do a build - it just doesn't matter - and that's the beauty of it. So just as CVS got rid of the notion of merging being hard, Git gets rid of the notion that branching is hard.

Variations on a theme:
- sure, it's nice to have a publicly accessible, backed up, repository. That's where Github comes in (and presumably whereever you want to install the equivalent. But wait, doesn't that make it just like SVN? Well, ok - yes, to some degree.
- there is much finer granularity of changes. because you can check in locally, and then I can pull from you, I can't get changes from someone that I might want to integrate with before (possilby) screwing up the whole thing.
- and even finer, (this may be a GitX thing) you can check in just *some* changes to a file. I had occasion where I changed a properties file (IIRC) and wanted to check in the new properties, but NOT the hard coded path to my specific files. I checked in just some of the changes to the file - sweet.

now back to figuring out how to update and check in...

It's TOO great a time to be a geek!

... especially if you can't decide what to do first. I was about to Tweet:

STILL looking 4 desktop or web client that shows msg thread: TweetDeck- click on "in reply to" Nambu shows *future* replies. Huh?

Last time I whined about that or similar, the reply I got (you know who you are!) was why not do one (write a client) yourself? OK, OK - i will. But that's the problem: I'm taking the Stanford class on iPhone development (http://www.stanford.edu/class/cs193p/cgi-bin/index.php), where a big homework assignment is to write a Twitter client, but that's 4 weeks starting 4/20.

But, wait - Dave Briccetti published a cool sample app showing how to do this in Scala (http://briccetti.blogspot.com/2009/03/simple-twitter-client-bay-area-scala.html)

And of course, there's the JavaFX version - TwitterFX (http://kenai.com/projects/twitterfx)

But that's after the iVolunteer project I signed up to help, and my day job!

Bottom line: Less Twitter *reading* and more Twitter *doing*!

Java WebStart success?

Just maybe, I'm on the verge of success gaining the Holy Grail: automatic continuous integration of a Java WebStart application! What are the pieces?

• Maven and Ant
• Artifactory
  
• Hudson
• Jetty
• JNLP servlet

Whew! Some details on each piece...

Ant and Maven
I like Maven for the convention it brings (as in Convention over Configuration). The repositories and versioning of dependencies are wonderful. The bad news is that a WebStart application isn't the traditional JAR or WAR build that Maven knows about. (N.B. - I may come back to this, since using the JNLP servlet, it *could* be a WAR file.) The good news would seem to be that of course, there a couple plugins that do neat WebStart things - including building the JNLP file from the dependencies. You know, DRY! The first one from CodeCrate offers great promise - handle the simple case! Cool. Well, it's a fail straightaway since it can't seem to find the dependencies at all and gets a ClassNotFoundExeception. Bummer. So I go for the full Monty - Maven Webstart Plugin . There's a "cookbook recipe" which shows a "Getting started" page. But no examples for the simple case - they go straight to WAR bundles. :( And the Maven jar signing plugin will only sign the main artifact - not all the dependencies. Double fail.
Ant to the rescue! Fortunately, Ant knows how to sign jars and Hudson will run multiple steps in the build, so it is easy to mix Ant and Maven. Ant will sign the dependency jars and Maven will build the rest of the project. The only loss is not generating the JNLP from the dependencies.

Artifactory, Hudson and Jetty
Both of these are handy web-based tools. Each is easy to install and configure - UNLESS you want password protection. My first thought is to bundle both into a Tomcat instance. Should be easy enough - both give instructions on that. Password protecting Tomcat sites should be easy enough. OK, start with Artifactory. Endless headaches, fails, NullPointerException, etc. Finally come across a suggestion to never use pre-installed Tomcat instances on Linux boxes. Oh! OK - download and install Tomcat 6.0.18 - works great! The password protection works. Now, just have to add the Hudson webapp. FAIL - OutOfMemoryError out the gazoo. Having used Jetty before with good success, I revert to separate Jetty installations - one for Artifactory, one for Hudson. Even with password protection!

JNLP Servlet
Providing links to a JNLP file and having it link all its resources works fine. Just one problem: the JNLP has the codebase in it, so at build/deploy time you have to know where you're deploying to. A bit of a problem if you want to build and deploy to a test server, then promote to production. But wait, the JNLP Servlet to the rescue: it filters the JNLP and drops in the proper codebase for you. Nice! Now let's see - where's the JNLP servlet?
The JnlpDownloadServeltGuide from the Sun JavaSE 6 TechNotes says: "The servlet is packaged into the jnlp-servlet.jar file, which can be found in the SDK under samples/jnlp/servlet/" OK - I've got lots of JDK/JRE installations.
find /usr/java -name samples
Nothing. Hmm... Oh, update 11 is out - let's download that. And there's a nice document in the Java Web Start Reference pages called "Packaging JNLP Applications in a Web Archive" - sweet. That's interesting, there's a "sample" directory in this JDK. Oh, even a "jnlp" directory under that. Cool - cooking with gas now. Follow all the instructions about deploying, etc. FAIL - java.lang.ClassNotFoundException: com.sun.javaws.servlet.JnlpDownloadServlet
WTF?
jar -tvf WEB-INF/lib/jnlp-servlet.jar
Oh look! All of them are "jnlp.sample" package, not "com.sun.javaws"! Thanks a lot. So change the servlet definition in the web.xml. Finally - success!

The only remaining piece is to have Maven or Ant copy the proper files to the Jetty webapp directory. How hard can that be?