Dear Future Me,

You’ve forgotten how to decode (or unescape) HTML or XML in Python again, haven’t you?  My, my, that old age does catch up with you.

Well, it turns out that xml.sax.saxutils.unescape() works like a charm.  I’m certain that edge cases lurk here and there, so caveat, um, coder.

I’ve been using my time at RepoCamp today to get the OAI-ORE plugin for WordPress validating again.  I’m having some trouble using the validator so I say that with some diffidence.  But the latest code which is now checked in to the WordPress plugins svn repo ought to be close, if not fully conformant, to the 0.9 version of the ORE spec.

I’m not sure the plugin is really useful; it’s just an Atom feed of all posts and pages in a WP instance.  I can think of some ways to make this more useful, by allowing blog authors to create their own aggregations, pulling in content outside of the particular instance.  I am certain that others can come up with even better uses.  I’m open to suggestions.

Thanks to Jay Datema for prodding me a bit, if indirectly.

In spite of some open questions, I’ve been making some progress on my Jython-based transport tool. Right now it’s pretty dumb and simple: it copies files to and fro via scp.

Being a newb at both Java and Jython made finding the right libraries a bit of a challenge, and so I’m posting some code here for folks in the same boat. It’s not particularly pretty due to 1) wanting to get something working very quickly, and 2) weird errors when I try to make things prettier (such as getting rid of the hard-coded bits), but I’ll resolve these soon.

The jython code:

# Biter.py
import java
from org.apache.tools.ant import Project
from org.apache.tools.ant.taskdefs.optional import ssh

class Biter(java.lang.Object):
    def __init__(self):
        self.keyfile = '/home/user/.ssh/id_rsa'

    def transport(self, from_uri, to_uri):
        "@sig public void transport(String from_uri, String to_uri)"
        scp = ssh.Scp()
        scp.setKeyfile(self.keyfile)
        scp.setPassphrase('')
        scp.setTrust(True)
        scp.setProject(Project())
        scp.setFile(from_uri)
        scp.setTodir(to_uri)
        scp.execute()
        print "%s -> %s" % (from_uri, to_uri)

I set my CLASSPATH to include /usr/share/jython2.2.1/jython.jar, /usr/share/java/jsch-0.1.28.jar, /usr/share/ant/lib/ant-jsch.jar, /usr/share/ant/lib/ant.jar, and ‘.’, and then compile to bytecode via jythonc -idp gov.loc.repository.transport Biter.py. Once it’s compiled, I can use the Biter class in Java code thusly:

# BiterClient.java
import gov.loc.repository.transport.Biter;

public class BiterClient {
    public static void main(String[] args) {
        Biter biter = new Biter();
        biter.transport("/home/user/tmp/test1", "user@example.org:tmp/test2");
        biter.transport("user@example.org:tmp/test2", "/home/user/tmp/test3");
    }
}

I should note that the passphrase and keyfile are not necessary if you’re passing valid username and password credentials in the remote to_ or from_uri (and the server in question supports password auth). But pubkey auth is useful for our purposes and that’s why it’s in there.

Hopefully someone somewhere finds this helpful. :)

It’s hard to believe but I’ve been at the new job for six months already, a full half-year come the 29th. Some days it seems like I’ve been here forever; others like I’m still a rank newb. I haven’t written terribly much about what I’ve been up to (but I assure you I’ve been busy). Let me rectify that.

The Transfer Problem

Two of the projects I’ve been working on relate to a fairly general problem that we like to call “transfer,” which revolves around, well, transferring files to and fro. Sounds simple. Is simple. That is, until you start thinking about preservation and accounting for a highly heterogeneous network with idiosyncratic nodes, esoteric storage software, and differential firewall rules. And that’s where it gets interesting (and problematic). The transferring itself, or copying of files from one location to another which we call “transport,” is the easiest part. We like to use common tools in our environment. It makes life easy. And so good ol’ scp seems like an obvious choice to handle the job.

Since preservation is a core aspect of our “repository,” a term which I use loosely, we must build certain other functionalities into the transfer process: validation, verification, inventory, backup, ingest, and so forth. Every time a file is copied to a non-transient location, we verify the file against a (SHA1) checksum and record an event for auditing purposes.

Repository Workflows

Some steps in the transfer process are routine and best handled by machines. Thus we automate them with scripts and code. Others require human intervention. This introduces another key aspect of our repository needs: workflows. No two projects will have the same workflow and yet all will have some steps in common. We’re using JBoss’s jBPM library for workflow management and it is more than capable of handling our workflow needs. It allows us to model complex and varied flows in a robust and not ad hoc way; it does seem preferable to me to model our workflows via jBPM’s graphical editor (serialized to JPDL XML) rather than copying around blocks of code and otherwise modeling the workflow procedurally in business logic.

One of my coworkers (the author of this) designed a complete workflow system in jBPM last summer and I’ve taken on implementing, tweaking, and testing this system, which has required bootstrapping my sorry-arse Java skills and learning jBPM. Though I find it difficult to think in Java patterns and generally find it a burdensome environment, I’m quite impressed by jBPM. I’ve been working on various updates of and unit test coverage for the workflow system, which has been a crash course in a number of Java technologies and a perfect first task at LC, as it gets me into the guts of things. The Java stack we use for our workflow is highly abstracted and componentized, which is conveniently modular… but it’s also Java: fairly heavy and arguably not as agile as dynamic languages such as Python.

Two Great Tastes?

So recently we’ve begun to think about implementing some transfer and workflow components in Jython (Python written in Java). Why? The value of Jython is as follows:

1. Ease of deployment - Deploying jar files to existing JVMs in production environments (which we do not control) is a simple task, or at least simpler than some other options.
2. Interoperability - Our stack is primarily Java-based and so interoperating with existing Java components means not having to rewrite functionality in other languages. Jython allows Python to talk to Java and vice versa.
3. Familiarity - It’s Python, and we like Python. It’s the closest my team has to a lingua franca and so it increases the chances of sharing code, maintaining code, and so forth.

It does not come without its drawbacks:

1. Currency - The Jython project went moribund for a few years or so and the latest stable version is now 2.2.1. Compare that to the latest version of Python: 2.5.2. I don’t begrudge the Jython developers, though. I’m glad some folks picked the project up, dusted it off, and breathed new life into it. I am also glad that they’re skipping 2.3 and 2.4 releases and plowing right into a 2.5 release. Because of this currency issue, some Python libraries won’t work with the latest Jython and that means you’re stuck looking for outdated, potentially vulnerable Python libraries, or hooking into Java libraries for the same functionality (which inevitably means more lines of code). I’m no lines of code fetishist but it does militate against the goal of agility somewhat.One does have the option of living on the edge and trying out the 2.5 branch, but that seems out of step with an infrastructure that is supposed to preserve terabytes upon terabytes of our nation’s, and the world’s, intellectual property. It’s a responsibility I do not take lightly, as much as I’d like to be on the bleeding edge.
2. Interoperability difficulties - Talking to Java from Jython is a snap: just import java at the top of your script, and, assuming your classpath is copacetic, voila: you have access to Java libraries in your Python code! Talking to your Jython modules from Java code is, well, a little more complicated. Read on.

Despite the caveats it does seem a sane, reasonable, and potentially productive path to go down. Right? I am specifically looking to implement two workflow components in Jython: one for transport (wrapping Ant’s JSch library, which provides a slick scp API) and the other for automation of ZFS filesystem/volume creation on the staging server. Nothing arcane, nothing tricky, nothing fancy. So it must be easy! Right? …

Lessons Learned?

I’m beginning to wonder about the feasibility of using Jython to make bits of our Java stack more agile. Specifically, there are three ways to get at Jython code from Java:

1. Compile to bytecode/jar via the Jython compiler (jythonc) and reference your Jython objects and methods as though they were POJOs
2. Embed a Jython interpreter
3. Instantiate a (JSR-223) script engine

Option 1 is nice because you get object- and method-level interop. However, jythonc is unsupported and will disappear. This does not seem sustainable though I might be able to limp along a while. And there are signs of hope:

Though jythonc is going away, all of the capabilities it provides will be present in 2.5 in other forms. We’re adding functionality to expose Python classes as Java classes using decorators to replace the docstring class creation that jythonc provided, and we’re adding static compilation of proxy classes so regular jython can run in applets and other environments with restrictive classloaders. We’re definitely doing something about jythonc.

That doesn’t help much now, of course, but just because jythonc goes away does not mean my jars will stop working.

The suggested methods for option 2 [1, 2] seem to be more trouble than they’re worth. If the goal is more agile development for certain components, the reliance upon multiple, separate Java classes — an interface class and an object factory, in the examples listed — to get a ten-line Jython script working, this seems suboptimal, both inefficient and not straightforwardly maintainable; it seems, to me and my Java-dumb ways, rather baroque.

Option 3 [1, 2] is more appealing than option 2 as it does not rely upon these other classes specifically for Jython code. But the number of lines of Java code that must be wrapped around the Jython to get it working looks like overkill for the drop-dead simple scripts I’m writing — it might be easier, for instance, to just write the darn things in Java and be done with it. (Did I just say that?)

Conclusions

Options 2 and 3 are similar as both involve embedding Jython code, or referencing files with Jython code, and interpreting the code within Java. Generally, I worry that either option would obviate the benefit of agile Jython scripting because you wind up wrapping the code in so much Java. I offer two disclaimers to counter my objections:

1. Cleverer Java coders than myself could, I am almost certain, find ways to build abstractions (or abstractions of abstractions) to eliminate the “lines of code” and “many separate classes per Jython script” issues
2. The value of Jython in our Java environment increases proportionately with the complexity of the component — given the overhead, a short Java class seems easier and more straightforward to implement than a short embedded Jython script. On the other hand, there’s value in embedding a Jython script that’d be an order of magnitude simpler than its Java analog.

At least, that’s the state of my head right now re: getting Jython and Java to play nice. If I make any breakthroughs or give up entirely, I’ll post follow-ups.

I am but a Java philistine and a Jython neophyte, so I remain humbly open-minded. I would greatly appreciate comments, questions, corrections, smackdowns, sagacious advice, and so on.

Deploying Rails (to Apache servers) is about to get much easier.  Hopefully.

 http://www.modrails.com/ 

Deployment has long been the bugaboo with Rails, so this should bode well for the framework.

This is very rough, but here’s a WordPress plugin that provides a resource map for the aggregation of all posts within an installation of WordPress. I’ll be working on this some more, but for now, it does appear to work and validate (as Atom). Useful? If so, I’ll zip it up and commit it to the wp-plugins svn.

Note:Ed reminds me that xsltproc can be used to transform the Atom-based resource map into RDF via GRDDL:

xsltproc http://www.openarchives.org/ore/atom-grddl.xsl http://lackoftalent.org/michael/blog/wp-content/plugins/oai-ore/rem.php

Update: The plugin has its own page.

I seem to be having issues successfully submitting comments to certain WordPress blogs lately — or perhaps Akismet has finally decided to (rightly) classify my comments are spam? Anyone know of any Firefox / WordPress comment bugs? My comments seem to be submitted — there are no errors — and Firefox winds up on a link like “http://example.org/blog/foo-bar-whatever/#comment-12309″. Any ideas? At any rate, I’m left to comment via trackback for now.

Thanks for the nod, Winona. Hopefully you folks will get some good use out of the XQuery-based OAI-PMH data provider I’ve been working on.

I just want to clarify that only one small bit of the code is specific to X-Hive, and that’s a call to an extension that gets last-modified dates from the X-Hive service. We do not reliably store this information in the metadata itself, and so I needed to go this route. Some folks do store this in MODS or elsewhere in descriptive or administrative metadata. It should be a two-line change to short-circuit this behavior (xhive-exts:last-update() is only invoked in two places, I believe).

I’m currently working on adding EAD support, modularizing things a bit more, and streamlining configuration. resumptionTokens will come after that, I hope.

I’ll be interested to hear more of UVM’s implementation and how I can make this thing more useful to others.

I proposed an NJ Library BarCamp some months ago, not realizing that efforts were already under way to do the same in NYC. In retrospect, I’m glad I didn’t do anything to get things moving; I wouldn’t have pulled things together nearly as well as the NYC folks did. The event was excellent. It was my first camp, and I’d definitely try another. A big thanks to Stephen Francoeur et al.

Here are the three sessions I attended, with links to the “official” wiki pages for summaries:

• Solr and Lucene (session moderated by AIP’s Mark Matienzo and NYU’s Jason Casden) seem to be gaining momentum in the library world. Having gone to the last Code4Lib conference, my head was already chock full of relevant tidbits, but the moderators did a great job of showing examples, evangelizing, and keeping the discussion going.
• Grid Services (session moderated by OCLC Openly Informatics’ Eric Hellman) might have been very interesting if I hadn’t kept receiving phone calls from an insurance company. I had to take the calls, and so this session was difficult to follow. The basic idea was to think of networked library services like the power grid. What would libraries want from the grid? What would they be willing to contribute back?
• Semantic Web (session moderated by NYU’s Corey Harper and CUNY’s Sunny Yoon) was the most widely attended session I went to: standing room only! When I first added the topic to the wiki, I had no idea it would draw this many people. Odd that I would suggest this topic since I had little to offer on the topic, so I gleaned an awful lot. The discussion was spirited and, as you might expect, the RDF vs. microformats arguments flew fast and furious across the room. I’m left wondering if the RDFa/GRRDL approach might not be a good middle-road between the “everything must be represented as RDF in a triplestore” camp and the “just embed microformats in xhtml” people.

And now, the requisite name-dropping. I got to reconnect with a bunch of people I hadn’t seen in a while, like Terry Catapano, Jay Datema, Nicole Engard, Valerie Forrestal, Kevin Reiss, and Sunny Yoon. And I got to meet LibLime’s Chris Cormack, NYPL’s Josh Greenberg, Corey Harper, Mark Matienzo, Jenkins Law’s RayAna Min Park, and Steven’s Tech’s Linda Scanlon, among other people.

It was about as good as any camp without kayaks and archery can be. Check out some more summaries.

1. Ruby

I gave a little talk on Ruby at today’s VALE-NJ Technology Awareness Group meeting, and so I’m using my slides to finally kick off this vaporseries.

[Slides are embedded as Flash here if you're reading in an aggregator.]

It seemed to be taken pretty well, though it could easily have been dismissed as the fanboyish ravings of a neophyte Rubyist. The folks from Rutgers were interested in having me come talk to their software architecture group, so it couldn’t have gone too badly, I suppose.

I need to stop cramming so much into presentations. Rambling about reflection and OO at a speed the Micro-Machines Guy would envy isn’t the best way to share enthusiasm about Ruby.

The rest of the meeting was interesting: Ron Jantz, the Digital Library Architect at Rutgers, posed a number of challenging questions about trust, authenticity, and reliability in the digital realm; Terry Catapano, Special Collections Analyst at Columbia, exposed the lack of data model inherent in many current metadata schemas and suggested ontologies as a potential direction; and Jeffery Triggs, Applications Programmer at Rutgers, demonstrated the Java DjVu Viewer Applet for displaying DjVu-based images outside the DjVu browser plug-in. Many good questions were asked, and new faces were in the crowd, so the future of TAG seems to be bright.

I’m hoping to follow up my Ruby talk with another on Rails. That was my initial goal for today, but there was far too much to cover, knowing the audience was relatively new to both technologies.

Matt Zumwalt of MediaShelf, LLC has been hard at work thinking about how to make Fedora RESTful. There is now a proposal on the Fedora wiki based on a PDF he sent to the fedora-commons-developers list.

It’s an interesting proposal. I’ve read over the PDF version quickly but it does bear a closer read.

Whether SOAP or REST is more appropriate for a Fedora API is something I’m not sure about, though I don’t mean to imply it’s an either/or situation.