Being a nerd, I tend to like the command-line. When I'm working on my laptop at home, I tend to like listening to music. Before I discovered that mplayer had a really convenient shuffle idiom, I would invoke it thusly (to listen to all my Pavement tracks in shuffle mode):

export IFS=$'\n'
for track in $(find /mnt/upnp/MediaTomb/Audio/Artists/Pavement -name \*.mp3 | ~/bin/shuffle.py); do mplayer $track; done

And the wee shuffle script I whipped together looks like this:

#!/usr/bin/env python
# shuffle.py
 
import sys
import random
 
args = list(sys.stdin)
random.shuffle(args)
sys.stdout.writelines(args)

And here's the convenient shuffle idiom that renders my arg-shuffling script somewhat useless:

find /mnt/upnp/MediaTomb/Audio/Artists/Pavement -name \*.mp3 | mplayer -playlist - -shuffle -loop 0

I've been periodically poking at getting Linked Data/RDF views hooked into the World Digital Library web application, following Ed Summers' lead from his work on Chronicling America. The RDF views also use the OAI-ORE vocabulary to express aggregations — in WDL, an item is an aggregation of its constituent files. The goal is to provide a semantically rich and holistic representation of a WDL item (identifier, constituent files, metadata, translations, and so on).

The ORE format is a new one for me so it's hard to say whether the output of my dev branch is valid ORE or not. Plus I'm a sucker for validators. Turns out Rob Sanderson has developed a Python library for validating ORE, and this little snippet is what I've been using to validate the ORE. I didn't put much effort into making it readable, so much as banging something functional out so I can meet deadlines, so mea culpa and all that. But without further hemming and hawing, the code:

# validate.py
import sys
from foresite import *
 
rem = RdfLibParser().parse(ReMDocument(sys.argv[1]))
aggr = rem.aggregation
n3 = RdfLibSerializer('n3')
rem2 = aggr.register_serialization(n3)
print rem2.get_serialization(n3).data

Most of this code is naively copied and pasted from Rob's excellent Foresite documentation.

I invoke it thusly: python validate.py {URL}

And the output:

@prefix _27: <http://www.semanticdesktop.org/ontologies/nfo#>.
@prefix _28: <http://localhost/en/item/1/id#>.
@prefix _29: <http://localhost/en/item/1/>.
@prefix bibo: <http://purl.org/ontology/bibo/>.
@prefix dc: <http://purl.org/dc/elements/1.1/>.
@prefix dcterms: <http://purl.org/dc/terms/>.
@prefix ore: <http://www.openarchives.org/ore/terms/>.
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>.
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>.
@prefix rdfs1: <http://www.w3.org/2001/01/rdf-schema#>.
 
 _28:ResourceMap a ore:ResourceMap;
     dc:format "text/rdf+n3";
     dcterms:created "2009-07-31T14:23:31Z";
     dcterms:modified "2009-07-31T14:23:31Z";
     ore:describes _29:id. 
 
 _29:id a bibo:Image,
         ore:Aggregation;
     dcterms:DDC "973";
     dcterms:alternative "Antietam, Maryland. Allan Pinkerton, President Lincoln, and Major General John A. McClernand"@en;
     dcterms:created "1862年10月3日"@zh,
         "3 de octubre de 1862"@es,
         "3 de outubro de 1862"@pt,
         "3 octobre 1862"@fr,
         "3 октября 1862 года"@ru,
         "October 3, 1862"@en,
         " ٣ آكتوبر، ١٨٦٢"@ar;
     dcterms:creator "Gardner, Alexander"@en,
         "Gardner, Alexander"@es,
         "Gardner, Alexander"@fr,
         "Gardner, Alexander"@pt,
         "Гарднер, Александр"@ru,
         "جاردنر, أليكسندر"@ar,
         "加德纳, 亚历山大"@zh;
... (and so on and so forth)
     dcterms:title "Antietam, Maryland. Allan Pinkerton, President Lincoln, and Major General John A. McClernand: Another View"@en,
         "Antietam, Maryland. Allan Pinkerton, el Presidente Lincoln y el General Principal John A. McClernand: Otra visión"@es,
         "Antietam, Maryland. Allan Pinkerton, le président Lincoln et le général-major John A. McClernand: Autre vue"@fr,
         "Antietam, Maryland. Allan Pinkerton,  Presidente Lincoln e Major-General John A. McClernand: Outra Vista"@pt,
         "Антитэм, штат Мэриленд. Аллан Пинкертон, президент Линкольн и генерал-майор Джон А. Макклернанд: Другой снимок"@ru,
         "أنتينام، ميريلاند ألان بينكرتون، الرئيس لينكولن، واللواء جون أ. ماكليرناند: منظر آخر"@ar,
         "安蒂特姆，马里兰州 艾伦·平克顿、林肯总统和少将约翰·A ·马克克拉南: 另一个视角"@zh;
     ore:aggregates <http://localhost/static/c/1/reference/04326u_thumb_item.gif>,
         <http://localhost/static/c/1/service/04326u.tif>;
     ore:isDescribedBy <http://localhost/en/item/1/item.rdf>;
     rdfs:seeAlso <http://hdl.loc.gov/loc.wdl/dlc.1>. 
 
 <http://localhost/static/c/1/reference/04326u_thumb_item.gif> a _27:FileDataObject;
     dcterms:format "image/gif";
     _27:fileSize "34531"^^<http://www.w3.org/2001/XMLSchema#long>. 
 
 <http://localhost/static/c/1/service/04326u.tif> a _27:FileDataObject;
     dcterms:format "image/tiff";
     _27:fileSize "1301614"^^<http://www.w3.org/2001/XMLSchema#long>. 
 
 ore:Aggregation rdfs1:isDefinedBy <http://www.openarchives.org/ore/terms/>;
     rdfs1:label "Aggregation". 
 
 ore:ResourceMap rdfs1:isDefinedBy <http://www.openarchives.org/ore/terms/>;
     rdfs1:label "ResourceMap".

You might pick up on some warts I have yet to fix, but there you go.

Context

Early on in the effort to develop the first public version of the World Digital Library web application, we developed a (non-public) Django-based cataloging application where Library of Congress catalogers could manage metadata for WDL items. Management in this sense includes creation of records, editing of records, versioning of edits, mapping of source records, and some light workflow for assignment of records to individual catalogers and for hooking into translation processes[1].

I worked primarily on the source record mapping tools. They take a number of formats as input and are called by the cataloging application to map metadata from these formats into the WDL domain model. Several though not all of which are XML-based, and thus easily dealt with in Python, via the etree module in the lxml package.

Dan recently kicked off a new R&D project for evaluating (any) metadata against any number of metadata profiles, mapping into a generic data dictionary, the goal being to determine how feasible it would be to develop a toolset for aiding remediation of metadata across any number of digital collections. I have been working on this project with Dan, and got started by seeing how generalizable the WDL metadata mapping tools are. Turns out they're fairly generalizable once you tweak the various format-specific mapping rules to map into the generic data dictionary model rather than the WDL model (around 15 elements, and somewhere between Dublin Core and MODS in terms of specificity but flatly structured like DC).

Some of the test data I am working with now, that has nothing to do with WDL, is SGML-based TEI 2 markup. The closest I worked with on WDL was TEI P5 for manuscript description which is serialized in XML. Turns out my TEI mapping rules from before blew up on this TEI 2 stuff, as lxml.etree (naturally) wasn't digging the non-XML input. I googled around a bit for how best to parse TEI (or any SGML) in Python and then discovered it's actually simple as pie.

Code

If you've got the BeautifulSoup module installed[2]:

>>> from BeautifulSoup import BeautifulSoup
>>> tei = open('foo.sgm').read()
>>> BeautifulSoup(tei).findAll('title')[0].string
u'[Memorandum to Dr. Botkin]: a machine readable transcription.'

If not, the lxml.html module works too:

>>> from lxml import html
>>> h = html.parse(open('foo.sgm'))
>>> h.xpath('//title')[0].text
'[Memorandum to Dr. Botkin]: a machine readable transcription.'

Data

And here's what the sample data looks like:

<!doctype tei2 public "-//Library of Congress - Historical Collections (American Memory)//DTD ammem.dtd//EN" 
[
<!entity % images system "07010101.ent"> %images;
]>
<tei2>
<teiheader type="text" date.created="1994/03/15" date.updated="2002/04/05" status="updated" creator="National Digital Library Program
, Library of Congress">
<filedesc>
<titlestmt>
<amid type="aggitemid">wpa0-07010101</amid>
<title>[Memorandum to Dr. Botkin]: a machine readable transcription.</title>
<amcol><amcolname>Life Histories from the Folklore Project, WPA Federal Writers&apos; Project, 1936-1940; American Memory, Library of Congress.</amcolname><amcolid type="aggid"></amcolid>
</amcol>
<respstmt>
<resp>Selected and converted.</resp>
<name>American Memory, Library of Congress.</name>
</respstmt></titlestmt>
<publicationstmt>
<p>Washington, DC, 1994.</p>
<p>Preceding element provides place and date of transcription only.</p>
<p>For more information about this text and this American Memory collection, refer to accompanying matter.</p>
</publicationstmt>
<sourcedesc>
<lccn></lccn>
<sourcecol>U.S. Work Projects Administration, Federal Writers&apos; Project (Folklore Project, Life Histories, 1936-39); Manuscript Division, Library of Congress.</sourcecol>
<copyright>Copyright status not determined; refer to accompanying matter.</copyright></sourcedesc>
</filedesc>
<encodingdesc>
<projectdesc><p>The National Digital Library Program at the Library of Congress makes digitized historical materials available for education and scholarship.</p></projectdesc>
<editorialdecl><p>This transcription is intended to have an accuracy of 99.95 percent or greater and is not intended to reproduce the appearance of the original work.  The accompanying images provide a facsimile of this work and represent the appearance of the original.</p></editorialdecl>
<encodingdate>1994/03/15</encodingdate>
<revdate>2002/04/05</revdate>
</encodingdesc>
</teiheader>
<text type="manuscript">
<body>
<div>
<pageinfo>
<controlpgno entity="I07010101">0001</controlpgno>
<printpgno></printpgno></pageinfo>
<p>Memorandum to Dr. Botkin from G. B. Roberts, May 26, 1941</p>
<p>Subject:  Alabama Material</p>
<p>This material has not yet been accessioned and has only 
<del rend="overstrike">beeen</del> been roughly classified as life histories, folklore, and miscellaneous data and copy save in the case of the 2 ex-slave items and the essay on Jesse Owens, each of which was recommended.</p>
<p>Total no. of items recommended:  3 (14 pp.) 
<handwritten>In progress</handwritten></p></div></body></text></tei2>

Notes

1. Catalogers cataloged stuff in the English language, but every metadata record needed to be translated into the other six U.N. languages: Spanish, Russian, French, Arabic, Chinese, and Portuguese.
2. And you are but one sudo easy_install BeautifulSoup away from that.

There must be a better way of viewing pretty-printed JSON from Firefox than this. (EDIT: Hail, JSONovich!)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

#!/usr/bin/env python
# ~/bin/jsonhandler.py 
# Take some JSON from a file or stdin, format it, output to a tempfile, 
# open in EDITOR
 
from __future__ import with_statement
import os
import sys
import simplejson
import tempfile
 
EDITOR = "/usr/bin/gedit"
 
if __name__ == "__main__":
    if len(sys.argv) == 2:
        # if invoked as jsonhandler.py {FILE}
        json = open(sys.argv[1])
    else:
        # if JSON is piped in (e.g., from Firefox, 
        # or cat {FILE} | jsonhandler.py)
        json = sys.stdin
    json = simplejson.load(json)
    # the with_statement is kind of gratuitous but I like it
    with open(tempfile.mktemp('.json'), 'w') as jsonfile:
        simplejson.dump(json, jsonfile, indent=4)
    # all of that and gedit doesn't even highlight JSON
    # I have emacs highlighting JSON but this generates a "stdin is not 
    # a tty" error, so EDITOR is not set to emacs
    # xemacs works a little better, but I need to click:
    # "Options > Syntax Highlighting > In this buffer" every time, despite 
    # saving to custom.el, so EDITOR is not set to xemacs
    # Very annoying!
    os.system("%s %s" % (EDITOR, jsonfile.name))

And then I set ~/bin/jsonhandler.py as the action for application/json in Edit | Preferences | Applications.

Yuck. Help?

[Update: Feel free to grab the code via bzr with bzr branch http://lackoftalent.org/bzr/shortcut_converter.]

Here's another entry in the "dumb little scripts that work for me and may or may not be helpful to other folks" department…

I use both Windows and Ubuntu at home, gradually transitioning from the former to the latter. I've accumulated a bunch of Windows URL shortcuts, mostly things I wanted to read once so instead of bookmarking them, I dragged their links to my desktop. This creates .URL files which are simple little plain-text two-liners. It turns out that on Ubuntu, and probably similar *nix systems, web shortcuts are also simple little plain-text files. These files have the .desktop extension (though you won't see the extension by looking at the desktop).

I wanted a way to convert my .URL files to .desktop files so that I can just toss them on my Ubuntu desktop and double-click them the same way I would if I were on Windows. This cruddy little Python script does the trick.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

#!/usr/bin/env python
# shortcut_converter.py
 
from __future__ import with_statement 
import os.path
import sys
 
TEMPLATE = """[Desktop Entry]
Version=1.0
Encoding=UTF-8
Name=%(basename)s
Type=Link
URL=%(url)s
Icon=gnome-fs-bookmark
"""
 
def convert(f):
    """ Takes a full filepath to a .URL file, converts it to a .desktop file 
        in the same directory """
    print "Converting %s" % f
    (filepath, filename) = os.path.split(f)
    (basename, extension) = os.path.splitext(filename)
    with open(f) as urlfile:
        lines = [line.strip() for line in urlfile.readlines()]
    url = lines[1].split('URL=')[1]
    dtfname = os.path.join(filepath, '%s.desktop' % basename)
    with open(dtfname, 'w') as dtfile:
        print "Writing %s" % dtfile.name
        dtfile.write(TEMPLATE % locals())
 
if __name__ == '__main__':
    for arg in sys.argv[1:]:
        if os.path.isfile(arg) and arg[-3:].lower() == 'url':
            convert(arg)
        else:
            print "*** %s is not a URL file" % arg

I used scp to pull over all my .URL files and then invoked the script thusly:

python shortcut_converter.py *.URL

worksforme!

Dear Future Me,

You've forgotten how to decode (or unescape) HTML or XML in Python again, haven't you?  My, my, that old age does catch up with you.

Well, it turns out that xml.sax.saxutils.unescape() works like a charm.  I'm certain that edge cases lurk here and there, so caveat, um, coder.

UPDATE: Edge case found. Note that unescape() will not work on ' or ", and so there is:
xml.sax.saxutils.unescape("<p>This is &quot;markup&quot;</p>", {"&apos;": "'", "&quot;": '"'})

In spite of some open questions, I've been making some progress on my Jython-based transport tool. Right now it's pretty dumb and simple: it copies files to and fro via scp.

Being a newb at both Java and Jython made finding the right libraries a bit of a challenge, and so I'm posting some code here for folks in the same boat. It's not particularly pretty due to 1) wanting to get something working very quickly, and 2) weird errors when I try to make things prettier (such as getting rid of the hard-coded bits), but I'll resolve these soon.

The jython code:

# Biter.py
import java
from org.apache.tools.ant import Project
from org.apache.tools.ant.taskdefs.optional import ssh

class Biter(java.lang.Object):
    def __init__(self):
        self.keyfile = '/home/user/.ssh/id_rsa'

    def transport(self, from_uri, to_uri):
        "@sig public void transport(String from_uri, String to_uri)"
        scp = ssh.Scp()
        scp.setKeyfile(self.keyfile)
        scp.setPassphrase('')
        scp.setTrust(True)
        scp.setProject(Project())
        scp.setFile(from_uri)
        scp.setTodir(to_uri)
        scp.execute()
        print "%s -> %s" % (from_uri, to_uri)

I set my CLASSPATH to include /usr/share/jython2.2.1/jython.jar, /usr/share/java/jsch-0.1.28.jar, /usr/share/ant/lib/ant-jsch.jar, /usr/share/ant/lib/ant.jar, and '.', and then compile to bytecode via jythonc -idp gov.loc.repository.transport Biter.py. Once it's compiled, I can use the Biter class in Java code thusly:

# BiterClient.java
import gov.loc.repository.transport.Biter;

public class BiterClient {
    public static void main(String[] args) {
        Biter biter = new Biter();
        biter.transport("/home/user/tmp/test1", "user@example.org:tmp/test2");
        biter.transport("user@example.org:tmp/test2", "/home/user/tmp/test3");
    }
}

I should note that the passphrase and keyfile are not necessary if you're passing valid username and password credentials in the remote to_ or from_uri (and the server in question supports password auth). But pubkey auth is useful for our purposes and that's why it's in there.

Hopefully someone somewhere finds this helpful. :)

It's hard to believe but I've been at the new job for six months already, a full half-year come the 29th. Some days it seems like I've been here forever; others like I'm still a rank newb. I haven't written terribly much about what I've been up to (but I assure you I've been busy). Let me rectify that.

The Transfer Problem

Two of the projects I've been working on relate to a fairly general problem that we like to call "transfer," which revolves around, well, transferring files to and fro. Sounds simple. Is simple. That is, until you start thinking about preservation and accounting for a highly heterogeneous network with idiosyncratic nodes, esoteric storage software, and differential firewall rules. And that's where it gets interesting (and problematic). The transferring itself, or copying of files from one location to another which we call "transport," is the easiest part. We like to use common tools in our environment. It makes life easy. And so good ol' scp seems like an obvious choice to handle the job.

Since preservation is a core aspect of our "repository," a term which I use loosely, we must build certain other functionalities into the transfer process: validation, verification, inventory, backup, ingest, and so forth. Every time a file is copied to a non-transient location, we verify the file against a (SHA1) checksum and record an event for auditing purposes.

Repository Workflows

Some steps in the transfer process are routine and best handled by machines. Thus we automate them with scripts and code. Others require human intervention. This introduces another key aspect of our repository needs: workflows. No two projects will have the same workflow and yet all will have some steps in common. We're using JBoss's jBPM library for workflow management and it is more than capable of handling our workflow needs. It allows us to model complex and varied flows in a robust and not ad hoc way; it does seem preferable to me to model our workflows via jBPM's graphical editor (serialized to JPDL XML) rather than copying around blocks of code and otherwise modeling the workflow procedurally in business logic.

One of my coworkers (the author of this) designed a complete workflow system in jBPM last summer and I've taken on implementing, tweaking, and testing this system, which has required bootstrapping my sorry-arse Java skills and learning jBPM. Though I find it difficult to think in Java patterns and generally find it a burdensome environment, I'm quite impressed by jBPM. I've been working on various updates of and unit test coverage for the workflow system, which has been a crash course in a number of Java technologies and a perfect first task at LC, as it gets me into the guts of things. The Java stack we use for our workflow is highly abstracted and componentized, which is conveniently modular… but it's also Java: fairly heavy and arguably not as agile as dynamic languages such as Python.

Two Great Tastes?

So recently we've begun to think about implementing some transfer and workflow components in Jython (Python written in Java). Why? The value of Jython is as follows:

1. Ease of deployment – Deploying jar files to existing JVMs in production environments (which we do not control) is a simple task, or at least simpler than some other options.
2. Interoperability – Our stack is primarily Java-based and so interoperating with existing Java components means not having to rewrite functionality in other languages. Jython allows Python to talk to Java and vice versa.
3. Familiarity – It's Python, and we like Python. It's the closest my team has to a lingua franca and so it increases the chances of sharing code, maintaining code, and so forth.

It does not come without its drawbacks:

1. Currency – The Jython project went moribund for a few years or so and the latest stable version is now 2.2.1. Compare that to the latest version of Python: 2.5.2. I don't begrudge the Jython developers, though. I'm glad some folks picked the project up, dusted it off, and breathed new life into it. I am also glad that they're skipping 2.3 and 2.4 releases and plowing right into a 2.5 release. Because of this currency issue, some Python libraries won't work with the latest Jython and that means you're stuck looking for outdated, potentially vulnerable Python libraries, or hooking into Java libraries for the same functionality (which inevitably means more lines of code). I'm no lines of code fetishist but it does militate against the goal of agility somewhat.One does have the option of living on the edge and trying out the 2.5 branch, but that seems out of step with an infrastructure that is supposed to preserve terabytes upon terabytes of our nation's, and the world's, intellectual property. It's a responsibility I do not take lightly, as much as I'd like to be on the bleeding edge.
2. Interoperability difficulties – Talking to Java from Jython is a snap: just import java at the top of your script, and, assuming your classpath is copacetic, voila: you have access to Java libraries in your Python code! Talking to your Jython modules from Java code is, well, a little more complicated. Read on.

Despite the caveats it does seem a sane, reasonable, and potentially productive path to go down. Right? I am specifically looking to implement two workflow components in Jython: one for transport (wrapping Ant's JSch library, which provides a slick scp API) and the other for automation of ZFS filesystem/volume creation on the staging server. Nothing arcane, nothing tricky, nothing fancy. So it must be easy! Right? …

Lessons Learned?

I'm beginning to wonder about the feasibility of using Jython to make bits of our Java stack more agile. Specifically, there are three ways to get at Jython code from Java:

1. Compile to bytecode/jar via the Jython compiler (jythonc) and reference your Jython objects and methods as though they were POJOs
2. Embed a Jython interpreter
3. Instantiate a (JSR-223) script engine

Option 1 is nice because you get object- and method-level interop. However, jythonc is unsupported and will disappear. This does not seem sustainable though I might be able to limp along a while. And there are signs of hope:

Though jythonc is going away, all of the capabilities it provides will be present in 2.5 in other forms. We're adding functionality to expose Python classes as Java classes using decorators to replace the docstring class creation that jythonc provided, and we're adding static compilation of proxy classes so regular jython can run in applets and other environments with restrictive classloaders. We're definitely doing something about jythonc.

That doesn't help much now, of course, but just because jythonc goes away does not mean my jars will stop working.

The suggested methods for option 2 [1, 2] seem to be more trouble than they're worth. If the goal is more agile development for certain components, the reliance upon multiple, separate Java classes — an interface class and an object factory, in the examples listed — to get a ten-line Jython script working, this seems suboptimal, both inefficient and not straightforwardly maintainable; it seems, to me and my Java-dumb ways, rather baroque.

Option 3 [1, 2] is more appealing than option 2 as it does not rely upon these other classes specifically for Jython code. But the number of lines of Java code that must be wrapped around the Jython to get it working looks like overkill for the drop-dead simple scripts I'm writing — it might be easier, for instance, to just write the darn things in Java and be done with it. (Did I just say that?)

Conclusions

Options 2 and 3 are similar as both involve embedding Jython code, or referencing files with Jython code, and interpreting the code within Java. Generally, I worry that either option would obviate the benefit of agile Jython scripting because you wind up wrapping the code in so much Java. I offer two disclaimers to counter my objections:

1. Cleverer Java coders than myself could, I am almost certain, find ways to build abstractions (or abstractions of abstractions) to eliminate the "lines of code" and "many separate classes per Jython script" issues
2. The value of Jython in our Java environment increases proportionately with the complexity of the component — given the overhead, a short Java class seems easier and more straightforward to implement than a short embedded Jython script. On the other hand, there's value in embedding a Jython script that'd be an order of magnitude simpler than its Java analog.

At least, that's the state of my head right now re: getting Jython and Java to play nice. If I make any breakthroughs or give up entirely, I'll post follow-ups.

I am but a Java philistine and a Jython neophyte, so I remain humbly open-minded. I would greatly appreciate comments, questions, corrections, smackdowns, sagacious advice, and so on.