As far as I'm concerned, the most exciting developments this year in repositories and digital curation have come out of the California Digital Library. It has been impossible not to notice their papers and presentations. Put simply, their idea is that digital curation is enabled by "micro-services" built upon well-known abstractions such as the filesystem. The benefits are obvious: filesystem tools are ubiquitous and cross-platform, and there are strong market forces to ensure the filesystem persists. The idea is radically simple and straightforward, though many questions remain about such a paradigm. I'll return to those later.

If you have not yet taken a look at CDL's curation micro-service specifications, most of which may be printed on as few as one or two sheets of paper, see the Digital Library Building Blocks.

My co-workers in the LC Repository Development Center have been chatting about these specs on and off throughout the year. After months of procrastinating, I finally read all of the specs on Thursday; it's wonderful that you can do so in the course of one reading session, I might add. Yesterday a bunch of us RDCers got together to chat (informally) about the specs: what they're for, how they work, and how they interact with one another. I learn by doing, by examples, so I combed through each of the specs in advance of our meeting and tried to construct a minimal repository[1] based on micro-services.

Here is a tree visualization of the final product, inevitable warts and all: The services I used were Namaste, Content Access Node (CAN), Pairtree, Dflat, Reverse Directory Deltas (ReDD), Class-based System for Managing Object Properties (CLOP), and BagIt (co-developed by LC and CDL).

As I mentioned in our Friday meeting, recounting my experience exploring the specs: the bad thing is that I spent an hour building a repository with rudimentary tools such as mkdir, touch, cp, ln, and emacs; but the good thing is that I built a repository in one hour using common, rudimentary tools. It's a very compelling paradigm. Ed's already built a tool implementing some of Dflat, further demonstrating how lightweight these micro-services are. (UPDATE: Ed notes that this code is a work in progress and is "barely functional.") (UPDATE 2: The dflat library has come a long way. Check it out if you're interested. Also, I just committed a pretty basic Namaste library: http://github.com/mjgiarlo/namaste. Only took about an hour, which is a testament to the power of lightweight specs.)

I am certain this will be a running thread at work as the specifications evolve and our understanding of them grows. Some questions and comments that occurred to me while exploring the micro-service specs and building the minimal repo:

• CAN was a bit puzzling. The spec is simple enough, but I found some of the conventions confusing, and I was left wondering what CAN provides other than a container. What I would like to see is a simple use case and perhaps more examples. Thus, the CAN stuff in my sample repo doesn't feel very useful only because I had a hard time working with the spec.
• CLOP feels like the least mature of the specifications. It seems generally useful to be able to put digital objects, however you define that, into classes and define properties on those classes. The spec did not clearly convey to me just how it accomplishes that aim. A few examples would go a very long way. I've got some CLOP stuff in the sample repo but I have no idea how close my implementation matches the spec.
• Is Dflat dependent on ReDD? One would assume not since there's an optional property in the dflat-info.txt file for specifying a delta scheme. But, say, could you stub out the v001 directory (reserved to hold the initial version of a digital object) and use a system such as git or bazaar?
  
  One might argue that these established delta schemes, if you want to call them that, have many more developers and users than a system such as ReDD and thus should persist longer and have more tools built around them. I imagine the micro-service viewpoint would acknowledge that point, but counter that the spirit of these specs is to avoid dependencies from outside the filesystem?
• Is the ReDD specification meaningful outside of a Dflat given that any one ReDD directory knows nothing of its successors and predecessors, or is it dependent upon Dflat?
• Could a BagIt bag live inside of the ReDD reserved "full" directory? That is, could the "full" directory be marked up appropriately to be a BagIt bag?
• How many tools exist for these specs? I notice there's code in CPAN for Pairtree and Namaste, which is a fabulous start. Tools are the difference between YAMF (Yet Another Messy Filesystem) and reliably managed curation services. Granted, tools such as cp and emacs already exist and are part of the appeal of these micro-services, but there's also tremendous room for error if operations are all done "by hand."
• To what extent has CDL transitioned to using these specs/tools?
• Are other institutions using these specs/tools? I have heard tell that digital library folks from the University of Michigan and the University of North Texas may be involved.

I hope I don't sound overly critical. I'm really glad our colleagues at the California Digital Library have written these specifications and applied their deep experience to what could be a transformative paradigm[2] in the digital curation world. Kudos to them!

Notes

1. Perhaps it's more in line with the specs to refer to this space as "a managed filesystem that drives repository and curation services," given the CDL philosophy that preservation is not a place/repository. But it's easier to say "repository," so there you go.
2. Please excuse the fanboyishness; this filesystem fetishism is exciting stuff!

About a month ago, I read on my colleague's blog that the Emory University Digital Library published a new book on sustaining digital libraries.  I've finally started reading it and figured I would post a note here.

The articles of this monograph provide resources for digital library stakeholders who seek to better understand how to effectively evolve such efforts from short-term projects to long-term sustainable programs. The monograph includes contributions from leaders in major digital libraries that have made such transitions or which are systematically considering the question of programmatic sustainability, including representatives from the National Digital Infrastructure and Information Preservation Program (NDIIPP) and the National Science Digital Library (NSDL).

I might also note that the book is available for free as a PDF.

So far I've read the introduction by the editors and the abstract from Leslie's paper, and the book looks like a high-quality read from cover to cover, with articles based on actual digital library experience.  It's a pragmatic approach for how to sustain digital library initiatives, looking beyond technical concerns towards the more challenging social and economic ones.  To some extent, we are getting pretty good at preserving bits and relationships between collections of bits — it is yet to be seen how good we will be at preserving the preservation systems themselves.

It's hard to believe but I've been at the new job for six months already, a full half-year come the 29th. Some days it seems like I've been here forever; others like I'm still a rank newb. I haven't written terribly much about what I've been up to (but I assure you I've been busy). Let me rectify that.

The Transfer Problem

Two of the projects I've been working on relate to a fairly general problem that we like to call "transfer," which revolves around, well, transferring files to and fro. Sounds simple. Is simple. That is, until you start thinking about preservation and accounting for a highly heterogeneous network with idiosyncratic nodes, esoteric storage software, and differential firewall rules. And that's where it gets interesting (and problematic). The transferring itself, or copying of files from one location to another which we call "transport," is the easiest part. We like to use common tools in our environment. It makes life easy. And so good ol' scp seems like an obvious choice to handle the job.

Since preservation is a core aspect of our "repository," a term which I use loosely, we must build certain other functionalities into the transfer process: validation, verification, inventory, backup, ingest, and so forth. Every time a file is copied to a non-transient location, we verify the file against a (SHA1) checksum and record an event for auditing purposes.

Repository Workflows

Some steps in the transfer process are routine and best handled by machines. Thus we automate them with scripts and code. Others require human intervention. This introduces another key aspect of our repository needs: workflows. No two projects will have the same workflow and yet all will have some steps in common. We're using JBoss's jBPM library for workflow management and it is more than capable of handling our workflow needs. It allows us to model complex and varied flows in a robust and not ad hoc way; it does seem preferable to me to model our workflows via jBPM's graphical editor (serialized to JPDL XML) rather than copying around blocks of code and otherwise modeling the workflow procedurally in business logic.

One of my coworkers (the author of this) designed a complete workflow system in jBPM last summer and I've taken on implementing, tweaking, and testing this system, which has required bootstrapping my sorry-arse Java skills and learning jBPM. Though I find it difficult to think in Java patterns and generally find it a burdensome environment, I'm quite impressed by jBPM. I've been working on various updates of and unit test coverage for the workflow system, which has been a crash course in a number of Java technologies and a perfect first task at LC, as it gets me into the guts of things. The Java stack we use for our workflow is highly abstracted and componentized, which is conveniently modular… but it's also Java: fairly heavy and arguably not as agile as dynamic languages such as Python.

Two Great Tastes?

So recently we've begun to think about implementing some transfer and workflow components in Jython (Python written in Java). Why? The value of Jython is as follows:

1. Ease of deployment – Deploying jar files to existing JVMs in production environments (which we do not control) is a simple task, or at least simpler than some other options.
2. Interoperability – Our stack is primarily Java-based and so interoperating with existing Java components means not having to rewrite functionality in other languages. Jython allows Python to talk to Java and vice versa.
3. Familiarity – It's Python, and we like Python. It's the closest my team has to a lingua franca and so it increases the chances of sharing code, maintaining code, and so forth.

It does not come without its drawbacks:

1. Currency – The Jython project went moribund for a few years or so and the latest stable version is now 2.2.1. Compare that to the latest version of Python: 2.5.2. I don't begrudge the Jython developers, though. I'm glad some folks picked the project up, dusted it off, and breathed new life into it. I am also glad that they're skipping 2.3 and 2.4 releases and plowing right into a 2.5 release. Because of this currency issue, some Python libraries won't work with the latest Jython and that means you're stuck looking for outdated, potentially vulnerable Python libraries, or hooking into Java libraries for the same functionality (which inevitably means more lines of code). I'm no lines of code fetishist but it does militate against the goal of agility somewhat.One does have the option of living on the edge and trying out the 2.5 branch, but that seems out of step with an infrastructure that is supposed to preserve terabytes upon terabytes of our nation's, and the world's, intellectual property. It's a responsibility I do not take lightly, as much as I'd like to be on the bleeding edge.
2. Interoperability difficulties – Talking to Java from Jython is a snap: just import java at the top of your script, and, assuming your classpath is copacetic, voila: you have access to Java libraries in your Python code! Talking to your Jython modules from Java code is, well, a little more complicated. Read on.

Despite the caveats it does seem a sane, reasonable, and potentially productive path to go down. Right? I am specifically looking to implement two workflow components in Jython: one for transport (wrapping Ant's JSch library, which provides a slick scp API) and the other for automation of ZFS filesystem/volume creation on the staging server. Nothing arcane, nothing tricky, nothing fancy. So it must be easy! Right? …

Lessons Learned?

I'm beginning to wonder about the feasibility of using Jython to make bits of our Java stack more agile. Specifically, there are three ways to get at Jython code from Java:

1. Compile to bytecode/jar via the Jython compiler (jythonc) and reference your Jython objects and methods as though they were POJOs
2. Embed a Jython interpreter
3. Instantiate a (JSR-223) script engine

Option 1 is nice because you get object- and method-level interop. However, jythonc is unsupported and will disappear. This does not seem sustainable though I might be able to limp along a while. And there are signs of hope:

Though jythonc is going away, all of the capabilities it provides will be present in 2.5 in other forms. We're adding functionality to expose Python classes as Java classes using decorators to replace the docstring class creation that jythonc provided, and we're adding static compilation of proxy classes so regular jython can run in applets and other environments with restrictive classloaders. We're definitely doing something about jythonc.

That doesn't help much now, of course, but just because jythonc goes away does not mean my jars will stop working.

The suggested methods for option 2 [1, 2] seem to be more trouble than they're worth. If the goal is more agile development for certain components, the reliance upon multiple, separate Java classes — an interface class and an object factory, in the examples listed — to get a ten-line Jython script working, this seems suboptimal, both inefficient and not straightforwardly maintainable; it seems, to me and my Java-dumb ways, rather baroque.

Option 3 [1, 2] is more appealing than option 2 as it does not rely upon these other classes specifically for Jython code. But the number of lines of Java code that must be wrapped around the Jython to get it working looks like overkill for the drop-dead simple scripts I'm writing — it might be easier, for instance, to just write the darn things in Java and be done with it. (Did I just say that?)

Conclusions

Options 2 and 3 are similar as both involve embedding Jython code, or referencing files with Jython code, and interpreting the code within Java. Generally, I worry that either option would obviate the benefit of agile Jython scripting because you wind up wrapping the code in so much Java. I offer two disclaimers to counter my objections:

1. Cleverer Java coders than myself could, I am almost certain, find ways to build abstractions (or abstractions of abstractions) to eliminate the "lines of code" and "many separate classes per Jython script" issues
2. The value of Jython in our Java environment increases proportionately with the complexity of the component — given the overhead, a short Java class seems easier and more straightforward to implement than a short embedded Jython script. On the other hand, there's value in embedding a Jython script that'd be an order of magnitude simpler than its Java analog.

At least, that's the state of my head right now re: getting Jython and Java to play nice. If I make any breakthroughs or give up entirely, I'll post follow-ups.

I am but a Java philistine and a Jython neophyte, so I remain humbly open-minded. I would greatly appreciate comments, questions, corrections, smackdowns, sagacious advice, and so on.

At long last, the paper that Ron Jantz and I wrote for the Journal of Archival Organization has been published in a special double issue. It's titled "Digital Archiving and Preservation: Technologies and Processes for a Trusted Repository" and is intended to be a fairly nitty-gritty piece on digital preservation (in the context of trusted repositories) for archivists. The abstract:

This article examines what is implied by the term "trusted" in the phrase "trusted digital repositories." Digital repositories should be able to preserve electronic materials for periods at least comparable to existing preservation methods. Our collective lack of experience with preserving digital objects and consensus about the reliability of our technological infrastructure raises questions about how we should proceed with digital-based preservation practices, an emerging role for academic libraries and archival institutions. This article reviews issues relating to building a trusted digital repository, highlighting some of the issues raised and possible solutions proposed by the authors in their work of implementing and acculturating a digital repository at Rutgers University Libraries.

This special double-issue of JAO will also be released in the manuscript, "Archives and the Digital Library."

Thanks to editors Bill Landis, Robin Chandler, Tom Frusciano, and Caryn Radick for seeing this through. And of course to Ron Jantz for getting me interested in this crazy stuff at a time when I had no direction or interest in my career.

A friend and former colleague asked if I would comment on a chapter in her upcoming book on digital rights management and I agreed. The chapter is about identification and authenticity of web resources. Throughout my review of the chapter, I kept coming back to a couple of very basic notions that underlie any effort to provide persistent identifiers for web resources. These notions are, to my mind, central to identifier persistence, and any other concerns rely upon this foundation:

1. Identifier persistence requires an organizational commitment. Persistence cannot be ensured by a few renegades in the skunk-works, nor can it be mandated from on high without the support of those who manage the identifiers or produce web resources. All individuals involved in the life-cycle of web resources must be committed to persistence in perpetuity if true persistence of identifiers is to be achieved.
2. No technology, no standard, no identifier scheme, no information architecture will get you persistence. Whether you choose native URIs, Handles, DOIs, PURLs, ARKs, UUIDs, or XRIs, you will never achieve identifier persistence without active management of your identifiers and web resources. This requires the aforementioned organizational commitment since such management cannot occur without sufficient resources. Management of web resources and identifiers requires time and due diligence and those don't come for free.

And, at the risk of being reductive, that's about it. Once you've got an organizational commitment and a person or team to manage your identifiers and web resources, the rest of the decisions are secondary. If you like semantically meaningful URLs that redirect, choose Handles; if you prefer opaque identifiers, go with ARKs; if you don't want to run your own software, consider PURLs. At that point, it really doesn't matter which scheme you choose, as long as its characteristics match your organization's values. You've already done the heavy lifting; rest easy.

Google and Princeton University went public twenty minutes ago in announcing their arrangement to digitize roughly one million of Princeton's public domain works as part of the ongoing Google Books initiative. Princeton is one of a growing number of academic institutions in the United States — including private institutions like Harvard and Stanford and state universities in Virginia, Texas, and California — that have worked with Google on this. For more information, see the post on the Google Books blog or the official announcement.

Many of the details need to be worked out yet, but I am hopeful this will bode well for our users here at Princeton and for the broader community.

The need for digital preservation just grows and grows.

I've posted a couple new tools during the past couple days. One is an update of Devon Smith's LinkPURL extension for Firefox 2.0.

The other is an ultra-lightweight WordPress plugin that embeds a linkpurl link tag for auto-discovery (so bookmarking agents can detect and grab the persistent URL rather than the impersistent URL up in the addressbar).

Based on a discussion in #code4lib earlier today, I realize that there are a lot of important questions, not to mention serious doubts, about persistent identifiers. I flip-flop on their utility myself, so I found the discussion very useful. (Thanks, edsu!) Maybe I'll write a post or two about persistent identifiers to flesh my thoughts out.

http://feeds.feedburner.com/RlgDiginews

The latest in my series of scraped-together feeds: RLG DigiNews

I noticed this response to the Fedora users survey on Peter Murray's blog, and figured I'd post a response. Since my previous employer did not use Fedora, and I haven't begun my new job yet, I'll be posting about our use of Fedora at Rutgers, The State University of New Jersey.

How did you hear about Fedora?
I heard about Fedora through Ron Jantz, Rutgers' Digital Library Architect, in the summer of 2002. I was working at Rutgers then, actively seeking and tinkering with digital repository software. I installed the 0.9 version of Fedora in December of 2002, if memory serves.

Why did you choose Fedora?
I'd already installed DSpace, Greenstone, and DLXS, and none of them seemed to suit our needs very well. Fedora was an easy choice to make for us since it seemed to be the only repository system that supported any metadata schemas we threw at it, which was a hard requirement, and also its ability to support behavior preservation via disseminators set it apart from the others. Additional advantages were that it exposed its methods through well-documented APIs, which other major repository systems still do not seem to do. In short, it was a clear decision.

The only drawback was the lack of an out-of-the-box interface, but we saw that as more of an opportunity to keep our back-end system separated from its interfaces, and our development team was eager for the chance to develop a customized set of interfaces.

Also, it's free.

Were there economic advantages to your project/org. in selecting Fedora?
I mentioned that Fedora is free, right? I know of no other economic advantages though perhaps one could argue that its inclusion in certain grants was beneficial.

What is Fedora's unique role in your production system?
Fedora serves as the back-end of RUcore: the digital institutional repository of Rutgers University, statewide digital initiatives such as the New Jersey Digital Highway, and electronic journals such as Pragmatic Case Studies in Psychotherapy. It is being used to preserve and provide access to metadata and data objects.

Is there one specific Fedora attribute that enables your project/organization to accomplish your overall goals.
Flexibility with regard to metadata; extensibility of Fedora via its service framework; support for custom disseminators and behavior preservation; usage of the filesystem rather than complete dependence upon RDBMS; separation of repository methods and interfaces; and exposure of methods via web services APIs.

Do you see yourself as an active member of the Fedora community? And why?
I see myself as an active consumer in the Fedora community. I haven't worked with Fedora in over a year due to a job change, so I haven't contributed anything in a while. I hope to remedy this in my new position, though I'm not yet convinced that Fedora is the right solution for them.

What would inspire you to become more involved?
An employer that has committed to Fedora.

What should be the mission of an ongoing Fedora organization?
Continuing support and upgrades, outreach to Fedora users, and advocacy to ensure that folks know what the benefits of Fedora are.

Peter Murray has written a series of pieces about the Fedora digital repository system over at the Disruptive Library Technology Jester blog.

In the first piece, On the Need for a General Purpose Digital Object Repository, it is argued that having a unified repository simplifies management of information systems or "silos."  For instance, there needn't be duplication of workflows or synchronization of content if a number of an organization's repositories, digital libraries, electronic journals, course management systems and so on are all built atop a robust institutional repository.  A unified repository is useful if one desires a search across previously disparate digital projects or collections, if one wishes to eliminate redundancies in coding, if one intends to have a particular object, collection of objects, or part of an object shared across different systems — e.g., a journal article repurposed in a course management system and deposited into an open archive.  With an open, flexible repository, like Fedora, such a configuration is possible assuming your organization, unit, or consortium has someone to devote to managing and customizing the repository. 

An advantage of using the Fedora system, as outlined in Why Fedora? Because You Don't Need Fedora, is that due to modular design and adherence to more or less open standards, one is not necessarily wedded to Fedora for the foreseeable future.  Items in a Fedora repository are serialized as XML objects, either in the Fedora-METS or FOXML format.  While some of this information is copied into a relational database system and an RDF triplestore for speed and convenience, it is all intact within the serialized XML objects which reside in a predictable directory hierarchy on the local filesystem.  There are at least two advantages to this design:

1. Should Fedora experience a catastrophic system glitch, one may rebuild the entire system via a built-in utility (cleverly named "fedora-rebuild") that goes through the objects on the filesystem and restocks the database and triplestore.  And assuming that the administrator of the system is worth his salt, there should be regular full backups of the filesystem, so the entire repository should be rebuildable.  As Peter notes, a simple copy of the filesystem on which the XML objects reside is a fine practice in a larger digital preservation strategy.
2. If one decides to move away from Fedora to the Next Best Thing™, it should be relatively simple to migrate content from Fedora into the new system because of Fedora's storage of all objects (and associated metadata, files, and disseminators) to the filesystem as serialized XML.  All one needs, perhaps, is a set of funky XSLT scripts to massage the objects into a format that works with the new system and voila.  (That is a gross oversimplification, but the point remains that open standards, simple file operations, and XML markup do make for more orderly migrations than black boxes, complex datastores, and loose coupling of information.)
3. Having one's objects stored as XML on the filesystem also opens up opportunities to see how tools which act thereupon might be glued into the repository infrastructure.  One such example might be for an XML-aware search engine (such as amberfish, Lucene, or Zebra).  Since you've got low-level access to these files, it would be fairly simple to tack on a search & indexing system that is independent of your choice of repository.

The third piece, Thinking about Our Fedora Disseminators, highlights Fedora as a repository system that's put real emphasis on digital preservation.  While other repository systems allow for preservation of an object and its metadata, Fedora grants one the ability to preserve the behavior of digital objects and the datastreams thereof, a potential approach to the issue of format migration/emulation.  Through a dissemination abstraction (the "behavior definition") one might apply the same abstract behaviors to items in different formats, saving one the time of defining redundant behaviors.  My explanation is rather vague and incomplete, so I would encourage you to read Peter's third piece in detail.  The point is that "for each record, the application simply asked the repository to deliver a thumbnail of the object. And the repository, regardless of media type, delivered one." 

Taken together, Peter makes a strong case for Fedora as a fine back-end for a unified, multi-purpose repository.  Unlike other repository systems that focus more on the front-end, Fedora focuses on being the plumbing, the "digital library operating system" as Ron Jantz calls it.    Were I not already a Fedora enthusiast, I would find it quite difficult not to consider Fedora (or something like it, such as LANL's aDORe Archive) at MPOW.  Now if someone can send me some hints on drumming up institutional support…