I believe in the concepts that the CDL specs represent I just didn't know how I could make them work. Well I think I'm a little closer. I've been looking at Ed Summer's "dfat.py" that you mention in this post and also Ben O'Steen's pairtree.py and they have helped me get a better understanding of how these specs function.

My big realization was that, if I can construct a really good metadata record made up from all the parts of the dflat, I can store objects in the repository, collect that metadata in the tool of my choice, and provide access to those object from an arbitrary set of urls.

Case in Point: I'm using IndexData's Zebra for several projects. Zebra is lightweight, stores the complete xml, but lets you retrieve the data in any format you can write an xslt stylesheet for, and provides an SRU interface. I use it as an XML database with a SRU interface.

So what I'm thinking is, I'll store my assets in a pairtree (I need to build the ingest using Ed and Ben's tools), and then point Zebra to the tree and let it index it (only looking for xml metadata files). I'll then create a public site (I'm using Zope with xslt-methods) that contains a "splash page" for the collection, a query page (SRU) [or combine the two], and then write xslt to handle the results of the query. The public site, while serving the assets from the pairtree repository, is essentially independent of the asset store and vis-versa (just what I want).

I think I'm going to create a pairtree site for each collection (good for humans) but harvest the metatdata into a central Zebra index. This way each collection can have it's own look and feel but the collections can be federated for search.

The critical thing here is getting everything you need to present (but nothing more) into that metadata record. I've been looking at UNT's model but I think you just have to do what's right for you and then convert to the standard models (DC, OAI-DC, LOM, etc.). Is METs still the standard or do we use OAI-RE?

While I don't want to add too many smarts to a pairtree one thing that might be nice is if you could ask the pairtree for all it's meatadata files in an Atom or OAI format. That way, like the current Omekasoftware that can display content from a feed, you can present this data as needed.

FWIW (for what it's worth),

–Brian

Also, I don't think Pairtree makes any assumptions about your identifiers, so using semantic ids (or "reasonable URLs" as you put it :) ) should be possible. You shouldn't need to use ARKs or opaque ids.

The filesystem is slightly less opaque than a database with opaque identifiers because you can still visually scan your store of objects and see your identifiers surfaced in the directory listing, the theory being that filesystem tools will likely outlive particular DB servers or schemas.

There are other specs for surfacing other information in the directory listing such as NAMASTe so that your object store is less opaque, but I'm less convinced of the value of some of these other specs. Still, it's a compelling pattern, and will be poking at all the specs to see how it all plays out.

I don't think you have to access only by identifier. The great thing about the micro-services is that they are, at least in theory, decoupled. So you can choose the ones that work for you and ignore the ones that don't. What's neat about the specs is that there's some overlap, and you have multiple options for storing a "digital object" on the filesystem, so if you don't like identifier-based access and you do want versioning, you can choose to model objects in DFlat w/ ReDDs rather than Pairtree.

Mix, match, rinse, lather, repeat!

Are there any areas in the CDL specs where you see an opportunity make good use of XML?

Thanks for writing, Brian.

I've been looking at this too.

I think most people know that your assets are going to outlive (hopefully) your software. I've been running Zope, DSpace, TKL (IndexData), and Omeka sites among others, and while they are all good, I keep thinking "what's going to happen when you reach the end of the road"? To me this is why the CDL things are so interesting.

I'm just small time and the folks at CDL, along with your self and the co-workers at LOC, have a whole lot more experience than me, so I don't always understand what's going on ;-)

Some of the questions I'm looking at right now are:
* do you serve out of your DFlat"? Why not?
* If you think of the OASiS reference model is this the AIP?
* do you build every app this way?

I'm thinking you have/build lots of "feeder apps" (think BibApp) that have their own models but support harvesting (OAI, ATOM, etc.) and then on the harvest you store in this model. On top of this file system model you can build other structures (RDB, XML, Fedora, what every suits your fancy, etc.) that support access to this store.

I'm having trouble with pairtrees and identifiers (ARK). If (big if) I serve out of this structure I want reasonable URLs (wcsu/library/archives/MS044/whatever.xml) how do I use these numbers (010203945065)? OK, unless I make those numbers "intelligent" (01=college, 02=division,03=format, you get the idea) then the storage on the file system is just as opaque as if you used a database. I understand the need for unique identifiers but do we have to access "only" by identifier (alternative is to use a resolution service I guess but I don't know).

It seems to me that the CDL folks don't like XML (probably for good reason) but I'm leaning toward and XML AIP, store assets on the filesystem (accessible by humans), AND have the ability to serve from that store.

We'll see….

–Brian