As far as I'm concerned, the most exciting developments this year in repositories and digital curation have come out of the California Digital Library. It has been impossible not to notice their papers and presentations. Put simply, their idea is that digital curation is enabled by "micro-services" built upon well-known abstractions such as the filesystem. The benefits are obvious: filesystem tools are ubiquitous and cross-platform, and there are strong market forces to ensure the filesystem persists. The idea is radically simple and straightforward, though many questions remain about such a paradigm. I'll return to those later.

If you have not yet taken a look at CDL's curation micro-service specifications, most of which may be printed on as few as one or two sheets of paper, see the Digital Library Building Blocks.

My co-workers in the LC Repository Development Center have been chatting about these specs on and off throughout the year. After months of procrastinating, I finally read all of the specs on Thursday; it's wonderful that you can do so in the course of one reading session, I might add. Yesterday a bunch of us RDCers got together to chat (informally) about the specs: what they're for, how they work, and how they interact with one another. I learn by doing, by examples, so I combed through each of the specs in advance of our meeting and tried to construct a minimal repository[1] based on micro-services.

Here is a tree visualization of the final product, inevitable warts and all: The services I used were Namaste, Content Access Node (CAN), Pairtree, Dflat, Reverse Directory Deltas (ReDD), Class-based System for Managing Object Properties (CLOP), and BagIt (co-developed by LC and CDL).

As I mentioned in our Friday meeting, recounting my experience exploring the specs: the bad thing is that I spent an hour building a repository with rudimentary tools such as mkdir, touch, cp, ln, and emacs; but the good thing is that I built a repository in one hour using common, rudimentary tools. It's a very compelling paradigm. Ed's already built a tool implementing some of Dflat, further demonstrating how lightweight these micro-services are. (UPDATE: Ed notes that this code is a work in progress and is "barely functional.") (UPDATE 2: The dflat library has come a long way. Check it out if you're interested. Also, I just committed a pretty basic Namaste library: http://github.com/mjgiarlo/namaste. Only took about an hour, which is a testament to the power of lightweight specs.)

I am certain this will be a running thread at work as the specifications evolve and our understanding of them grows. Some questions and comments that occurred to me while exploring the micro-service specs and building the minimal repo:

• CAN was a bit puzzling. The spec is simple enough, but I found some of the conventions confusing, and I was left wondering what CAN provides other than a container. What I would like to see is a simple use case and perhaps more examples. Thus, the CAN stuff in my sample repo doesn't feel very useful only because I had a hard time working with the spec.
• CLOP feels like the least mature of the specifications. It seems generally useful to be able to put digital objects, however you define that, into classes and define properties on those classes. The spec did not clearly convey to me just how it accomplishes that aim. A few examples would go a very long way. I've got some CLOP stuff in the sample repo but I have no idea how close my implementation matches the spec.
• Is Dflat dependent on ReDD? One would assume not since there's an optional property in the dflat-info.txt file for specifying a delta scheme. But, say, could you stub out the v001 directory (reserved to hold the initial version of a digital object) and use a system such as git or bazaar?
  
  One might argue that these established delta schemes, if you want to call them that, have many more developers and users than a system such as ReDD and thus should persist longer and have more tools built around them. I imagine the micro-service viewpoint would acknowledge that point, but counter that the spirit of these specs is to avoid dependencies from outside the filesystem?
• Is the ReDD specification meaningful outside of a Dflat given that any one ReDD directory knows nothing of its successors and predecessors, or is it dependent upon Dflat?
• Could a BagIt bag live inside of the ReDD reserved "full" directory? That is, could the "full" directory be marked up appropriately to be a BagIt bag?
• How many tools exist for these specs? I notice there's code in CPAN for Pairtree and Namaste, which is a fabulous start. Tools are the difference between YAMF (Yet Another Messy Filesystem) and reliably managed curation services. Granted, tools such as cp and emacs already exist and are part of the appeal of these micro-services, but there's also tremendous room for error if operations are all done "by hand."
• To what extent has CDL transitioned to using these specs/tools?
• Are other institutions using these specs/tools? I have heard tell that digital library folks from the University of Michigan and the University of North Texas may be involved.

I hope I don't sound overly critical. I'm really glad our colleagues at the California Digital Library have written these specifications and applied their deep experience to what could be a transformative paradigm[2] in the digital curation world. Kudos to them!

Notes

1. Perhaps it's more in line with the specs to refer to this space as "a managed filesystem that drives repository and curation services," given the CDL philosophy that preservation is not a place/repository. But it's easier to say "repository," so there you go.
2. Please excuse the fanboyishness; this filesystem fetishism is exciting stuff!

I wrote in June that the I2 subgroup surveyed "repository managers to determine the current practices and needs of the repository community regarding institutional identifiers. Results from the survey will inform a set of use cases that will be shared with the community, and that are expected to drive the development of a new standard for institutional identifiers."

The survey closed in July, and the subgroup spent August writing a report on the survey results. That report is now final and it's available to the public. Feedback may be sent to our (woefully underutilized) public i2info mailing list, left as a comment on this post, or e-mailed to me privately which I can forward to our internal list.

The next step is to build upon the report to draw yet more conclusions from the data — there's an awful lot there — and flesh out some repository use cases for institutional identifiers. The I2 core group is moving quickly towards finalizing identifier metadata elements so that a standard may be drafted, and I think having some use cases documented will help drive the standard in a direction the community can get behind.

Onward and upward.

As I mentioned earlier, I've been learning about linked data in the context of dropping it into the World Digital Library project. I am hopeful we'll be able to deploy the RDF views[1] before too long. In advance of that, I thought it might be helpful to share a sample of what our RDF would look like. The RDF below represents the WDL item for the U.S. Constitution. I appreciate constructive criticism.

A few things to note:

• Mmm, Unicode.
• Item types are from the Bibliographic Ontology.
• Most of the properties are from the Dublin Core Metadata Element Set ontology, especially used where literals are objects rather than resources identified by URI.
• Where possible I dug up or found URIs and used the Dublin Core Metadata Terms ontology.
• An item is modeled as an aggregation of its constituent files, as defined in OAI-ORE. The notion here is that an ORE aggregation of an item, as expressed in a resource map which is discoverable via a link header in each item detail page, is a "whole" item, including all of its files[2], metadata, and translations.
• I'm also making light use of the NEPOMUK File Ontology to express that constituent files are files, and to be explicit about file sizes so that folks know in advance of retrieving it how large files are.
• Links out to DDC (Decimalised Database of Concepts), Lingvoj, DBpedia, and Library of Congress Authorities & Vocabularies (e.g., LC Subject Headings) are included where possible. [3] I'd be especially stoked to hear of other vocabs I might link to. The more linked the data, the better.
• The output below is Turtle for readability, but the application will offer up RDF/XML.

The data after the jump:

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix dc: <http://purl.org/dc/elements/1.1/> .
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix nfo: <http://www.semanticdesktop.org/ontologies/nfo#> .
@prefix ore: <http://www.openarchives.org/ore/terms/> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
 
<http://localhost/static/c/2708/service/00303_2003_001_pr.jpg>
    dc:format "image/jpeg" ;
    nfo:fileSize "259485"^^<http://www.w3.org/2001/XMLSchema#long> ;
    a nfo:FileDataObject .
 
<http://localhost/static/c/2708/service/00303_2003_003_pr.jpg>
    dc:format "image/jpeg" ;
    nfo:fileSize "267031"^^<http://www.w3.org/2001/XMLSchema#long> ;
    a nfo:FileDataObject .
 
<http://localhost/static/c/2708/reference/00303_2003_004_pr_thumb_item.gif>
    dc:format "image/gif" ;
    nfo:fileSize "56620"^^<http://www.w3.org/2001/XMLSchema#long> ;
    a nfo:FileDataObject .
 
<http://localhost/static/c/2708/service/00303_2003_004_pr.jpg>
    dc:format "image/jpeg" ;
    nfo:fileSize "233875"^^<http://www.w3.org/2001/XMLSchema#long> ;
    a nfo:FileDataObject .
 
<http://localhost/static/c/2708/service/00303_2003_002_pr.jpg>
    dc:format "image/jpeg" ;
    nfo:fileSize "245809"^^<http://www.w3.org/2001/XMLSchema#long> ;
    a nfo:FileDataObject .
 
<http://localhost/item/2708/about.rdf>
    dcterms:created "2009-08-10T18:11:25-04:00"^^dcterms:W3CDTF ;
    dcterms:creator <http://dbpedia.org/resource/World_Digital_Library> ;
    dcterms:modified "2009-08-10T18:11:25-04:00"^^dcterms:W3CDTF ;
    ore:describes <http://localhost/item/2708/about.rdf#item> ;
    a ore:ResourceMap .
 
<http://localhost/item/2708/about.rdf#item>
    dc:created "17 Septembre 1787"@fr, "17 de septiembre de 1787"@es, "17 de setembro de 1787"@pt, "17 сентября 1787 г."@ru, "1787年9月17日"@zh, "September 17, 1787"@en, """١٧ ايلول ١٧٨٧
"""@ar ;
    dc:creator "Constitutional Convention, United States"@en, "Convención Constituyente, Estados Unidos"@es, "Convention constitutionnelle, États-Unis"@fr, "Convenção Constitucional, Estados Unidos"@pt, "Конституционная Конвенция, Соединенные Штаты"@ru, "الاتفاقية الدستورية، الولايات المتحدة"@ar, "制宪会议，美国"@zh ;
    dc:extent "Manuscript (4 pages of parchment)"@en, "Manuscrit (4 pages de parchemin)"@fr, "Manuscrito (4 páginas de pergamino)"@es, "Manuscrito (4 páginas em pergaminho)"@pt, "Рукопись (4 пергаментных страницы)"@ru, "مخطوطة (٤ صفحات من الورق النفيس)"@ar, "手草本（4 页羊皮纸）"@zh ;
    dc:language "Anglais"@fr, "English"@en, "Inglés"@es, "Inglês"@pt, "Английский язык"@ru, "الإنجليزية"@ar, "英语"@zh ;
    dc:publisher "Administração de Registros e Arquivos Nacionais"@pt, "Archives Nationales et Administration des documents (NARA) des États-Unis d'Amérique "@fr, "Los Archivos Nacionales y Administración de Documentos (NARA) de los Estados Unidos de América"@es, "National Archives and Records Administration"@en, "Управление национальных архивов и документов"@ru, "الإدارة الأمريكية للوثائق والسجلات الوطنية"@ar, "美国国家文件与档案管理局"@zh ;
    dc:subject "Constituciones"@es, "Constituições"@pt, "Constitutional & administrative law"@en, "Constitutions"@en, "Constitutions"@fr, "Derecho constitucional y administrativo"@es, "Direito constitucional e administrativo"@pt, "Droit constitutionnel et administratif"@fr, "Politics and government"@en, "Politique et gouvernement"@fr, "Política e governo"@pt, "Política y gobierno"@es, "Конституции"@ru, "Конституционное и административное право"@ru, "Политика и правительство"@ru, "الدساتير"@ar, "السياسة والحكومة"@ar, "القانون الدستوري والإداري."@ar, "宪法"@zh, "宪法 & 行政法"@zh, "政治和政府"@zh ;
    dc:title "Constitución de los Estados Unidos"@es, "Constituição dos Estados Unidos"@pt, "Constitution des États-Unis"@fr, "Constitution of the United States"@en, "Конституция Соединенных Штатов"@ru, "دستور الولايات المتحدة"@ar, "美国宪法"@zh ;
    dcterms:DDC "342" ;
    dcterms:LCSH <http://id.loc.gov/authorities/label/Constitutions> ;
    dcterms:alternative "Constitution of the United States"@en ;
    dcterms:dateSubmitted "2009-05-07T06:45:21-04:00"^^dcterms:W3CDTF ;
    dcterms:description "1787 年 5 月 14 日，制宪会议在费城的议会大楼（独立厅）召开，目的是修订《邦联条例》。 由于开始时只有两个州的代表团出席，成员不得不一天天地休会，直到 5 月 25 日与会人数达到法定的七个州。 通过讨论和争辩，6 月中旬时明确显示大会与其修改现有的《联邦条例》不如为政府重新起草一份全新的框架。 整个夏季，代表们都在非公开会议中辩论、起草、重新起草新宪法的条款。 主要的争论问题包括要赋予中央政府多大权利、允许各州在国会中有多少个代表席位以及这些代表应该如何选举产生——由人民直接选举还是由各州立法人员选举产生。 这部宪法是很多人智慧的结晶，是合作政治运作和妥协艺术的典范。"@zh, "A Convenção Federal reuniu-se na Casa de Estado (Hall da Independência), em Filadélfia, em 14 de maio de 1787 para revisar os Artigos da Confederação. Em virtude de estarem presentes, inicialmente, as delegações de apenas dois estados, os membros suspenderam os trabalhos, dia após dia, até que fosse atingido o quórum de sete estados em 25 de maio. Através de discussões e debates ficou claro, em meados de junho que, em vez de alterar os atuais artigos da Confederação, a convenção deveria elaborar uma estrutura inteiramente nova para o governo. Ao longo de todo o verão, os delegados debateram, elaboraram e reelaboraram os artigos da nova Constituição em sessões fechadas. Entre os principais pontos em questão estavam o grau de poder permitido ao governo central, o número de representantes no Congresso para cada Estado, e como estes representantes deveriam ser eleitos - diretamente pelo povo ou pelos legisladores do estado. A Constituição foi o trabalho de muitas mentes e permanece como um modelo de cooperação entre lideranças políticas e da arte da condescendência."@pt, "La Convención Federal se reunió en la Cámara del Estado (Salón de la Independencia) en Filadelfia el 14 de mayo de 1787, para revisar los artículos de la Confederación. Debido a que las delegaciones de sólo dos estados estuvieron presentes inicialmente, los miembros levantaron sesión de un día para el siguiente hasta que se obtuvo un quórum de siete estados el 25 de mayo. A través de la discusión y el debate se hizo evidente a mediados de junio que, en lugar de modificar los actuales artículos de la Confederación, la convención prepararía un marco totalmente nuevo para el gobierno. Durante todo el verano, los delegados debatieron, prepararon y redactaron nuevamente los artículos de la nueva Constitución en sesiones a puerta cerrada. Entre los principales puntos en cuestión estuvieron cuánto poder otorgar al gobierno central, el número de representantes en el Congreso que se iban a permitir a cada Estado y la forma en que estos representantes debían ser elegidos, directamente por el pueblo o por los legisladores estatales. La Constitución fue el resultado del trabajo de muchas mentes y se erige como modelo de cooperación política y del arte del compromiso."@es, "La Convention Fédérale s'assembla dans la Chambre Législative (Independence Hall) à Philadelphie le 14 mai 1787, pour réviser les articles de la Confédération. En raison de la seule présence initiale des délégations de deux États, les membres ajournèrent d'un jour à l'autre jusqu'à ce que le quorum de sept États soit obtenu le 25 mai. Â travers les discussions et les débats, il devint clair dès la mi-juin que, plutôt que de modifier les articles existants de la Confédération, la convention allait plutôt ébaucher un cadre entièrement nouveau pour le gouvernement. Tout au long de l'été, les délégués débattirent, élaborèrent, et remanièrent les articles de la nouvelle Constitution, à huis clos. Les principaux points litigieux portaient sur la puissance à accorder au gouvernement central, sur le nombre de représentants au Congrès pour chaque État, et sur le mode d'élection de ces représentants - directement par le peuple ou par les législateurs de l'état. La Constitution fut l'œuvre de nombreux esprits et reste un modèle de coopération politique et de l'art du compromis."@fr, "The Federal Convention convened in the State House (Independence Hall) in Philadelphia on May 14, 1787, to revise the Articles of Confederation. Because the delegations from only two states were present initially, the members adjourned from one day to the next until a quorum of seven states was obtained on May 25. Through discussion and debate it became clear by mid-June that, rather than amend the existing Articles of Confederation, the convention would draft an entirely new framework for the government. All through the summer, the delegates debated, drafted, and redrafted the articles of the new Constitution in closed sessions. Among the chief points at issue were how much power to allow the central government, how many representatives in Congress to allow each state, and how these representatives should be elected--directly by the people or by the state legislators. The Constitution was the work of many minds and stands as a model of cooperative statesmanship and the art of compromise."@en, "Федеральное собрание собралось на заседание в Доме правительства (зал Независимости) 14 мая 1787 года для пересмотра Статей Конфедерации. Поскольку вначале на заседании присутствовали представители только двух штатов, Собрание было распущено на несколько дней до тех пор, пока 25 мая не был обеспечен кворум из представителей семи штатов. В ходе дискуссий и дебатов к середине июня стало понятно, что собрание было намерено скорее составить новый вариант структуры правительства, нежели чем пересматривать существующие Статьи Конфедерации. В течение всего лета делегаты обсуждали, составляли черновые варианты статей новой Конституции и тут же их пересматривали в ходе закрытых заседаний. Среди основных обсуждавшихся вопросов были вопросы степени власти и полномочий, которыми должно быть наделено центральное правительство, количества представителей в Конгрессе от каждого штата, а также процедуры переизбрания этих представителей — непосредственно жителями штатов или законодательными собраниями штатов. Конституция была плодом работы многих политиков и является ярким примером сотрудничества государственных деятелей и искусства компромисса."@ru, "اجتمع ممثلو الاتحاد الفدرالي في قصر الدولة (قاعة الاستقلال) في فيلادلفيا يوم ١٤  أيار ١٧٨٧ لتعديل النظام الأساسي للاتحاد. وحيث حضر وفدان اثنان فقط من وفود الولايات في البداية، رفع الأعضاء الحضور الجلسة من يوم إلى آخر حتى اكتمل النصاب القانوني بحضور وفود سبع ولايات في ٢٥ أيار. وقد اتضح خلال المناقشات والحوار بحلول منتصف حزيران أنه بدلا من تعديل مواد الاتحاد الكونفدرالي القائمة، كان على المؤتمرين صياغة إطار جديد تماما بالنسبة للحكومة. وطوال ذلك الصيف، ناقش المندوبون وصاغوا ثم أعادوا صياغة مواد الدستور الجديد في جلسات مغلقة. ومن بين النقاط الرئيسية التي دار حولها الجدل مدى صلاحيات الحكومة المركزية وعدد الممثلين في الكونغرس لكل ولاية ، وكيفية انتخاب هؤلاء ممثلين -- بالانتخاب المباشر من الشعب أو من قبل مشرّعي الولايات. لقد كان الدستور من عمل عقول كثيرة وهو يمثل نموذجا لفن الحكم التعاوني حنكة التوصل إلى الحلول الوسط."@ar ;
    dcterms:identifier "http://localhost/item/2708/about.rdf#item" ;
    dcterms:language <http://www.lingvoj.org/lang/en> ;
    dcterms:publisher <http://dbpedia.org/resource/National_Archives_and_Records_Administration> ;
    dcterms:spatial <http://dbpedia.org/resource/North_America>, <http://dbpedia.org/resource/United_States_of_America>, "América del Norte"@es, "América do Norte"@pt, "Amérique du Nord"@fr, "Estados Unidos da América"@pt, "Estados Unidos de América"@es, "North America"@en, "United States of America"@en, "États-Unis d'Amérique"@fr, "Северная Америка"@ru, "Соединенные Штаты Америки"@ru, "أمريكا الشمالية"@ar, "الولايات المتحدة الأمريكية"@ar, "北美"@zh, "美国"@zh ;
    dcterms:subject <http://dbpedia.org/resource/Constitutions> ;
    dcterms:temporal "1700 AD - 1799 AD"@en, "1700 ap. J.-C. - 1799 ap. J.-C."@fr, "1700 d.C. - 1799 d.C."@es, "1700 d.C. - 1799 d.C."@pt, "1700 н.э. - 1799 н.э."@ru, "1700 公元 - 1799 公元"@zh, "١٧٠٠ م - ١٧٩٩ م"@ar ;
    dcterms:title <http://dbpedia.org/resource/Constitution_of_the_United_States> ;
    ore:aggregates <http://localhost/static/c/2708/reference/00303_2003_004_pr_thumb_item.gif>, <http://localhost/static/c/2708/service/00303_2003_001_pr.jpg>, <http://localhost/static/c/2708/service/00303_2003_002_pr.jpg>, <http://localhost/static/c/2708/service/00303_2003_003_pr.jpg>, <http://localhost/static/c/2708/service/00303_2003_004_pr.jpg> ;
    ore:isDescribedBy <http://localhost/item/2708/about.rdf> ;
    a <http://purl.org/ontology/bibo/Manuscript> ;
    rdfs:seeAlso <http://hdl.loc.gov/loc.wdl/dna.2708> .

Notes

1. Sadly, the URIs are uglyish due to some constraints from our caching configuration. I figure we can redirect uglyish URIs to cool ones and make use of owl:sameAs if those constraints go away.
2. sans certain low-quality derivatives such as small thumbnails and tiles for the zoom interface
3. I was poking through the DBpedia output for Geonames URIs as well, but my method was way too slow and clunky, so that's disabled for the time being. Clients can always follow their noses from the DBpedia output.

I've been periodically poking at getting Linked Data/RDF views hooked into the World Digital Library web application, following Ed Summers' lead from his work on Chronicling America. The RDF views also use the OAI-ORE vocabulary to express aggregations — in WDL, an item is an aggregation of its constituent files. The goal is to provide a semantically rich and holistic representation of a WDL item (identifier, constituent files, metadata, translations, and so on).

The ORE format is a new one for me so it's hard to say whether the output of my dev branch is valid ORE or not. Plus I'm a sucker for validators. Turns out Rob Sanderson has developed a Python library for validating ORE, and this little snippet is what I've been using to validate the ORE. I didn't put much effort into making it readable, so much as banging something functional out so I can meet deadlines, so mea culpa and all that. But without further hemming and hawing, the code:

# validate.py
import sys
from foresite import *
 
rem = RdfLibParser().parse(ReMDocument(sys.argv[1]))
aggr = rem.aggregation
n3 = RdfLibSerializer('n3')
rem2 = aggr.register_serialization(n3)
print rem2.get_serialization(n3).data

Most of this code is naively copied and pasted from Rob's excellent Foresite documentation.

I invoke it thusly: python validate.py {URL}

And the output:

@prefix _27: <http://www.semanticdesktop.org/ontologies/nfo#>.
@prefix _28: <http://localhost/en/item/1/id#>.
@prefix _29: <http://localhost/en/item/1/>.
@prefix bibo: <http://purl.org/ontology/bibo/>.
@prefix dc: <http://purl.org/dc/elements/1.1/>.
@prefix dcterms: <http://purl.org/dc/terms/>.
@prefix ore: <http://www.openarchives.org/ore/terms/>.
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>.
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>.
@prefix rdfs1: <http://www.w3.org/2001/01/rdf-schema#>.
 
 _28:ResourceMap a ore:ResourceMap;
     dc:format "text/rdf+n3";
     dcterms:created "2009-07-31T14:23:31Z";
     dcterms:modified "2009-07-31T14:23:31Z";
     ore:describes _29:id. 
 
 _29:id a bibo:Image,
         ore:Aggregation;
     dcterms:DDC "973";
     dcterms:alternative "Antietam, Maryland. Allan Pinkerton, President Lincoln, and Major General John A. McClernand"@en;
     dcterms:created "1862年10月3日"@zh,
         "3 de octubre de 1862"@es,
         "3 de outubro de 1862"@pt,
         "3 octobre 1862"@fr,
         "3 октября 1862 года"@ru,
         "October 3, 1862"@en,
         " ٣ آكتوبر، ١٨٦٢"@ar;
     dcterms:creator "Gardner, Alexander"@en,
         "Gardner, Alexander"@es,
         "Gardner, Alexander"@fr,
         "Gardner, Alexander"@pt,
         "Гарднер, Александр"@ru,
         "جاردنر, أليكسندر"@ar,
         "加德纳, 亚历山大"@zh;
... (and so on and so forth)
     dcterms:title "Antietam, Maryland. Allan Pinkerton, President Lincoln, and Major General John A. McClernand: Another View"@en,
         "Antietam, Maryland. Allan Pinkerton, el Presidente Lincoln y el General Principal John A. McClernand: Otra visión"@es,
         "Antietam, Maryland. Allan Pinkerton, le président Lincoln et le général-major John A. McClernand: Autre vue"@fr,
         "Antietam, Maryland. Allan Pinkerton,  Presidente Lincoln e Major-General John A. McClernand: Outra Vista"@pt,
         "Антитэм, штат Мэриленд. Аллан Пинкертон, президент Линкольн и генерал-майор Джон А. Макклернанд: Другой снимок"@ru,
         "أنتينام، ميريلاند ألان بينكرتون، الرئيس لينكولن، واللواء جون أ. ماكليرناند: منظر آخر"@ar,
         "安蒂特姆，马里兰州 艾伦·平克顿、林肯总统和少将约翰·A ·马克克拉南: 另一个视角"@zh;
     ore:aggregates <http://localhost/static/c/1/reference/04326u_thumb_item.gif>,
         <http://localhost/static/c/1/service/04326u.tif>;
     ore:isDescribedBy <http://localhost/en/item/1/item.rdf>;
     rdfs:seeAlso <http://hdl.loc.gov/loc.wdl/dlc.1>. 
 
 <http://localhost/static/c/1/reference/04326u_thumb_item.gif> a _27:FileDataObject;
     dcterms:format "image/gif";
     _27:fileSize "34531"^^<http://www.w3.org/2001/XMLSchema#long>. 
 
 <http://localhost/static/c/1/service/04326u.tif> a _27:FileDataObject;
     dcterms:format "image/tiff";
     _27:fileSize "1301614"^^<http://www.w3.org/2001/XMLSchema#long>. 
 
 ore:Aggregation rdfs1:isDefinedBy <http://www.openarchives.org/ore/terms/>;
     rdfs1:label "Aggregation". 
 
 ore:ResourceMap rdfs1:isDefinedBy <http://www.openarchives.org/ore/terms/>;
     rdfs1:label "ResourceMap".

You might pick up on some warts I have yet to fix, but there you go.

What happens to a digital object defined?[1]



Does its identifier dry up

like a raisin in the sun?

Or its relationships fester like a sore–

And then run?

Do its bits rot like meat?

Or become overwritten–

like some throw-away sheet?



Maybe its metadata just sags

like a heavy load.



Or does it fade into code?

Notes

1. Inspired by Langston Hughes's "A Dream Deferred" and a spirited conversation in the office today.

[Series]

Near the end of my strawman post, I wrote:

The I2 repositories subgroup will be sending out its survey on identifier use cases in the coming week. It will be interesting to see if the requirements we have thus far identified still obtain in light of the data we collect from the survey.

We completed the survey late last week and began distributing it. Here's what we sent out:

The NISO I2 Working Group is surveying repository managers to determine the current practices and needs of the repository community regarding institutional identifiers. We value your time and your input in the process to create a standard for a new institutional identifier. We hope that you will complete the survey which should take less than 15 minutes. The survey will remain open through Monday, July 6th.

Here is a link to the survey:
http://www.surveymonkey.com/s.aspx?sm=RGQgZ3090DVrb3kFzr3P3Q_3d_3d

Please feel free to share this message with other interested parties.

First we used Survey Monkey to send the survey link to approximately one-hundred repository managers that the subgroup identified. Our process for identifying repository managers involved pulling together a list of prominent repositories from subgroup members, and then gathering more from OpenDOAR, "an authoritative directory of academic open access repositories." Then subgroup members were encouraged to share the survey link with colleagues, and post it far and wide via blogs, listservs, and tweets. The listservs we targeted were: JISC-REPOSITORIES, metadataLibrarians, digital-curation, SPARC-IR, ir-net, REPOMAN-L, PALINET-IR-L, dspace-general, fedora-commons-users, DC-IDENTIFIERS, and code4lib.

I've already received a few responses and have gotten useful feedback. Two of the hardest questions to answer so far have been: "What is an institutional identifier?" and "What is a repository?"

Institutional identifier

An institutional identifier is defined as a symbol or code that uniquely identifies an institution. Domain-specific examples of existing identifiers include SAN, IPEDS, GLN, MARC Org Code, and ISIL. Another example might be a Handle prefix or ARK name authority assigning number.

Repository

Institutional repositories and subject repositories like arxiv.org are clearly 'repositories', but beyond that it is a somewhat ill-defined term. One might look to the Kahn-Wilensky architecture, or the OAIS reference model (PDF), or even Wikipedia for definitions, but it's not clear that even the authorities agree on what constitutes a repository.

It's a system. It's network-accessible and typically has a web interface of some sort. Files and groups of files sometimes known as objects tend to be deposited in them, perhaps for some combination of management, access, or preservation. Many run Fedora, DSpace, and ePrints, and factor heavily in scholarly communication. Some are document-centric. Some will accept anything. To some, a learning management system may be a repo. To others, a content management system may fit.

My background is in academia so my own definition is somewhat based in that context, but I wouldn't say the term is necessarily limited to that context. There are other NISO I2 scenarios for library workflows and electronic resources, so it's safe to assume that repository does not mean ILS or OPAC or ERP system. My hope is that folks have their own working definitions of the term and can decide for themselves what it means.

We've given folks a little over two weeks to respond to the survey, so the constant I2 drum-beating will quiet down for a while around here. I am very interested in what sorts of responses we get from the survey. Fun times!

Oh, and perhaps it goes without saying, but if you're a repository owner, manager, expert, developer, or stakeholder with an interest in identifiers, please feel free to take the survey!

[Series]

In the prior I2 post, I wrote about the requirements the repositories subgroup has come up with for an institutional identifier standard (with the hope that our findings re: repositories could be generalized to other scenarios).

My strawman proposal of sorts is to explore how well linked data patterns fit this problem space. Linked data, briefly, is a way to expose and link data on the web in a more semantically meaningful way, and is often summarized using the four principles put forward by Tim Berners-Lee:

1. Use URIs as names for things
2. Use HTTP URIs so that people can look up those names.
3. When someone looks up a URI, provide useful information.
4. Include links to other URIs. so that they can discover more things.

That's the crux of it.  Linked data takes well-known patterns on the web (linking, dereferencing, etc.) and applies them to data, which in this case could be metadata for identifying institutions.

Let's examine each of the requirements and the applicability of linked data thereto.

1. Should be agnostic to type of institution, e.g., libraries, museums, personal collections, historical societies: The web is already agnostic to type of institution.  HTTP URIs do not favor one type of institution over another.
   
2. Should handle varying institutional granularity, e.g., institution-level, campus-level, division-level, unit-level: HTTP URIs are flexible in this regard.  Hierarchy, should one wish it to be surfaced in the identifier, may be encoded in either a DNS hostname or the path appended to the DNS name.  One can imagine a URI like "http://department.division.institution.tld/unit/subunit" or "http://institution.tld/campus/office/individual".
   
   Hierarchy needn't be surfaced in the identifier if one favors opacity, in which case "http://registry.tld/xnjsdasd" would suffice as an identifier, and may instead be entirely reflected in the (RDF) representation returned by dereferencing the URI.
   
3. Should handle linking among institutions and subordinate units: Linked data handles linking via well-known HTTP mechanisms, referenced in the fourth principle of linked data.  Unlike the HTTP link, which has limited semantics, linked data links are semantically rich and extensible.
   
4. Should express different sorts of relationships among these institutions and units: The "useful information" in the third principle of linked data is typically provided by an RDF representation, which is itself a list of assertions.  These assertions, or triples, consist of subjects, predicates, and objects.  The ability to express the relationships in this requirement is limited only by the availability of vocabularies that contain sets of predicates and classes for subjects and objects.  Think of the predicates as elements defined within a metadata standard, e.g., Dublin Core "creator", MODS "relatedItem", and so forth.  Vocabularies that contain these predicates and classes are growing and evolving daily, and should there not be a vocabulary that contains the relationship one wishes to express, it is fairly easy to create a custom vocabulary.
   
   The ability to mix and match vocabularies provides an expressiveness that is often not found in document-based metadata formats and the flexibility to express radically different relationships on a per-industry or per-institution basis.  This latter point is important as the I2 group has identified both core metadata elements for identifying institutions of different types and additional elements for specific types of institutions.  Why re-invent a new metadata format or schema when all one needs to express may already be contained in others?
   
5. Should relate to existing relevant identifiers and registries: Same as requirement#4.  Linked data is all about expressing relationships between things, e.g., institutions, identifiers, registries, etc.
   
6. Should be globally unique: HTTP URIs are guaranteed to be globally unique by virtue of the distributed DNS system and hierarchical naming within each HTTP service.
   
7. Should be actionable: HTTP URIs provide dereferenceability/actionability via the well-known HTTP protocol.
   
8. Should enable retrieval of metadata sufficient to identify the institution, which may vary widely by institution: HTTP URIs are actionable per requirement #7 and the metadata returned is flexible per requirement #4.
   
9. Should accommodate changes as institutions come and go and re-organize and be able to relate defunct institutions to new ones: Linked data patterns provide for redirecting from defunct representations (institutional identifiers) to new ones via HTTP redirects.  One may also add assertions to institutional metadata such as owl:sameAs, for instance, which says that the institution identified by the given URI is the same as another institution identified by another URI.
   

This seems like a compelling path to follow for the I2 standard.

The I2 repositories subgroup will be sending out its survey on identifier use cases in the coming week.  It will be interesting to see if the requirements we have thus far identified still obtain in light of the data we collect from the survey.  If so, I would like to explore the idea of linked data for institutional identifiers a bit more.

[Series]

The I2 IR scenario subgroup approached the issue of institutional identifiers in repositories by first brainstorming about the various issues, problems, and sticking points that make identifiers in this space (and elsewhere) such a complex topic. Folks on the subgroup are repository managers or are otherwise involved with or knowledgeable about the repository space, so the brainstorming exercise yielded a good number of concerns.

The purpose of the exercise was to enumerate concerns and issues that could inform a draft survey to be administered to repository managers and experts around the globe in different organizational contexts: libraries, subject disciplines, archives, historical societies, etc. The purpose of the survey is to get an idea of the use cases and constraints around institutional identifiers in these different repository contexts, the assumption being that we ought to have requirements grounded in real world usage before we go off building a standard.

I will note here that the subgroup has worked up a draft survey that has just recently been reviewed by a small group of folks who know about survey design, and we hope to administer the survey to the aforementioned Reporati this week[1]. Which is to say that I don't yet have a strong grasp of the use cases out there in the wild, and this series should be construed as my own premature cognitive fumblings. But let's assume for now that what we learn from the survey results matches our initial brainstorming exercise.

Here is a slightly modified and boiled down version of the concerns and issues the subgroup came up with for a potential institutional identifier standard, which resembles a set of minimum requirements:

1. Should be agnostic to type of institution, e.g., libraries, museums, personal collections, historical societies
2. Should handle varying institutional granularity, e.g., institution-level, campus-level, division-level, unit-level
3. Should handle linking among institutions and subordinate units
4. Should express different sorts of relationships among these institutions and units
5. Should relate to existing relevant identifiers and registries
6. Should be globally unique
7. Should be actionable
8. Should enable retrieval of metadata sufficient to identify the institution, which may vary widely by institution
9. Should accommodate changes as institutions come and go and re-organize and be able to relate defunct institutions to new ones

I doubt the list is exhaustive; I am almost certain we will uncover all sorts of tangly and esoteric use cases that add requirements. I expect it. Why else would we be gathering to discuss the need for an institutional identifier if it were a solved problem or a simple one? [2]

Nevertheless, looking at the above list, the task we've taken on starts to feel less onerous. And thinking about identifier systems constrained by the list of concerns, the mind starts to cook up all sorts of possible solutions. I'll share one in the next post in this series, a strawman proposal of sorts, and how it addresses each of these requirements.

Notes

1. We will also x-post to repo-related mailing lists as well, and some of us may blog or tweet about it. My inclination is to cast as wide a net as possible so as not to miss important use cases. We can always scope things out later on, but it's useful to be inclusive at this point lest our own assumptions carry the group forward.
2. The cynical among you might have interesting answers to this question.

[Series]

This is the first in a series of posts about institutional identifiers[1].

In my last post, I alluded to some documentation that I've written. That was somewhat misleading, which will soon be apparent, but I liked the parallel construction I had going, and I am but a slave to orderliness.

For about the past six months, I have been working with a NISO group looking into how institutions are identified within information systems:

The I2 (Institutional Identifiers — pronounced "I 2") working group will build on work from the Journal Supply Chain Efficiency Improvement Pilot (http://www.journalsupplychain.com/), which demonstrated the improved efficiencies of using an institutional identifier in the journal supply chain. The NISO working group will develop a standard for an institutional identifier that can be implemented in all library and publishing environments. The standard will include definition of the metadata required to be collected with the identifier and what uses can be made of that metadata. …

The I2 group is split into a few subgroups which have been charged with looking into how institutional identifiers are used in particular scenarios. These scenarios are e-resources, repositories and e-learning systems, and library resource workflows. The scenario names pain me a bit, but so be it; this is our industry, and there are bigger windmills to tilt at.

I am currently co-chairing the subgroup looking at repositories and e-learning, and apparently I am its "tech lead." I don't want to get caught up on names and roles and titles, though; this series isn't about those at all. I'm just setting the scene and explaining why my head's in this space and laying bare my stake in the issue.

The remainder of this series will provide a bit more detail on the issues around institutional identifiers, share how the repository subgroup is grappling with identifier issues and engaging the repository community to assess needs, propose an approach for an identifier system that may meet said needs, and explore what seems to be the thorniest issue[2].

Notes

1. I offer that very tentatively, knowing what a spectacular failure my last attempt at a series was.
2. Hint: management. I know, "duh," right?

While doing some reading for a little talk my colleague, Ed Summers, and I are giving at code4lib 2009, I came across a paragraph that sparked a crazy thought. So crazy that it's not crazy at all. So not crazy that I am sure other people have thought of it. But nonetheless, here I am writing about it just in case.

From Sarah Currier's paper on SWORD (emphasis mine):

One of the most frequently cited barriers to academics depositing their teaching materials into repositories is the keystroke-count involved in logging into a repository, uploading the resource, creating metadata, perhaps selecting a licence, and publishing the resource. It was a quick win, therefore, to create a drag-and-drop desktop tool to allow a single keystroke deposit of resources, including multiple resources in one action. For a repository that supports automatic metadata generation, administrative metadata can be created at the point of entry to the repository without the user needing to create any.

And I wondered how many repositories supported automatic metadata generation. I wondered how many repositories supported automatic generation of rich metadata. And lastly I wondered, might this be a more or less natural role for catalogers: augmenting stub metadata records or doing original cataloging for institutional repository deposits? Especially at a time when many of them are being reclassified as acquisitions specialists or digital projects managers?

Potential issues and questions:

• Author ignorance: Maybe catalogers are already doing this and I'm a moron?
• Scale: Is it realistic to expect to be able to "keep up" with repository deposits?
• Granularity: Does cataloging at the level of articles, and perhaps at even finer granularities, introduce challenges?
• Duplication: If pre-prints are cataloged in the IR, for instance, will they need to be cataloged again later?
• … there are others I thought of on my commute this morning but have since forgotten them. Feel free to add comments.

I will admit here that I've been somewhat out of the (academic) institutional repository space a while, and cataloging is something I don't share thoughts about very often because my exposure is limited to having taken one course a couple years ago.

I assume there's a body of research about this out there somewhere but I figured I'd post this anyway.