τεχνοσοφια » Development

Command-line shuffle

Michael Giarlo — Sat, 26 Sep 2009 20:50:05 +0000

Being a nerd, I tend to like the command-line. When I'm working on my laptop at home, I tend to like listening to music. Before I discovered that mplayer had a really convenient shuffle idiom, I would invoke it thusly (to listen to all my Pavement tracks in shuffle mode):

export IFS=$'\n'
for track in $(find /mnt/upnp/MediaTomb/Audio/Artists/Pavement -name \*.mp3 | ~/bin/shuffle.py); do mplayer $track; done

And the wee shuffle script I whipped together looks like this:

#!/usr/bin/env python
# shuffle.py
 
import sys
import random
 
args = list(sys.stdin)
random.shuffle(args)
sys.stdout.writelines(args)

And here's the convenient shuffle idiom that renders my arg-shuffling script somewhat useless:

find /mnt/upnp/MediaTomb/Audio/Artists/Pavement -name \*.mp3 | mplayer -playlist - -shuffle -loop 0

JSONovich emerges

Michael Giarlo — Mon, 24 Aug 2009 18:18:50 +0000

JSONovich has now emerged from the Mozilla Add-ons sandbox and is available to the masses: http://addons.mozilla.org/en-US/firefox/addon/10122.

Linking World Digital Library Data

Michael Giarlo — Mon, 10 Aug 2009 22:44:29 +0000

As I mentioned earlier, I've been learning about linked data in the context of dropping it into the World Digital Library project. I am hopeful we'll be able to deploy the RDF views[1] before too long. In advance of that, I thought it might be helpful to share a sample of what our RDF would look like. The RDF below represents the WDL item for the U.S. Constitution. I appreciate constructive criticism.

A few things to note:

Mmm, Unicode.
Item types are from the Bibliographic Ontology.
Most of the properties are from the Dublin Core Metadata Element Set ontology, especially used where literals are objects rather than resources identified by URI.
Where possible I dug up or found URIs and used the Dublin Core Metadata Terms ontology.
An item is modeled as an aggregation of its constituent files, as defined in OAI-ORE. The notion here is that an ORE aggregation of an item, as expressed in a resource map which is discoverable via a link header in each item detail page, is a "whole" item, including all of its files[2], metadata, and translations.
I'm also making light use of the NEPOMUK File Ontology to express that constituent files are files, and to be explicit about file sizes so that folks know in advance of retrieving it how large files are.
Links out to DDC (Decimalised Database of Concepts), Lingvoj, DBpedia, and Library of Congress Authorities & Vocabularies (e.g., LC Subject Headings) are included where possible. [3] I'd be especially stoked to hear of other vocabs I might link to. The more linked the data, the better.
The output below is Turtle for readability, but the application will offer up RDF/XML.

The data after the jump:

@prefix rdf:  .
@prefix dc:  .
@prefix dcterms:  .
@prefix nfo:  .
@prefix ore:  .
@prefix rdfs:  .
 

    dc:format "image/jpeg" ;
    nfo:fileSize "259485"^^ ;
    a nfo:FileDataObject .
 

    dc:format "image/jpeg" ;
    nfo:fileSize "267031"^^ ;
    a nfo:FileDataObject .
 

    dc:format "image/gif" ;
    nfo:fileSize "56620"^^ ;
    a nfo:FileDataObject .
 

    dc:format "image/jpeg" ;
    nfo:fileSize "233875"^^ ;
    a nfo:FileDataObject .
 

    dc:format "image/jpeg" ;
    nfo:fileSize "245809"^^ ;
    a nfo:FileDataObject .
 

    dcterms:created "2009-08-10T18:11:25-04:00"^^dcterms:W3CDTF ;
    dcterms:creator  ;
    dcterms:modified "2009-08-10T18:11:25-04:00"^^dcterms:W3CDTF ;
    ore:describes  ;
    a ore:ResourceMap .
 

    dc:created "17 Septembre 1787"@fr, "17 de septiembre de 1787"@es, "17 de setembro de 1787"@pt, "17 сентября 1787 г."@ru, "1787年9月17日"@zh, "September 17, 1787"@en, """١٧ ايلول ١٧٨٧
"""@ar ;
    dc:creator "Constitutional Convention, United States"@en, "Convención Constituyente, Estados Unidos"@es, "Convention constitutionnelle, États-Unis"@fr, "Convenção Constitucional, Estados Unidos"@pt, "Конституционная Конвенция, Соединенные Штаты"@ru, "الاتفاقية الدستورية، الولايات المتحدة"@ar, "制宪会议，美国"@zh ;
    dc:extent "Manuscript (4 pages of parchment)"@en, "Manuscrit (4 pages de parchemin)"@fr, "Manuscrito (4 páginas de pergamino)"@es, "Manuscrito (4 páginas em pergaminho)"@pt, "Рукопись (4 пергаментных страницы)"@ru, "مخطوطة (٤ صفحات من الورق النفيس)"@ar, "手草本（4 页羊皮纸）"@zh ;
    dc:language "Anglais"@fr, "English"@en, "Inglés"@es, "Inglês"@pt, "Английский язык"@ru, "الإنجليزية"@ar, "英语"@zh ;
    dc:publisher "Administração de Registros e Arquivos Nacionais"@pt, "Archives Nationales et Administration des documents (NARA) des États-Unis d'Amérique "@fr, "Los Archivos Nacionales y Administración de Documentos (NARA) de los Estados Unidos de América"@es, "National Archives and Records Administration"@en, "Управление национальных архивов и документов"@ru, "الإدارة الأمريكية للوثائق والسجلات الوطنية"@ar, "美国国家文件与档案管理局"@zh ;
    dc:subject "Constituciones"@es, "Constituições"@pt, "Constitutional & administrative law"@en, "Constitutions"@en, "Constitutions"@fr, "Derecho constitucional y administrativo"@es, "Direito constitucional e administrativo"@pt, "Droit constitutionnel et administratif"@fr, "Politics and government"@en, "Politique et gouvernement"@fr, "Política e governo"@pt, "Política y gobierno"@es, "Конституции"@ru, "Конституционное и административное право"@ru, "Политика и правительство"@ru, "الدساتير"@ar, "السياسة والحكومة"@ar, "القانون الدستوري والإداري."@ar, "宪法"@zh, "宪法 & 行政法"@zh, "政治和政府"@zh ;
    dc:title "Constitución de los Estados Unidos"@es, "Constituição dos Estados Unidos"@pt, "Constitution des États-Unis"@fr, "Constitution of the United States"@en, "Конституция Соединенных Штатов"@ru, "دستور الولايات المتحدة"@ar, "美国宪法"@zh ;
    dcterms:DDC "342" ;
    dcterms:LCSH  ;
    dcterms:alternative "Constitution of the United States"@en ;
    dcterms:dateSubmitted "2009-05-07T06:45:21-04:00"^^dcterms:W3CDTF ;
    dcterms:description "1787 年 5 月 14 日，制宪会议在费城的议会大楼（独立厅）召开，目的是修订《邦联条例》。 由于开始时只有两个州的代表团出席，成员不得不一天天地休会，直到 5 月 25 日与会人数达到法定的七个州。 通过讨论和争辩，6 月中旬时明确显示大会与其修改现有的《联邦条例》不如为政府重新起草一份全新的框架。 整个夏季，代表们都在非公开会议中辩论、起草、重新起草新宪法的条款。 主要的争论问题包括要赋予中央政府多大权利、允许各州在国会中有多少个代表席位以及这些代表应该如何选举产生——由人民直接选举还是由各州立法人员选举产生。 这部宪法是很多人智慧的结晶，是合作政治运作和妥协艺术的典范。"@zh, "A Convenção Federal reuniu-se na Casa de Estado (Hall da Independência), em Filadélfia, em 14 de maio de 1787 para revisar os Artigos da Confederação. Em virtude de estarem presentes, inicialmente, as delegações de apenas dois estados, os membros suspenderam os trabalhos, dia após dia, até que fosse atingido o quórum de sete estados em 25 de maio. Através de discussões e debates ficou claro, em meados de junho que, em vez de alterar os atuais artigos da Confederação, a convenção deveria elaborar uma estrutura inteiramente nova para o governo. Ao longo de todo o verão, os delegados debateram, elaboraram e reelaboraram os artigos da nova Constituição em sessões fechadas. Entre os principais pontos em questão estavam o grau de poder permitido ao governo central, o número de representantes no Congresso para cada Estado, e como estes representantes deveriam ser eleitos - diretamente pelo povo ou pelos legisladores do estado. A Constituição foi o trabalho de muitas mentes e permanece como um modelo de cooperação entre lideranças políticas e da arte da condescendência."@pt, "La Convención Federal se reunió en la Cámara del Estado (Salón de la Independencia) en Filadelfia el 14 de mayo de 1787, para revisar los artículos de la Confederación. Debido a que las delegaciones de sólo dos estados estuvieron presentes inicialmente, los miembros levantaron sesión de un día para el siguiente hasta que se obtuvo un quórum de siete estados el 25 de mayo. A través de la discusión y el debate se hizo evidente a mediados de junio que, en lugar de modificar los actuales artículos de la Confederación, la convención prepararía un marco totalmente nuevo para el gobierno. Durante todo el verano, los delegados debatieron, prepararon y redactaron nuevamente los artículos de la nueva Constitución en sesiones a puerta cerrada. Entre los principales puntos en cuestión estuvieron cuánto poder otorgar al gobierno central, el número de representantes en el Congreso que se iban a permitir a cada Estado y la forma en que estos representantes debían ser elegidos, directamente por el pueblo o por los legisladores estatales. La Constitución fue el resultado del trabajo de muchas mentes y se erige como modelo de cooperación política y del arte del compromiso."@es, "La Convention Fédérale s'assembla dans la Chambre Législative (Independence Hall) à Philadelphie le 14 mai 1787, pour réviser les articles de la Confédération. En raison de la seule présence initiale des délégations de deux États, les membres ajournèrent d'un jour à l'autre jusqu'à ce que le quorum de sept États soit obtenu le 25 mai. Â travers les discussions et les débats, il devint clair dès la mi-juin que, plutôt que de modifier les articles existants de la Confédération, la convention allait plutôt ébaucher un cadre entièrement nouveau pour le gouvernement. Tout au long de l'été, les délégués débattirent, élaborèrent, et remanièrent les articles de la nouvelle Constitution, à huis clos. Les principaux points litigieux portaient sur la puissance à accorder au gouvernement central, sur le nombre de représentants au Congrès pour chaque État, et sur le mode d'élection de ces représentants - directement par le peuple ou par les législateurs de l'état. La Constitution fut l'œuvre de nombreux esprits et reste un modèle de coopération politique et de l'art du compromis."@fr, "The Federal Convention convened in the State House (Independence Hall) in Philadelphia on May 14, 1787, to revise the Articles of Confederation. Because the delegations from only two states were present initially, the members adjourned from one day to the next until a quorum of seven states was obtained on May 25. Through discussion and debate it became clear by mid-June that, rather than amend the existing Articles of Confederation, the convention would draft an entirely new framework for the government. All through the summer, the delegates debated, drafted, and redrafted the articles of the new Constitution in closed sessions. Among the chief points at issue were how much power to allow the central government, how many representatives in Congress to allow each state, and how these representatives should be elected--directly by the people or by the state legislators. The Constitution was the work of many minds and stands as a model of cooperative statesmanship and the art of compromise."@en, "Федеральное собрание собралось на заседание в Доме правительства (зал Независимости) 14 мая 1787 года для пересмотра Статей Конфедерации. Поскольку вначале на заседании присутствовали представители только двух штатов, Собрание было распущено на несколько дней до тех пор, пока 25 мая не был обеспечен кворум из представителей семи штатов. В ходе дискуссий и дебатов к середине июня стало понятно, что собрание было намерено скорее составить новый вариант структуры правительства, нежели чем пересматривать существующие Статьи Конфедерации. В течение всего лета делегаты обсуждали, составляли черновые варианты статей новой Конституции и тут же их пересматривали в ходе закрытых заседаний. Среди основных обсуждавшихся вопросов были вопросы степени власти и полномочий, которыми должно быть наделено центральное правительство, количества представителей в Конгрессе от каждого штата, а также процедуры переизбрания этих представителей — непосредственно жителями штатов или законодательными собраниями штатов. Конституция была плодом работы многих политиков и является ярким примером сотрудничества государственных деятелей и искусства компромисса."@ru, "اجتمع ممثلو الاتحاد الفدرالي في قصر الدولة (قاعة الاستقلال) في فيلادلفيا يوم ١٤  أيار ١٧٨٧ لتعديل النظام الأساسي للاتحاد. وحيث حضر وفدان اثنان فقط من وفود الولايات في البداية، رفع الأعضاء الحضور الجلسة من يوم إلى آخر حتى اكتمل النصاب القانوني بحضور وفود سبع ولايات في ٢٥ أيار. وقد اتضح خلال المناقشات والحوار بحلول منتصف حزيران أنه بدلا من تعديل مواد الاتحاد الكونفدرالي القائمة، كان على المؤتمرين صياغة إطار جديد تماما بالنسبة للحكومة. وطوال ذلك الصيف، ناقش المندوبون وصاغوا ثم أعادوا صياغة مواد الدستور الجديد في جلسات مغلقة. ومن بين النقاط الرئيسية التي دار حولها الجدل مدى صلاحيات الحكومة المركزية وعدد الممثلين في الكونغرس لكل ولاية ، وكيفية انتخاب هؤلاء ممثلين -- بالانتخاب المباشر من الشعب أو من قبل مشرّعي الولايات. لقد كان الدستور من عمل عقول كثيرة وهو يمثل نموذجا لفن الحكم التعاوني حنكة التوصل إلى الحلول الوسط."@ar ;
    dcterms:identifier "http://localhost/item/2708/about.rdf#item" ;
    dcterms:language  ;
    dcterms:publisher  ;
    dcterms:spatial , , "América del Norte"@es, "América do Norte"@pt, "Amérique du Nord"@fr, "Estados Unidos da América"@pt, "Estados Unidos de América"@es, "North America"@en, "United States of America"@en, "États-Unis d'Amérique"@fr, "Северная Америка"@ru, "Соединенные Штаты Америки"@ru, "أمريكا الشمالية"@ar, "الولايات المتحدة الأمريكية"@ar, "北美"@zh, "美国"@zh ;
    dcterms:subject  ;
    dcterms:temporal "1700 AD - 1799 AD"@en, "1700 ap. J.-C. - 1799 ap. J.-C."@fr, "1700 d.C. - 1799 d.C."@es, "1700 d.C. - 1799 d.C."@pt, "1700 н.э. - 1799 н.э."@ru, "1700 公元 - 1799 公元"@zh, "١٧٠٠ م - ١٧٩٩ م"@ar ;
    dcterms:title  ;
    ore:aggregates , , , ,  ;
    ore:isDescribedBy  ;
    a  ;
    rdfs:seeAlso  .

Notes

Sadly, the URIs are uglyish due to some constraints from our caching configuration. I figure we can redirect uglyish URIs to cool ones and make use of owl:sameAs if those constraints go away.
sans certain low-quality derivatives such as small thumbnails and tiles for the zoom interface
I was poking through the DBpedia output for Geonames URIs as well, but my method was way too slow and clunky, so that's disabled for the time being. Clients can always follow their noses from the DBpedia output.

Validating ORE from the Command-line

Michael Giarlo — Fri, 31 Jul 2009 18:52:54 +0000

I've been periodically poking at getting Linked Data/RDF views hooked into the World Digital Library web application, following Ed Summers' lead from his work on Chronicling America. The RDF views also use the OAI-ORE vocabulary to express aggregations — in WDL, an item is an aggregation of its constituent files. The goal is to provide a semantically rich and holistic representation of a WDL item (identifier, constituent files, metadata, translations, and so on).

The ORE format is a new one for me so it's hard to say whether the output of my dev branch is valid ORE or not. Plus I'm a sucker for validators. Turns out Rob Sanderson has developed a Python library for validating ORE, and this little snippet is what I've been using to validate the ORE. I didn't put much effort into making it readable, so much as banging something functional out so I can meet deadlines, so mea culpa and all that. But without further hemming and hawing, the code:

# validate.py
import sys
from foresite import *
 
rem = RdfLibParser().parse(ReMDocument(sys.argv[1]))
aggr = rem.aggregation
n3 = RdfLibSerializer('n3')
rem2 = aggr.register_serialization(n3)
print rem2.get_serialization(n3).data

Most of this code is naively copied and pasted from Rob's excellent Foresite documentation.

I invoke it thusly: python validate.py {URL}

And the output:

@prefix _27: .
@prefix _28: .
@prefix _29: .
@prefix bibo: .
@prefix dc: .
@prefix dcterms: .
@prefix ore: .
@prefix rdf: .
@prefix rdfs: .
@prefix rdfs1: .
 
 _28:ResourceMap a ore:ResourceMap;
     dc:format "text/rdf+n3";
     dcterms:created "2009-07-31T14:23:31Z";
     dcterms:modified "2009-07-31T14:23:31Z";
     ore:describes _29:id. 
 
 _29:id a bibo:Image,
         ore:Aggregation;
     dcterms:DDC "973";
     dcterms:alternative "Antietam, Maryland. Allan Pinkerton, President Lincoln, and Major General John A. McClernand"@en;
     dcterms:created "1862年10月3日"@zh,
         "3 de octubre de 1862"@es,
         "3 de outubro de 1862"@pt,
         "3 octobre 1862"@fr,
         "3 октября 1862 года"@ru,
         "October 3, 1862"@en,
         " ٣ آكتوبر، ١٨٦٢"@ar;
     dcterms:creator "Gardner, Alexander"@en,
         "Gardner, Alexander"@es,
         "Gardner, Alexander"@fr,
         "Gardner, Alexander"@pt,
         "Гарднер, Александр"@ru,
         "جاردنر, أليكسندر"@ar,
         "加德纳, 亚历山大"@zh;
... (and so on and so forth)
     dcterms:title "Antietam, Maryland. Allan Pinkerton, President Lincoln, and Major General John A. McClernand: Another View"@en,
         "Antietam, Maryland. Allan Pinkerton, el Presidente Lincoln y el General Principal John A. McClernand: Otra visión"@es,
         "Antietam, Maryland. Allan Pinkerton, le président Lincoln et le général-major John A. McClernand: Autre vue"@fr,
         "Antietam, Maryland. Allan Pinkerton,  Presidente Lincoln e Major-General John A. McClernand: Outra Vista"@pt,
         "Антитэм, штат Мэриленд. Аллан Пинкертон, президент Линкольн и генерал-майор Джон А. Макклернанд: Другой снимок"@ru,
         "أنتينام، ميريلاند ألان بينكرتون، الرئيس لينكولن، واللواء جون أ. ماكليرناند: منظر آخر"@ar,
         "安蒂特姆，马里兰州 艾伦·平克顿、林肯总统和少将约翰·A ·马克克拉南: 另一个视角"@zh;
     ore:aggregates ,
         ;
     ore:isDescribedBy ;
     rdfs:seeAlso . 
 
  a _27:FileDataObject;
     dcterms:format "image/gif";
     _27:fileSize "34531"^^. 
 
  a _27:FileDataObject;
     dcterms:format "image/tiff";
     _27:fileSize "1301614"^^. 
 
 ore:Aggregation rdfs1:isDefinedBy ;
     rdfs1:label "Aggregation". 
 
 ore:ResourceMap rdfs1:isDefinedBy ;
     rdfs1:label "ResourceMap".

You might pick up on some warts I have yet to fix, but there you go.

WDL metadata mapping, and, parsing TEI in Python

Michael Giarlo — Mon, 13 Jul 2009 22:27:46 +0000

Context

Early on in the effort to develop the first public version of the World Digital Library web application, we developed a (non-public) Django-based cataloging application where Library of Congress catalogers could manage metadata for WDL items. Management in this sense includes creation of records, editing of records, versioning of edits, mapping of source records, and some light workflow for assignment of records to individual catalogers and for hooking into translation processes[1].

I worked primarily on the source record mapping tools. They take a number of formats as input and are called by the cataloging application to map metadata from these formats into the WDL domain model. Several though not all of which are XML-based, and thus easily dealt with in Python, via the etree module in the lxml package.

Dan recently kicked off a new R&D project for evaluating (any) metadata against any number of metadata profiles, mapping into a generic data dictionary, the goal being to determine how feasible it would be to develop a toolset for aiding remediation of metadata across any number of digital collections. I have been working on this project with Dan, and got started by seeing how generalizable the WDL metadata mapping tools are. Turns out they're fairly generalizable once you tweak the various format-specific mapping rules to map into the generic data dictionary model rather than the WDL model (around 15 elements, and somewhere between Dublin Core and MODS in terms of specificity but flatly structured like DC).

Some of the test data I am working with now, that has nothing to do with WDL, is SGML-based TEI 2 markup. The closest I worked with on WDL was TEI P5 for manuscript description which is serialized in XML. Turns out my TEI mapping rules from before blew up on this TEI 2 stuff, as lxml.etree (naturally) wasn't digging the non-XML input. I googled around a bit for how best to parse TEI (or any SGML) in Python and then discovered it's actually simple as pie.

Code

If you've got the BeautifulSoup module installed[2]:

>>> from BeautifulSoup import BeautifulSoup
>>> tei = open('foo.sgm').read()
>>> BeautifulSoup(tei).findAll('title')[0].string
u'[Memorandum to Dr. Botkin]: a machine readable transcription.'

If not, the lxml.html module works too:

>>> from lxml import html
>>> h = html.parse(open('foo.sgm'))
>>> h.xpath('//title')[0].text
'[Memorandum to Dr. Botkin]: a machine readable transcription.'

Data

And here's what the sample data looks like:

"-//Library of Congress - Historical Collections (American Memory)//DTD ammem.dtd//EN" 
[
"07010101.ent"> %images;
]>
>
 type="text" date.created="1994/03/15" date.updated="2002/04/05" status="updated" creator="National Digital Library Program
, Library of Congress">
>
>
 type="aggitemid">wpa0-07010101>
>[Memorandum to Dr. Botkin]: a machine readable transcription.>
>>Life Histories from the Folklore Project, WPA Federal Writers' Project, 1936-1940; American Memory, Library of Congress.> type="aggid">>
>
>
>Selected and converted.>
>American Memory, Library of Congress.>
>>
>
>Washington, DC, 1994.>
>Preceding element provides place and date of transcription only.>
>For more information about this text and this American Memory collection, refer to accompanying matter.>
>
>
>>
>U.S. Work Projects Administration, Federal Writers' Project (Folklore Project, Life Histories, 1936-39); Manuscript Division, Library of Congress.>
>Copyright status not determined; refer to accompanying matter.>>
>
>
>>The National Digital Library Program at the Library of Congress makes digitized historical materials available for education and scholarship.>>
>>This transcription is intended to have an accuracy of 99.95 percent or greater and is not intended to reproduce the appearance of the original work.  The accompanying images provide a facsimile of this work and represent the appearance of the original.>>
>1994/03/15>
>2002/04/05>
>
>
 type="manuscript">
>
>
>
 entity="I07010101">0001>
>>>
>Memorandum to Dr. Botkin from G. B. Roberts, May 26, 1941>
>Subject:  Alabama Material>
>This material has not yet been accessioned and has only 
 rend="overstrike">beeen> been roughly classified as life histories, folklore, and miscellaneous data and copy save in the case of the 2 ex-slave items and the essay on Jesse Owens, each of which was recommended.>
>Total no. of items recommended:  3 (14 pp.) 
>In progress>>>>>>

Notes

Catalogers cataloged stuff in the English language, but every metadata record needed to be translated into the other six U.N. languages: Spanish, Russian, French, Arabic, Chinese, and Portuguese.
And you are but one sudo easy_install BeautifulSoup away from that.

JSONovich update

Michael Giarlo — Tue, 30 Dec 2008 01:00:32 +0000

JSONovich is now up to version 0.6. Recent revisions have added the following functionality:

Reads in JSON and converts to UTF-8 for some naive Unicode handling
Wraps long lines at the right edge of the window
Adds a check to see if a native JSON parsing library is already loaded (as will be the case in Firefox 3.1). Uses that library if so, otherwise loads the module included in JSONovich.
Handles JSON syntax errors more gracefully. Used to eat bad data and display nothing, but syntax errors (from the JSON parser) are now surfaced.

I've also tossed the source up on code.google.com for version control.

In the meantime, those of you who are using JSONovich can help increase its exposure by heading over to its entry at addons.mozilla.org, logging in, downloading, rating, and reviewing the extension. Reviews and ratings help get extensions "promoted" from the sandbox to the public site, which provides the ability for automatic updates when new versions of the extension are released.

Molotovs away!

Michael Giarlo — Wed, 24 Dec 2008 03:29:50 +0000

Lest I be criticized for unfairly calling out former employers in my recent Burn the Walled Gardens rant, I share news that the Rutgers University Libraries have boldly ventured into the world of open source software: RUcore Open Source Development. Huzzah! Thanks to the molotov-hurling Shaun Ellis, a peacenik/code monkey/musician extraordinaire, for all of his work and for bringing this to my attention.

On the RUcore open source page you can get a list of ongoing projects, a release schedule, and a rationale for their licensing decisions (i.e., choosing GPL 3).

The first project to be released (as of 2008/12/19) is the METS-based bibliographic utility, OpenMIC:

OpenMIC is an open source, web-based cataloging tool that can be used as a standalone application or integrated with other repository architectures by a wide range of organizations. It provides a complete metadata creation system for analog and digital materials, with services to export these metadata in standard formats.

Low overhead and infrastructure requirements

Events-based model for management and rights documentation

Mapping and import from standard and in-house formats

Unicode and CJK vernacular character support

OpenMIC is a core application for the Moving Image Collections (MIC) initiative developed at the Rutgers University Libraries with funding from the Library of Congress.

I look forward to following along as Rutgers releases yet more of the tools they have developed as part of their impressive digital library infrastructure. It will be even more interesting to hear what their model will be for taking patches / commits from the broader open source community. These things do take time, even though I failed to show an appreciation for that in my original rant, but I am reminded (by Jonathan Rochkind) that it's better to take the time and get it right. I cringe a bit to say that, knowing full well how things tend to languish in committees and fall victim to analysis paralysis in academia; surely there is some middle ground? There are some very talented and experienced folks at Rutgers, so I will be excited to see them take a leadership role in this space.

Go, Scarlet Knights!

JSONovich in the sandbox

Michael Giarlo — Tue, 23 Dec 2008 15:22:58 +0000

JSONovich is now in the "sandbox" over at addons.mozilla.org, where it will remain until it's been tested a bit more, and rated and reviewed by users. Until that point, it will be marked as "experimental" and will require users to login before they can download it. If any of you would like to give JSONovich a quick spin and rate/review it over at the Mozilla add-ons site, that would be solid. Once it's gotten a few reviews and I'm more comfortable about it working cross-platform and cross-version, I'll nominate it to be promoted.

Here's where it lives: https://addons.mozilla.org/en-US/firefox/addon/10122

Much obliged, folks. And thanks to those of you who have already downloaded it, installed it, tested it, left comments, or some combination thereof.

Burn the Walled Gardens

Michael Giarlo — Mon, 15 Dec 2008 23:18:03 +0000

Issue five of the code4lib journal is out. This issue looks to be just as good as the past four issues, but I'd like to highlight one article in particular: the column by Kansas State's Web Development Librarian, Dale Askey: We Love Open Source Software. No, You Can't Have Our Code.

Librarians are among the strongest proponents of open source software. Paradoxically, libraries are also among the least likely to actively contribute their code to open source projects. This article identifies and discusses six main reasons this dichotomy exists and offers ways to get around them.

If you're at a library doing open source software, or if you think you're doing open source software, or if you're considering jumping into the fray, you would do yourself, your institution, your users, and the open source community (whatever the heck that is) a great, great service by reading this column. Srsly.

I've worked in academic libraries where open source was given lip service the likes of which Jimmie Walker would envy and yet, well, show me the code already!

See, here's the thing, if you're doing open source, or think you're doing open source, it is necessary that you release code to the public under an open source license. The public includes your institution, your partner institutions, all of academia, random hackers in East Kaboomistan, and everyone else; if you make code available to partners only, that's not open source; that's multi-institutional closed source. It's a walled garden with teleportation devices leading to other walled gardens. And we're tired of hearing about your damned azaleas. We want to see them, and take cuts from them, and grow our own, and contribute some back to you.

Releasing code is necessary to claim you're doing open source, yes, but it is not sufficient. There is some value in just throwing code over the wall. Sure, once the code is out there you've satisfied a definition or two and you can go off and pat yourself on the back and do the happy Ewok dance and maybe some more grant funds will come your way. But if you want to add value to your involvement in open source, and add value to user-facing services built upon open source software, and add value for the vast community of open source developers champing at the bit to get at your code and make it better and work with you towards crafting a shiny, happy world, for goodness' sake, Mr. Gorbachev, tear down that wall. Stop the madness; no more "just fork it because it wasn't invented here." Take commits from the world. There's the value in open source.

Do we get this? I hope we get this. None of this is new. Some of us in libraries have been banging the open source drum for nearly a decade, some even longer. The rhythm is a well known one, but the drum quite apparently needs yet more beating. And louder beating. Thank you, Dale, for keeping the beat. Now, if only this rhythm section had a bassist.

Introducing JSONovich

Michael Giarlo — Fri, 12 Dec 2008 20:33:21 +0000

JSONovich is a Firefox extension that pretty-prints and colorizes JSON.

Feedback is welcome.