Greenstone tutorial exercise

Back to wiki
Back to index
Devised for Greenstone version: 2.88|3.09
Modified for Greenstone version: 2.87|3.11

Using the UnknownConverterPlugin to make unsupported document formats searchable

This is an advanced tutorial, in that it not only supposes you have familiarised yourself with most of what you've learned in preceding tutorials, but that you're also comfortable with downloading and installing software from the web, and have a little familiarity with using image editing software.

The UnknownConverterPlugin builds on the idea of the UnknownPlugin, in that it can be configured to handle documents of unknown format and file extension. It can also be made to handle documents with known file extensions in a custom manner.

The UnknownConverterPlugin extends the UnknownPlugin's abilities by letting you launch a tool you have installed on your own PC that can be run from the commandline to convert from the "unknown" file format to either text, html or gif/jpg/png images, or a folder of these. If you know how to launch this tool from the commandline to do the conversion, then you would configure the UnknownConverterPlugin by supplying the file format (file extension) of the documents it should process, the expected output file format (text, html or paged images), and the tool's conversion command that the UnknownConverterPlugin should launch to perform the conversion. In place of the input file and the output file or folder, you provide placeholders in the command to run. Once configured, the UnknownConverterPlugin will be used during building to process documents that match the specified file format. It will launch the commandline conversion tool with the command provided, and the expected output files as specified can then be processed by Greenstone in the usual manner.

An example scenario would be if your collection contained djvu files, for which Greenstone provides no custom plugin. However, there's a free commandline tool available that can convert from djvu to one of the text based formats that Greenstone can process, text or html. So in this case, you could try using the UnknownConverterPlugin with the commandline tool on djvu files that you've gathered. The result should be that the djvu files in your Greenstone collection are now searchable.

Working with DjVu documents in Greenstone

DjVu (pronounced like the French phrase déjà vu) is a document format suited for archiving digital documents. DjVuLibre, which provides open source tools for processing DjVu documents, describes DjVu as

"a web-centric format and software platform for distributing documents and images. DjVu can advantageously replace PDF, PS, TIFF, JPEG, and GIF for distributing scanned documents, digital documents, or high-resolution pictures. DjVu content downloads faster, displays and renders faster, looks nicer on a screen, and consume less client resources than competing formats. DjVu images display instantly and can be smoothly zoomed and panned with no lengthy re-rendering. DjVu is used by hundreds of academic, commercial, governmental, and non-commercial web sites around the world."

In this part of the tutorial we'll see how to get Greenstone to not just include a collection's DjVu documents, but make them searchable too. There are several tools out there to convert a DjVu document into text or HTML. For instance, Linux users can install the ocrodjvu package and use its djvu2hocr tool to extract the text content in HTML format. Janusz S. Bien, a Greenstone user on the mailing list, has recommended it as being of possible use to Greenstone users, as it's a front-end to OCR programs. In this tutorial, however, we'll look at using djvutxt which is part of the DjVuLibre suite of tools and which is also available for other operating systems like Windows.

Extracting the text from DjVu documents with DjVuLibre's djvutxt

  1. Start up GLI and create a new collection called DjVu Collection.

  1. Visit the 'DjVu-Digital vs. "Super Hero" PDF' page. The page compares a PDF sample document to its equivalent DjVu version and provides download links for both.

    Download their sample DjVu document (originally here) into your DjVu Collection's import folder at Greenstone → web → sites → localsite → collect → djvucoll → import. If you're offline, you can also get this file from sample_files → djvu → superhero.djvu.

  1. Back in GLI, in the Collection view of the Gather pane, right click and select Refresh folder view. You should now see your new document "superhero.djvu" ready to be built.

  1. Head over to the Create pane and build the collection. The document isn't recognised. You can press Preview to confirm that there's nothing much to look at in this collection.

    If you were to search through the Design pane's Document Plugins for a "DjVuPlugin", you wouldn't find one, because Greenstone hasn't got one. Greenstone knows about a lot of common formats, but there's a great many formats that different people like to work with that Greenstone knows nothing about and which Greenstone developers have not created a custom plugin for.

You've already learnt about the UnknownPlugin in the Multimedia tutorial and know that it can be configured to process document formats for which Greenstone has no custom plugin. However, UnknownPlugin cannot index textual document formats that are unknown to Greenstone to make them searchable upon building, because it doesn't know anything about their internal structure and consequently doesn't know how to extract their text content.

This is where the UnknownConverterPlugin comes in. It builds on the idea of the UnknownPlugin, allowing you to work with document formats unknown to Greenstone. But it offers the additional advantage of being able to extract the text of the unknown document, depending on an important proviso: that you have a software tool installed on your machine, one that can be run readily from the commandline, which can perform the process of converting the unknown document format into text or HTML (or a series of images). If the tool can convert the document to text or HTML, Greenstone can proceed as usual to index the content to make it searchable on previewing.

  1. So in order to process the "superhero.djvu" document in our collection, such that its text content gets indexed for searching, we need to do a number of things: find out if there's a free djvu to text conversion tool out there, work out how to run it from the commandline and finally configure the UnknownConverterPlugin to automatically run this commandline tool for us, so Greenstone can take care of the rest.

    We're in luck, because among the DjVu related tools that DjVuLibre provides is one called "djvutxt" that can perform the text extraction for us. DjVuLibre is available for Windows, Mac and Linux:

  1. The next step is to find out how to run DjVuLibre's djvutxt conversion tool from the commandline.

    The general format of the command is

    djvutxt input.djvu output.txt

    Open a DOS prompt on Windows (to do so: press the letter r while holding down the Windows key—the key located between the Alt and Ctrl keys on your keyboard— to launch the Windows Run popup dialog, then type cmd into it and hit Enter), or a terminal on Mac/Linux and experiment to see what it takes to convert your Greenstone installation's web/sites/localsite/collect/DjVuColl/superhero.djvu file.

    You may have to invoke djvutxt using its full filepath, in which case on Windows the command would look like:

    C:\PATH\TO\YOUR\djvutxt C:\PATH\TO\YOUR\GS\web\sites\localsite\collect\DjVuColl\import\superhero.djvu C:\PATH\TO\YOUR\GS\superhero.txt

    while on Unix systems the command would look like:

    /PATH/TO/YOUR/djvutxt /PATH/TO/YOUR/GS/web/sites/localsite/collect/DjVuColl/import/superhero.djvu /PATH/TO/YOUR/GS/superhero.txt

    If you're on a Mac and had installed DjView.app into your Mac Applications folder, then the command you run in the Mac Terminal would look something like:

    /Volumes/Macintosh\ HD/Applications/DjView.app/Contents/MacOS/djvutxt /PATH/TO/YOUR/GS/web/sites/localsite/collect/DjVuColl/import/superhero.djvu /PATH/TO/YOUR/GS/superhero.txt

    If you compiled up djvulibre from source, djvutxt will be in /PATH/TO/YOUR/djvulibre/bin/djvutxt.

    Once you have the command working, inspect the output file. You should see mostly legible text in it. Only when you've been able to successfully complete this step should you proceed to the next steps.

Processing DjVu documents with the UnknownConverterPlugin

  1. Now that you know how to run the djvutxt conversion tool from the commandline, open up the DjVu Collection in GLI. Go into the Design pane's Document Plugins section and add a new UnknownConverterPlugin instance. (There's already one in the Document Plugin pipeline, but it is not set up for processing djvu files.) Press <Configure Plugin...> and set up the plugin as follows:

    If you have any spaces in any filepaths in your exec_cmd, make sure to always nest that entire filepath in escaped double quotes (\"), so Greenstone can preserve the spaces in it.

    If any filepaths, other than %%INPUT_FILE and %%OUTPUT are within your Greenstone installation, you can use the %%GSDLHOME, %%GSDL3SRCHOME and %%GSDL3HOME (the latter for Greenstone 3's web folder) as placeholders and write out your filepaths relative to this. For instance, if your DjVuLibre is installed in your Greenstone's ext subfolder, then you would start the filepath to djvutxt with %%GSDL3SRCHOME/ext.

    The value for your exec_cmd field may look something like the following, if you have DjVuLibre installed in C:\Program Files. Note the escaped double quotes bookending the path to djvutxt, to protect spaces in its filepath:

    \"C:\Program Files\DjVuLibre\djvutxt\" %%INPUT_FILE %%OUTPUT

    On Unix systems, adjust the command you ran in the command line to now leave out any backslash protecting spaces in the command's filepaths, but ensure you have escaped double-quotes around such filepaths containing spaces. For example,

    /Volumes/Macintosh\ HD/Applications/DjView.app/Contents/MacOS/djvutxt /PATH/TO/YOUR/GS/web/sites/localsite/collect/DjVuColl/import/superhero.djvu /PATH/TO/YOUR/GS/superhero.txt

    becomes

    \"/Volumes/Macintosh HD/Applications/DjView.app/Contents/MacOS/djvutxt\" %%INPUT_FILE %%OUTPUT

  1. Having sufficiently configured the UnknownConverterPlugin, click on the OK button close the plugin's Configuration dialog. Move to the Create pane and build the collection. Your document has now been recognised. What's more, if you preview it and search for the term "Interoperability", a term that occurs in our collection's superhero.djvu document, you should now get a search result linking to that document. So Greenstone has successfully indexed the document's text, thanks to DjVuLibre's djvutxt tool extracting the text which got fed into the rest of Greenstone's building pipeline.

Associating an icon with DjVu documents in Greenstone

  1. When previewing the search result, you may notice that there's no proper icon for the document superhero.djvu. The Greenstone extracted text variant of the document has an icon, a plain text one. However, the superhero.djvu has the "unknown document format" icon, the one with the question mark on it. We can change this.

  1. Go back to the Design pane to configure your UnknownConverterPlugin once more. This time, enable the srcicon field and set its value to icondjvu.

    This is a macroname we're just inventing, though we're following existing Greenstone convention in naming document icon macros, in that it's of the form "icon<file-extension>".

    Click OK to close the UnknownConverterPlugin configuration dialog. Quit GLI, since there's a little more work to do.

  1. Greenstone doesn't have an icon for DjVu documents, since it doesn't know about the format. If you Google for the djvu icon, you'd probably find the Wikipedia page for it.

    Save one of their DjVu icon images. Then open the image in Windows Paint or GIMP or another image editor, and use the application's scaling feature to scale the image's height or the width (whichever is greater) to anywhere between 26 and 32 pixels. Save the scaled image as a GIF file with the name "idjvu.gif", storing it in your Greenstone installation's web/interfaces/default/images folder. You can also use free online image resizing websites to carry out this step.

    If you're working offline, you can get a resized and ready copy of the idjvu.gif file from sample_files → djvu → idjvu.gif. Put it in your Greenstone 3 installation's web/interfaces/default/images folder.

  1. Greenstone knows nothing about the icondjvu macro we defined as the value for UnknownConverterPlugin's srcicon field, so we have to teach Greenstone about this new macro. Use a text editor to open your Greenstone 3's web/sites/localsite/siteConfig.xml file.

    Locate the line

    <replace macro="_iconunknown_" scope="metadata" text="&lt;img src='interfaces/default/images/iunknown.gif' border='0'/&gt;" resolve="false"/>

    Add a similar line above or below it and adjust it to say:

    <replace macro="_icondjvu_" scope="metadata" text="&lt;img src='interfaces/default/images/idjvu.gif' border='0'/&gt;" resolve="false"/>

    Save the file.

    The above has now associated the icon image we want appearing for the djvu document with the macro we defined for the srcicon field in UnknownConverterPlugin's configuration.

  1. Restart GLI, which will restart the Greenstone server, reloading the siteConfig.xml you have just edited. Rebuild the DjVu Collection again and preview it. This time, when you browse the collection, you should see the djvu icon appearing in place of the unknown icon for your DjVu document.

  1. Having designed your collection to handle DjVu documents, you can now add any other documents, including more DjVu documents. Greenstone should now be able to index the text content of DjVu documents in the collection to make them searchable, in all instances where text can be successfully extracted from them by djvutxt.

    Make the search format statement look like below (you can copy it from sample_files → djvu → formats → format_tweaks.txt), then try searching:

    <gsf:template match="documentNode">
    <td valign="top">
    <gsf:link type="document">
    <gsf:icon type="document"/>
    </gsf:link>
    </td>
    <td valign="top">
    <gsf:link type="source">
    <gsf:choose-metadata>
    <gsf:metadata name="thumbicon"/>
    <gsf:metadata name="srcicon"/>
    </gsf:choose-metadata>
    </gsf:link>
    </td>
    <td>
    <gsf:link type="document">
    <xsl:call-template name="choose-title"/>
    </gsf:link>
    <gsf:switch>
    <gsf:metadata name="equivDocLink"/>
    <gsf:when test="exists">
    Also available as: <gsf:metadata name="equivDocLink"/><gsf:metadata name="equivDocIcon"/><gsf:metadata name="/equivDocLink"/>
    </gsf:when>
    </gsf:switch>
    </td>
    </gsf:template>


Copyright © 2005-2019 by the New Zealand Digital Library Project at the University of Waikato, New Zealand
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled “GNU Free Documentation License.”