Working with Custom Vocabularies

The ability to work with custom vocabularies is necessary for many use cases. The use cases range from being able to detect various types of named entities specific of a company or to detect and work with concepts from a specific domain - such as life science as shown by the following figure,

Customizing Stanbol for specific Domains

The aim of this usage scenario is to provide Apache Stanbol users with all the required knowledge to customize Apache Stanbol to be used in their specific domain. This includes

Overview

The following figure shows the typical Enhancement workflow that may start with some preprocessing steps (e.g. the conversion of rich text formats to plain text) followed by the Natural Language Processing phase. Next 'Semantic Lifting' aims to connect the results of text processing and link it to the application domain of the user. During Postprocessing those results may get further refined.

![Typical Enhancement Workflow](enhancementworkflow.png "The typical Enhancement Chain includes the

This usage scenario is all about the Semantic Lifting phase. This phase is most central to for how well enhancement results to match the requirements of the users application domain. Users that need to process health related documents will need to provide vocabularies containing life science related entities otherwise the Stanbol Enhancer will not perform as expected on those documents. Similar processing Customer requests can only work if Stanbol has access to data managed by the CRM.

This scenario aims to provide Stanbol users with all information necessary to use Apache Stanbol in scenarios where domain specific vocabularies are required.

Managing Custom Vocabularies with the Stanbol Entityhub

By default the Stanbol Enhancer does use the Entityhub component for linking Entities with mentions in the processed text. While Users may extend the Enhancer to allow the usage of other sources this is outside of the scope of this scenario.

The Stanbol Entityhub provides two possibilities to manage vocabularies

  1. Managed Sites: A fully read/write able storage for Entities. Once created users can use a RESTful interface to create, update, retrieve, query and delete entities.
  2. Referenced Site: A read-only version of a Site that can either be used as a local cache of remotely managed data (such as a Linked Data server) or use a fully local index of the knowledge base - the relevant case in the context of this scenario.

As a rule of thump users should prefer to use a Managed Site if the vocabulary does change regularly and those changes need to be reflected in enhancement results of processed documents. A Referenced Site is typically the better choice for vocabularies that do not change on a regular base and/or for users that what to use apply advanced rules while indexing a dataset.

Using a Entityhub Managed Site

How to use a Managed Site is already described in detail by the Documentation of Managed Sites. To configure a new Managed Site on the Entityhub users need to create two components:

  1. the Yard - the storage component of the Stanbol Entityhub. While there are multiple Yard implementations, when used for EntiyLinking the SolrYard implementation should be used. Second the
  2. the YardSite - the component that implements the ManagedSite interface.

After completing those two steps an empty Managed site should be ready to use available under

http://{stanbol-host}/entityhub/sites/{managed-site-name}/

and users can start to upload the Entities of the controlled vocabulary by using the RESTful interface such as

curl -i -X PUT -H "Content-Type: application/rdf+xml" -T {rdf-xml-data} \
    "http://{stanbol-host}/entityhub/site/{managed-site-name}/entity"

In case you have opted to use a Managed Site for managing your entities you can now skip the next section until section 'B. Configure and use the index with the Apache Stanbol Enhancer'

Using a Entityhub Referenced Site

Referenced Sites are used by the Stanbol Entityhub to reference external knowledge bases. This can be done by configuring remote services for dereferencing and querying information, but also by providing a full local index of the referenced knowledge base.

When using a Referenced Site in combination with the Stanbol Enhancer it is highly recommended for performance considerations to provide a full local index. To create such local indexes Stanbol provides the Entityhub Indexing Tool. See the following section for detailed information on how to use this tool.

Building full local indexes with the Entityhub Indexing Tool

The Entityhub Indexing Tool allows to create full local indexes of knowledge bases that can be loaded to the Stanbol Entityhub as Referenced Sites. Users that do use Managed Sites may want to skip this section.

Users of the Entityhub Indexing Tool will typically need to complete the steps described in the following sub sections.

Step 1 : Compile and assemble the indexing tool

The indexing tool provides a default configuration for creating an Apache Solr index of RDF files (e.g. a SKOS export of a thesaurus or a set of foaf files).

To build the indexing tool from source - recommended - you will need to checkout Apache Stanbol form SVN (or download a source-release). Instructions for this can be found here. However if you want to skip this you can also obtain a binary version from the IKS development server (search the sub-folders of the different versions for a file named like "org.apache.stanbol.entityhub.indexing.genericrdf-*-jar-with-dependencies.jar").

In case you downloaded or "svn co" the source to {stanbol-source} and successfully build the source as described in the Tutorial you still need to assembly the indexing tool by

$ cd {stanbol-source}/entityhub/indexing/genericrdf/
$ mvn assembly:single

and move the assembled indexing tool from

{stanbol-source}/entityhub/indexing/genericrdf/target/ _
org.apache.stanbol.entityhub.indexing.genericrdf-*-jar-with-dependencies.jar

into a the directory you plan to use for the indexing process. We will refer to this directory as {indexing-working-dir}.

Step 2 : Create the index

Initialize the tool with

$ cd {indexing-working-dir}
$ java -jar org.apache.stanbol.entityhub.indexing.genericrdf-*-jar-with-dependencies.jar init

This will create/initialize the default configuration for the Indexing Tool including (relative to {indexing-working-dir}):

After the initialization you will need to provide the following configurations in files located in the configuration folder ({indexing-working-dir}/indexing/config)

Finally you will also need to copy your source files into the source directory {indexing-working-dir}/indexing/resources/rdfdata. All files within this directory will be indexed. THe indexing tool support most common RDF serialization. You can also directly index compressed RDF files.

For more details about possible configurations, please consult the README.

Once all source files are in place, you can start the index process by running

$ cd {indexing-working-dir}
$ java -Xmx1024m -jar org.apache.stanbol.entityhub.indexing.genericrdf-*-jar-with-dependencies.jar index

Depending on your hardware and on complexity and size of your sources, it may take several hours to built the index. As a result, you will get an archive of an Apache Solr index together with an OSGI bundle to work with the index in Stanbol. Both files will be located within the indexing/dist folder.

IMPORTANT NOTES:

Step 3 : Initialize the index within Apache Stanbol

We assume that you already have a running Apache Stanbol instance at http://{stanbol-host} and that {stanbol-working-dir} is the working directory of that instance on the local hard disk. To install the created index you need to

You find both files in the {indexing-working-dir}/indexing/dist/ folder.

After the installation your data will be available at

http://{stanbol-instance}/entityhub/site/{name}

You can use the Web UI of the Stanbol Enhancer to explore your vocabulary. Note, that in case of big vocabulary it might take some time until the site becomes functional.

Configuring the Stanbol Enhancer for your custom Vocabularies

This section covers how to configure the Apache Stanbol Enhancer to recognize and link entities of your custom vocabulary with processed documents.

Generally there are two possible ways you can use to recognize entities of your vocabulary:

  1. Named Entity Linking: This first uses Named Entity Recoqunition (NER) for spotting "named entities" in the text and second try to link those named entities with entities defined in your vocabulary. This approach is limited to entities with the type person, organization and places. So if your vocabulary contains entities of other types, they will not be recognized. In addition it also requires the availability of NER for the language(s) of the processed documents.
  2. Keyword Linking: This uses the labels of entities in your vocabulary for the recognition and linking process. Natural Language Processing (NLP) techniques such as part-of-speach (POS) detection can be used to improve performance and results but this works also without NLP support. As extraction and linking is based on labels mentioned in the analyzed content this method has no restrictions regarding the types of your entities.

For more information about this you might also have a look at the introduction of the multi lingual usage scenario.

TIP: If you are unsure about what to use you can also start with configuring both options to give it a try.

Depending on if you want to use named entity linking or keyword linking the configuration of the enhancement chain and the enhancement engine making use of your vocabulary will be different.

Configuring Named Entity Linking

In case named entity linking is used the linking with the custom vocabulary is done by the Named Entity Tagging Engine. For the configuration of this engine you need to provide the following parameters

  1. The "name" of the enhancement engine. It is recommended to use "{name}Linking" - where {name} is the name of the Entityhub Site (ReferenceSite or ManagedSite).
  2. The name of the referenced site holding your vocabulary. Here you have to configure the {name}.
  3. Enable/disable persons, organizations and places and if enabled configure the rdf:type used by your vocabulary for those type. If you do not want to restrict the type, you can also leave the type field empty.
  4. Define the property used to match against the named entities detected by the used NER engine(s).

For more detailed information please see the documentation of the Named Entity Tagging Engine.

Note, that for using named entity linking you need also ensure that an enhancement engine that provides NER (Named Entity Recoqunition) is available in the enhancement chain. See Stanbol NLP processing Language Support section for detailed information on Languages with NER support.

The following Example shows a enhancement chain for named entity linking based on OpenNLP and CELI as NLP processing modules

Both the weighted chain and the list chain can be used for the configuration of such a chain.

Configuring Named Entity Linking

First it is important to note the difference between Named Entity Linking and Entity Linking. While Named Entity Linking only considers Named Entities detected by NER (Named Entity Recognition) Entity Linking does work on Words (Tokens). Because of that is has much lower NLP requirements and can even operate for languages where only word tokenization is supported. However extraction results AND performance do greatly improve with POS (Part of Speech) tagging support. Also Chunking (Noun Phrase detection), NER and Lemmatization results can be consumed by Entity Linking to further improve extraction results. For details see the documentation of the Entity Linking Process.

The second big difference is that Named Entity Linking can only support Entity types supported by the NER modles (Persons, Organizations and Places). Entity Linking does not have this restriction. This advantage comes also with the disadvantage that Entity Lookups to the Controlled Vocabulary are only based on Label similarities. Named Entity Linking does also use the type information provided by NER.

To use Entity Linking with a custom Vocabulary Users need to configure an instance of the Entityhub Linking Engine. While this Engine provides more than twenty configuration parameters the following list provides an overview about the most important. For detailed information please see the documentation of the Engine.

  1. The "Name" of the enhancement engine. It is recommended to use something like "{name}Extraction" - where {name} is the name of the Entityhub Site
  2. The name of the "Managed- / Referenced Site" holding your vocabulary. Here you have to configure the {name}
  3. The "Label Field" is the URI of the property in your vocabulary providing the labels used for matching. You can only use a single field. If you want to use values of several fields you have two options: (1) to adapt your indexing configuration to copy the values of those fields to a single one (e.g. the values of "skos:prefLabel" and "skos:altLabel" are copied to "rdfs:label" in the default configuration of the Entityhub indexing tool (see {indexing-working-dir}/indexing/config/mappings.txt) (2) to configure multiple EntityubLinkingEngines - one for each label field. Option (1) is preferable as long as you do not need to use different configurations for the different labels.
  4. The "Link ProperNouns only": If the custom Vocabulary contains Proper Nouns (Named Entities) than this parameter should be activated. This options causes the Entity Linking process to not making queries for commons nouns and by that receding the number of queries agains the controlled vocabulary by ~70%. However this is not feasible if the vocabulary does contain Entities that are common nouns in the language.
  5. The "Type Mappings" might be interesting for you if your vocabulary contains custom types as those mappings can be used to map 'rdf:type's of entities in your vocabulary to 'dc:type's used for 'fise:TextAnnotation's - created by the Apache Stanbol Enhancer to annotate occurrences of extracted entities in the parsed text. See the type mapping syntax and the usage scenario for the Apache Stanbol Enhancement Structure for details.

The following Example shows an Example of an enhancement chain using OpenNLP for NLP

Both the weighted chain and the list chain can be used for the configuration of such a chain.

The documentation of the Stanbol NLP processing module provides detailed Information about integrated NLP frameworks and suupported languages.

How to use enhancement chains

In the default configuration the Apache Stanbol Enhancer provides several enhancement chains including:

1) a "default" chain providing Named Entity Linking based on DBpedia and Entity Linking based on the Entityhub 2) the "language" chain that is intended to be used to detect the language of parsed content. 3) a "dbpedia-proper-noun-linking" chain showing Named Entity Linking based on DBpedia

Change the enhancement chain bound to "/enhancer"

The enhancement chain bound to

http://{stanbol-host}/enhancer

is determined by the following rules

  1. the chain with the name "default". If more than one chain is present with that name, than the above rules for resolving name conflicts apply. If none,
  2. the chain with the highest "service.ranking". If several have the same ranking,
  3. the chain with the lowest "service.id".

You can change this by configuring the names and/or the "service.ranking" of the enhancements chains. Note, that (2) and (3) are also used to resolve name conflicts of chains. If you configure two enhancement chains with the same name, only the one with the highest "service.ranking" and lowest "service.id" will be accessible via the restful API.

Examples

While this usage scenario provides the basic information about how to index/use custom vocabularies there are a lot of additional possibilities to configure the indexing process and the enhancement engines.

If you are interested in the more advanced options, the following resources/examples might be of interest to you.

Demos and Resources

The IKS development server runs an rather advanced configuration of Stanbol with a lot of custom datasets and according Enhancement Chain configuration. You can access this Stanbol instance at http://dev.iks-project.eu:8081/. In addition this server also hosts a set of prebuilt indexes