This project has retired. For details please refer to its Attic page.
Apache Stanbol -

Basic Chinese language support based on Lucene Smartcn Analyzer

As Chinese does not use Whiespace characters for word tokenization the default tokenizers used by Stanbol are not capable to properly process Chinese language texts. Therefore users that need to process Chinese texts need to add special modules even for basic language support.

The integration of the Stanbol NLP processing module with the Lucene Smartcn Analyzer provides this by


The Smartcn integration consists of three bundles as referenced by the Smartcn Bundle List. Users can either include this BundleList in their Custom Launcher configuration by including the


or alternatively manually installing the tree bundles referenced by the Smartcn Bundle List to the Stanbol Environment (e.g. by copying them to the stanbol/fileinstall directory)

Stanbol Enhancer configuration

When the Smartcn Analyzer is installed to the Stanbol Environment two new EnhancementEngiens will be available that can be used to configure an EnhancementChain for Chinese texts. A typical EnhancementChain for Chinese text will look like:


where '{your-entitylinking}' will typically be an EntityhubLinkingEngine engine configured for your vocabulary containing the Entities with Chinese labels.

Solr Configuration

When you plan to use the Smartcn Analyzer to process Chinese texts it is important to also properly configure the Solr schema.xml used by the Entityhub SolrYard.

For that you will need to add two things:

  1. A fieldType specification for Chinese

    <fieldType name="text_zh" class="solr.TextField" positionIncrementGap="100">
        <analyzer type="index">
            <tokenizer class="solr.SmartChineseSentenceTokenizerFactory"/>
            <filter class="solr.SmartChineseWordTokenFilterFactory"/>
            <filter class="solr.LowerCaseFilterFactory"/>
            <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
        <analyzer type="query">
            <tokenizer class="solr.SmartChineseSentenceTokenizerFactory"/>
            <filter class="solr.SmartChineseWordTokenFilterFactory"/>
            <filter class="solr.LowerCaseFilterFactory"/>
            <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
            <filter class="solr.PositionFilterFactory" />
  2. A dynamic field using this field type that matches against Chinese language literals

     Dynamic field for Chinese languages.
    <dynamicField name="@zh*" type="text_zh" indexed="true" stored="true" multiValued="true" omitNorms="false"/>

The is identical with the default configuration but uses the above fieldType and dynamicField specification.

Usage with the EntityhubIndexing Tool

  1. Extract the to the "indexing/config" directory
  2. Rename the "indexing/config/smartcn" directory to the {site-name} (the value of the "name" property of the "indexing/config/" file).

As an alternative to (2) you can also explicitly configure the name of the solr config as value to the "solrConf:smartcn" of SolrYardIndexingDestination.


Usage with the Entityhub SolrYard

If you want to create an empty SolrYard instance using the configuration you will need to

  1. copy the to the datafile directory of your Stanbol instance ({working-dir}/stanbol/datafiles)
  2. rename it to the {name} of the SolrYard you want to create. The file name needs to be {name}
  3. create the SolrYard instance and configure the "Solr Index/Core" (org.apache.stanbol.entityhub.yard.solr.solrUri) to {name}. Make sure the "Use default SolrCore configuration" (org.apache.stanbol.entityhub.yard.solr.useDefaultConfig) is disabled.

If you want to use the as default you can rename the file in the datafilee folder to "" and the enable the "Use default SolrCore configuration" (org.apache.stanbol.entityhub.yard.solr.useDefaultConfig) when you configure a SolrYard instance.

See also the documentation on how to configure a managed site.