LuceneSail (Eclipse RDF4J 3.0.5-SNAPSHOT API)

java.lang.Object
- org.eclipse.rdf4j.sail.helpers.SailWrapper
- - org.eclipse.rdf4j.sail.helpers.NotifyingSailWrapper
  - - org.eclipse.rdf4j.sail.lucene.LuceneSail

All Implemented Interfaces:

FederatedServiceResolverClient, NotifyingSail, Sail, StackableSail
```
public class LuceneSail
extends NotifyingSailWrapper
```
A LuceneSail wraps an arbitrary existing Sail and extends it with support for full-text search on all Literals.
Setting up a LuceneSail
LuceneSail works in two modes: storing its data into a directory on the harddisk or into a RAMDirectory in RAM (which is discarded when the program ends). Example with storage in a folder:
```
 // create a sesame memory sail
 MemoryStore memoryStore = new MemoryStore();

 // create a lucenesail to wrap the memorystore
 LuceneSail lucenesail = new LuceneSail();
 // set this parameter to store the lucene index on disk
 lucenesail.setParameter(LuceneSail.LUCENE_DIR_KEY, "./data/mydirectory");

 // wrap memorystore in a lucenesail
 lucenesail.setBaseSail(memoryStore);

 // create a Repository to access the sails
 SailRepository repository = new SailRepository(lucenesail);
 repository.initialize();
 
```
Example with storage in a RAM directory:
```
 // create a sesame memory sail
 MemoryStore memoryStore = new MemoryStore();

 // create a lucenesail to wrap the memorystore
 LuceneSail lucenesail = new LuceneSail();
 // set this parameter to let the lucene index store its data in ram
 lucenesail.setParameter(LuceneSail.LUCENE_RAMDIR_KEY, "true");

 // wrap memorystore in a lucenesail
 lucenesail.setBaseSail(memoryStore);

 // create a Repository to access the sails
 SailRepository repository = new SailRepository(lucenesail);
 repository.initialize();
 
```
Asking full-text queries
Text queries are expressed using the virtual properties of the LuceneSail. An example query looks like this (SERQL): SELECT Subject, Score, Snippet FROM {Subject} {} {}; {"my Lucene query"}; {Score}; {Snippet} In SPARQL: SELECT ?subject ?score ?snippet ?resource WHERE { ?subject [ a ; "my Lucene query" ; ?score ; ?snippet ; ?resource ] } When defining queries, these properties type and query are mandatory. Also, the matches relation is mandatory. When one of these misses, the query will not be executed as expected. The failure behavior can be configured, setting the Sail property "incompletequeryfail" to true will throw a SailException when such patterns are found, this is the default behavior to help finding inaccurate queries. Set it to false to have warnings logged instead. Multiple queries can be issued to the sail, the results of the queries will be integrated. Note that you cannot use the same variable for multiple Text queries, if you want to combine text searches, use Lucenes query syntax.
Fields are stored/indexed
All fields are stored and indexed. The "text" fields (gathering all literals) have to be stored, because when a new literal is added to a document, the previous texts need to be copied from the existing document to the new Document, this does not work when they are only "indexed". Fields that are not stored, cannot be retrieved using full-text querying.
Deleting a Lucene index
At the moment, deleting the lucene index can be done in two ways:
- Delete the folder where the data is stored while the application is not running
- Call the repository's RepositoryConnection.clear(org.eclipse.rdf4j.model.Resource[]) method with no arguments. clear(). This will delete the index.
Handling of Contexts
Each lucene document contains a field for every contextIDs that contributed to the document. NULL contexts are marked using the String LuceneIndex#CONTEXT_NULL ("null") and stored in the lucene field LuceneIndex#CONTEXT_FIELD_NAME ("context"). This means that when adding/appending to a document, all additional context-uris are added to the document. When deleting individual triples, the context is ignored. In clear(Resource ...) we make a query on all Lucene-Documents that were possibly created by this context(s). Given a document D that context C(1-n) contributed to. D' is the new document after clear(). - if there is only one C then D can be safely removed. There is no D' (I hope this is the standard case: like in ontologies, where all triples about a resource are in one document) - if there are multiple C, remember the uri of D, delete D, and query (s,p,o, ?) from the underlying store after committing the operation- this returns the literals of D', add D' as new document This will probably be both fast in the common case and capable enough in the multiple-C case.
Defining the indexed Fields
The property INDEXEDFIELDS is to configure which fields to index and to project a property to another. Syntax:
```
 # only index label and comment
 index.1=http://www.w3.org/2000/01/rdf-schema#label
 index.2=http://www.w3.org/2000/01/rdf-schema#comment
 # project http://xmlns.com/foaf/0.1/name to rdfs:label
 http\://xmlns.com/foaf/0.1/name=http\://www.w3.org/2000/01/rdf-schema#label
 
```
Datatypes
Datatypes are ignored in the LuceneSail.

Field Summary

Fields
Modifier and Type	Field and Description
`static String`	`ANALYZER_CLASS_KEY` Set this key as sail parameter to configure the Lucene analyzer class implementation to use for text analysis.
`static String`	`DEFAULT_INDEX_CLASS`
`static String`	`DEFAULT_LUCENE_DIR` Set the default directory of the Lucene index files.
`static String`	`EVALUATION_MODE_KEY` See `TupleFunctionEvaluationMode`.
`static String`	`INCOMPLETE_QUERY_FAIL_KEY` Set this key as sail parameter to influence whether incomplete queries are treated as failure (Malformed queries) or whether they are ignored.
`static String`	`INDEX_CLASS_KEY` Set this key to configure the SearchIndex class implementation.
`static String`	`INDEXEDFIELDS` Set the parameter "indexedfields=..." to configure a selection of fields to index, and projections of properties.
`static String`	`LUCENE_DIR_KEY` Set the key "lucenedir=<path>" as sail parameter to configure the Lucene Directory on the filesystem where to store the lucene index.
`static String`	`LUCENE_RAMDIR_KEY` Set the key "useramdir=true" as sail parameter to let the LuceneSail store its Lucene index in RAM.
`static String`	`MAX_DOCUMENTS_KEY` Set the key "maxDocuments=<n>" as sail parameter to limit the maximum number of documents to return from a search query.
`protected Properties`	`parameters`
`static String`	`REINDEX_QUERY_KEY` Set the parameter "reindexQuery=" to configure the statements to index over.
`static String`	`SIMILARITY_CLASS_KEY` Set this key as sail parameter to configure `Similarity` class implementation to use for text analysis.
`static String`	`WKT_FIELDS` Set this key to configure which fields contain WKT and should be spatially indexed.

Constructor Summary

Constructors
Constructor and Description

LuceneSail()

Constructors
Constructor and Description
`LuceneSail()`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`protected boolean`	`acceptStatementToIndex(Statement s)`
`protected static SearchIndex`	`createSearchIndex(Properties parameters)` Deprecated.
`NotifyingSailConnection`	`getConnection()` Opens a connection on the Sail which can be used to query and update data.
`TupleFunctionEvaluationMode`	`getEvaluationMode()` See EVALUATION_MODE_KEY parameter.
`FederatedServiceResolver`	`getFederatedServiceResolver()`
`SearchIndex`	`getLuceneIndex()`
`String`	`getParameter(String key)`
`Set<String>`	`getParameterNames()`
`String`	`getReindexQuery()` See REINDEX_QUERY_KEY parameter.
`protected Collection<SearchQueryInterpreter>`	`getSearchQueryInterpreters()`
`TupleFunctionRegistry`	`getTupleFunctionRegistry()`
`void`	`initialize()` Initializes the Sail.
`protected void`	`initializeLuceneIndex()`
`boolean`	`isIncompleteQueryFails()` When this is true, incomplete queries will trigger a SailException.
`Statement`	`mapStatement(Statement statement)`
`void`	`registerStatementFilter(IndexableStatementFilter filter)` Sets a filter which determines whether a statement should be considered for indexing when performing complete reindexing.
`void`	`reindex()` Starts a reindexation process of the whole sail.
`void`	`setDataDir(File dataDir)` Sets the data directory for the Sail.
`void`	`setEvaluationMode(TupleFunctionEvaluationMode mode)` See EVALUATION_MODE_KEY parameter.
`void`	`setFederatedServiceResolver(FederatedServiceResolver resolver)` Sets the `FederatedServiceResolver` to use for this client.
`void`	`setIncompleteQueryFails(boolean incompleteQueryFails)` Set this to true, so that incomplete queries will trigger a SailException.
`void`	`setLuceneIndex(SearchIndex luceneIndex)`
`void`	`setParameter(String key, String value)`
`void`	`setReindexQuery(String query)` See REINDEX_QUERY_KEY parameter.
`void`	`setTupleFunctionRegistry(TupleFunctionRegistry registry)`
`void`	`shutDown()` Shuts down the Sail, giving it the opportunity to synchronize any stale data.

Methods inherited from class org.eclipse.rdf4j.sail.helpers.NotifyingSailWrapper
addSailChangedListener, getBaseSail, removeSailChangedListener, setBaseSail

Methods inherited from class org.eclipse.rdf4j.sail.helpers.SailWrapper
getDataDir, getDefaultIsolationLevel, getSupportedIsolationLevels, getValueFactory, isWritable, verifyBaseSailSet

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.eclipse.rdf4j.sail.Sail
getDataDir, getDefaultIsolationLevel, getSupportedIsolationLevels, getValueFactory, init, isWritable

- Field Detail
  - REINDEX_QUERY_KEY
```
public static final String REINDEX_QUERY_KEY
```
    Set the parameter "reindexQuery=" to configure the statements to index over. Default value is "SELECT ?s ?p ?o ?c WHERE {{?s ?p ?o} UNION {GRAPH ?c {?s ?p ?o.}}} ORDER BY ?s" . NB: the query must contain the bindings ?s, ?p, ?o and ?c and must be ordered by ?s.
    
    See Also:
    
    Constant Field Values
  - INDEXEDFIELDS
```
public static final String INDEXEDFIELDS
```
    Set the parameter "indexedfields=..." to configure a selection of fields to index, and projections of properties. Only the configured fields will be indexed. A property P projected to Q will cause the index to contain Q instead of P, when triples with P were indexed. Syntax of indexedfields - see above
    
    See Also:
    
    Constant Field Values
  - LUCENE_DIR_KEY
```
public static final String LUCENE_DIR_KEY
```
    Set the key "lucenedir=<path>" as sail parameter to configure the Lucene Directory on the filesystem where to store the lucene index.
    
    See Also:
    
    Constant Field Values
  - DEFAULT_LUCENE_DIR
```
public static final String DEFAULT_LUCENE_DIR
```
    Set the default directory of the Lucene index files. The value is always relational to the dataDir location as a parent directory.
    
    See Also:
    
    Constant Field Values
  - LUCENE_RAMDIR_KEY
```
public static final String LUCENE_RAMDIR_KEY
```
    Set the key "useramdir=true" as sail parameter to let the LuceneSail store its Lucene index in RAM. This is not intended for production environments.
    
    See Also:
    
    Constant Field Values
  - MAX_DOCUMENTS_KEY
```
public static final String MAX_DOCUMENTS_KEY
```
    Set the key "maxDocuments=<n>" as sail parameter to limit the maximum number of documents to return from a search query. The default is to return all documents. NB: this may involve extra cost for some SearchIndex implementations as they may have to determine this number.
    
    See Also:
    
    Constant Field Values
  - WKT_FIELDS
```
public static final String WKT_FIELDS
```
    Set this key to configure which fields contain WKT and should be spatially indexed. The value should be a space-separated list of URIs. Default is http://www.opengis.net/ont/geosparql#asWKT.
    
    See Also:
    
    Constant Field Values
  - INDEX_CLASS_KEY
```
public static final String INDEX_CLASS_KEY
```
    Set this key to configure the SearchIndex class implementation. Default is org.eclipse.rdf4j.sail.lucene.LuceneIndex.
    
    See Also:
    
    Constant Field Values
  - DEFAULT_INDEX_CLASS
```
public static final String DEFAULT_INDEX_CLASS
```
    See Also:
    
    Constant Field Values
  - ANALYZER_CLASS_KEY
```
public static final String ANALYZER_CLASS_KEY
```
    Set this key as sail parameter to configure the Lucene analyzer class implementation to use for text analysis.
    
    See Also:
    
    Constant Field Values
  - SIMILARITY_CLASS_KEY
```
public static final String SIMILARITY_CLASS_KEY
```
    Set this key as sail parameter to configure Similarity class implementation to use for text analysis.
    
    See Also:
    
    Constant Field Values
  - INCOMPLETE_QUERY_FAIL_KEY
```
public static final String INCOMPLETE_QUERY_FAIL_KEY
```
    Set this key as sail parameter to influence whether incomplete queries are treated as failure (Malformed queries) or whether they are ignored. Set to either "true" or "false". When ommitted in the properties, true is default (failure on incomplete queries). see isIncompleteQueryFails()
    
    See Also:
    
    Constant Field Values
  - EVALUATION_MODE_KEY
```
public static final String EVALUATION_MODE_KEY
```
    See TupleFunctionEvaluationMode.
    
    See Also:
    
    Constant Field Values
  - parameters
```
protected final Properties parameters
```
- Constructor Detail
  - LuceneSail
```
public LuceneSail()
```
- Method Detail
  - setLuceneIndex
```
public void setLuceneIndex(SearchIndex luceneIndex)
```
  - getLuceneIndex
```
public SearchIndex getLuceneIndex()
```
  - getConnection
```
public NotifyingSailConnection getConnection()
                                      throws SailException
```
    Description copied from interface: Sail
    
    Opens a connection on the Sail which can be used to query and update data. Depending on how the implementation handles concurrent access, a call to this method might block when there is another open connection on this Sail.
    
    Specified by:
    
    getConnection in interface NotifyingSail
    
    Specified by:
    
    getConnection in interface Sail
    
    Overrides:
    
    getConnection in class NotifyingSailWrapper
    
    Throws:
    
    SailException - If no transaction could be started, for example because the Sail is not writable.
  - shutDown
```
public void shutDown()
              throws SailException
```
    Description copied from interface: Sail
    
    Shuts down the Sail, giving it the opportunity to synchronize any stale data. Care should be taken that all initialized Sails are being shut down before an application exits to avoid potential loss of data. Once shut down, a Sail can no longer be used until it is re-initialized.
    
    Specified by:
    
    shutDown in interface Sail
    
    Overrides:
    
    shutDown in class SailWrapper
    
    Throws:
    
    SailException - If the Sail object encountered an error or unexpected situation internally.
  - setDataDir
```
public void setDataDir(File dataDir)
```
    Description copied from interface: Sail
    
    Sets the data directory for the Sail. The Sail can use this directory for storage of data, parameters, etc. This directory must be set before the Sail is initialized.
    
    Specified by:
    
    setDataDir in interface Sail
    
    Overrides:
    
    setDataDir in class SailWrapper
  - initialize
```
public void initialize()
                throws SailException
```
    Description copied from interface: Sail
    
    Initializes the Sail. Care should be taken that required initialization parameters have been set before this method is called. Please consult the specific Sail implementation for information about the relevant parameters.
    
    Specified by:
    
    initialize in interface Sail
    
    Overrides:
    
    initialize in class SailWrapper
    
    Throws:
    
    SailException - If the Sail could not be initialized.
  - createSearchIndex
```
@Deprecated
protected static SearchIndex createSearchIndex(Properties parameters)
                                                    throws Exception
```
    Deprecated.
    
    The method is relocated to SearchIndexUtils.createSearchIndex(java.util.Properties).
    
    Parameters:
    
    parameters -
    
    Returns:
    
    Throws:
    
    Exception
  - initializeLuceneIndex
```
protected void initializeLuceneIndex()
                              throws Exception
```
    Throws:
    
    Exception
  - setParameter
```
public void setParameter(String key,
                         String value)
```
  - getParameter
```
public String getParameter(String key)
```
  - getParameterNames
```
public Set<String> getParameterNames()
```
  - getReindexQuery
```
public String getReindexQuery()
```
    See REINDEX_QUERY_KEY parameter.
  - setReindexQuery
```
public void setReindexQuery(String query)
```
    See REINDEX_QUERY_KEY parameter.
  - isIncompleteQueryFails
```
public boolean isIncompleteQueryFails()
```
    When this is true, incomplete queries will trigger a SailException. You can set this value either using setIncompleteQueryFails(boolean) or using the parameter "incompletequeryfail"
    
    Returns:
    
    Returns the incompleteQueryFails.
  - setIncompleteQueryFails
```
public void setIncompleteQueryFails(boolean incompleteQueryFails)
```
    Set this to true, so that incomplete queries will trigger a SailException. Otherwise, incomplete queries will be logged with level WARN. Default is true. You can set this value also using the parameter "incompletequeryfail".
    
    Parameters:
    
    incompleteQueryFails - true or false
  - getEvaluationMode
```
public TupleFunctionEvaluationMode getEvaluationMode()
```
    See EVALUATION_MODE_KEY parameter.
  - setEvaluationMode
```
public void setEvaluationMode(TupleFunctionEvaluationMode mode)
```
    See EVALUATION_MODE_KEY parameter.
  - getTupleFunctionRegistry
```
public TupleFunctionRegistry getTupleFunctionRegistry()
```
  - setTupleFunctionRegistry
```
public void setTupleFunctionRegistry(TupleFunctionRegistry registry)
```
  - getFederatedServiceResolver
```
public FederatedServiceResolver getFederatedServiceResolver()
```
  - setFederatedServiceResolver
```
public void setFederatedServiceResolver(FederatedServiceResolver resolver)
```
    Description copied from interface: FederatedServiceResolverClient
    
    Sets the FederatedServiceResolver to use for this client.
    
    Specified by:
    
    setFederatedServiceResolver in interface FederatedServiceResolverClient
    
    Overrides:
    
    setFederatedServiceResolver in class SailWrapper
    
    Parameters:
    
    resolver - The resolver to use.
  - reindex
```
public void reindex()
             throws Exception
```
    Starts a reindexation process of the whole sail. Basically, this will delete and add all data again, a long-lasting process.
    
    Throws:
    
    IOException
    
    Exception
  - registerStatementFilter
```
public void registerStatementFilter(IndexableStatementFilter filter)
```
    Sets a filter which determines whether a statement should be considered for indexing when performing complete reindexing.
  - acceptStatementToIndex
```
protected boolean acceptStatementToIndex(Statement s)
```
  - mapStatement
```
public Statement mapStatement(Statement statement)
```
  - getSearchQueryInterpreters
```
protected Collection<SearchQueryInterpreter> getSearchQueryInterpreters()
```

Class LuceneSail

Setting up a LuceneSail

Asking full-text queries

Fields are stored/indexed

Deleting a Lucene index

Handling of Contexts

Defining the indexed Fields

Datatypes

Field Summary

Constructor Summary

Method Summary

Methods inherited from class org.eclipse.rdf4j.sail.helpers.NotifyingSailWrapper

Methods inherited from class org.eclipse.rdf4j.sail.helpers.SailWrapper

Methods inherited from class java.lang.Object

Methods inherited from interface org.eclipse.rdf4j.sail.Sail

Field Detail

REINDEX_QUERY_KEY

INDEXEDFIELDS

LUCENE_DIR_KEY

DEFAULT_LUCENE_DIR

LUCENE_RAMDIR_KEY

MAX_DOCUMENTS_KEY

WKT_FIELDS

INDEX_CLASS_KEY

DEFAULT_INDEX_CLASS

ANALYZER_CLASS_KEY

SIMILARITY_CLASS_KEY

INCOMPLETE_QUERY_FAIL_KEY

EVALUATION_MODE_KEY

parameters

Constructor Detail

LuceneSail

Method Detail

setLuceneIndex

getLuceneIndex

getConnection

shutDown

setDataDir

initialize

createSearchIndex

initializeLuceneIndex

setParameter

getParameter

getParameterNames

getReindexQuery

setReindexQuery

isIncompleteQueryFails

setIncompleteQueryFails

getEvaluationMode

setEvaluationMode

getTupleFunctionRegistry

setTupleFunctionRegistry

getFederatedServiceResolver

setFederatedServiceResolver

reindex

registerStatementFilter

acceptStatementToIndex

mapStatement

getSearchQueryInterpreters