tantivy-py/docs/reference.md

Reference

Setup

We'll use a test index for the examples that follow.

import os
from tantivy import SchemaBuilder, Index, Document
schema = (
    SchemaBuilder()
        .add_integer_field("doc_id", indexed=True, stored=True)
        .add_text_field("title", stored=True)
        .add_text_field("body")
        .build()
)
index = Index(schema=schema, path=None)
writer = index.writer(heap_size=15_000_000, num_threads=1)
doc = Document()
doc.add_integer("doc_id", 1)
doc.add_text("title", "The Old Man and the Sea")
doc.add_text(
    "body",
    (
        "He was an old man who fished alone in a skiff in"
        "the Gulf Stream and he had gone eighty-four days "
        "now without taking a fish."
    ),
)
writer.add_document(doc)

doc = Document()
doc.add_integer("doc_id", 2)
doc.add_text("title", "The Old Man and the Sea II")
doc.add_text("body", "He was an old man who sailed alone.")

writer.add_document(doc)
writer.commit()
index.reload()

Valid Query Formats

tantivy-py supports the query language used in tantivy. Below a few basic query formats are shown:

• AND and OR conjunctions.

searcher = index.searcher()
query = index.parse_query('(Old AND Man) OR Stream', ["title", "body"])
(best_score, best_doc_address) = searcher.search(query, 3).hits[0]
best_doc = searcher.doc(best_doc_address)

• +(includes) and -(excludes) operators.

query = index.parse_query('+Old +Man chef -fished', ["title", "body"])
(best_score, best_doc_address) = searcher.search(query, 3).hits[0]
best_doc = searcher.doc(best_doc_address)

Note: in a query like above, a word with no +/- acts like an OR.

• phrase search.

query = index.parse_query('"eighty-four days"', ["title", "body"])
(best_score, best_doc_address) = searcher.search(query, 3).hits[0]
best_doc = searcher.doc(best_doc_address)

• integer search

query = index.parse_query('1', ["doc_id"])
(best_score, best_doc_address) = searcher.search(query, 3).hits[0]
best_doc = searcher.doc(best_doc_address)

Note: for integer search, the integer field should be indexed.

For more possible query formats and possible query options, see Tantivy Query Parser Docs.

Escape quotes inside a query string

The tantivy docs for the query parser say that special characters like quotes can be escaped inside query values. However, it will also be necessary to surround the search query in additional quotes, as if a phrase query were being used.

The following will NOT work:

try:
    index.parse_query(r'sea\"', ["title", "body"])
except ValueError as e:
    assert str(e) == r'Syntax Error: sea\"'

However, the following will succeed:

# Works!
index.parse_query(r'"sea\""', ["title", "body"])

Note that whether the included (and escaped) quote actually gets used to match documents depends on the tokenizer used for the field. For example, the default tokenizer will not match the document "sea"s" with the query "sea"", because this tokenizer discards punctuation.