Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.paradedb.com/llms.txt

Use this file to discover all available pages before exploring further.

The edge ngram tokenizer first splits text into words at character-class boundaries, then generates n-grams anchored to the beginning of each word. This makes it ideal for “search-as-you-type” functionality, where users find matches as they type partial words. The tokenizer takes two required arguments: the minimum and maximum gram length. For each word, it emits prefix tokens from min_gram to max_gram characters long (clamped to the word length). Words shorter than min_gram are skipped. 
CREATE INDEX search_idx ON mock_items
USING bm25 (id, (description::pdb.edge_ngram(2,5)))
WITH (key_field='id');
To get a feel for this tokenizer, run the following command and replace the text with your own: 
SELECT 'Quick Fox'::pdb.edge_ngram(2,5)::text[];
Expected Response
            text
-----------------------------
 {qu,qui,quic,quick,fo,fox}
(1 row)

​
Token Chars

By default, the edge ngram tokenizer treats letters and digits as token content and everything else (spaces, punctuation, symbols) as word delimiters. You can customize this with token_chars, which accepts a comma-separated list of character classes: letter, digit, whitespace, punctuation, symbol. Character classification uses Unicode general categories, matching Elasticsearch’s behavior. For example, including punctuation keeps hyphens as part of words: 
SELECT 'Quick-Fox'::pdb.edge_ngram(2,5,'token_chars=letter,digit,punctuation')::text[];
Expected Response
          text
-------------------------
 {qu,qui,quic,quick}
(1 row)