> ## Documentation Index
> Fetch the complete documentation index at: https://docs.paradedb.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Multiple Tokenizers Per Field

> Apply different token configurations to the same field

In many cases, a text field needs to be tokenized multiple ways. For instance, using the [unicode](/documentation/tokenizers/available-tokenizers/unicode)
tokenizer for search, and the [literal](/documentation/tokenizers/available-tokenizers/literal) tokenizer for [Top K ordering](/documentation/sorting/topk).

To tokenize a field in more than one way, append an `alias=<alias_name>` argument to the additional tokenizer configurations.
The alias name can be any string you like. For instance, the following statement tokenizes `description` using both the simple and literal tokenizers.

```sql theme={null}
CREATE INDEX search_idx ON mock_items
USING bm25 (
  id,
  (description::pdb.literal),
  (description::pdb.simple('alias=description_simple'))
) WITH (key_field='id');
```

Under the hood, two distinct fields are created in the index: a field called `description`, which uses the literal tokenizer,
and an aliased field called `description_simple`, which uses the simple tokenizer.

To query against the aliased field, cast it to `pdb.alias('alias_name')`:

<CodeGroup>
  ```sql SQL theme={null}
  -- Query against `description_simple`
  SELECT description, rating, category
  FROM mock_items
  WHERE description::pdb.alias('description_simple') ||| 'Sleek running shoes';

  -- Query against `description`
  SELECT description, rating, category
  FROM mock_items
  WHERE description ||| 'Sleek running shoes';
  ```

  ```ts Drizzle theme={null}
  import { search } from "@paradedb/drizzle-paradedb";

  // Query against `description_simple`
  await db
    .select({
      description: mockItems.description,
      rating: mockItems.rating,
      category: mockItems.category,
    })
    .from(mockItems)
    .where(
      search.matchAny(
        search.alias(mockItems.description, "description_simple"),
        "Sleek running shoes",
      ),
    );

  // Query against `description`
  await db
    .select({
      description: mockItems.description,
      rating: mockItems.rating,
      category: mockItems.category,
    })
    .from(mockItems)
    .where(search.matchAny(mockItems.description, "Sleek running shoes"));
  ```

  ```python Django theme={null}
  from paradedb import MatchAny, ParadeDB

  # Query against `description_simple`
  MockItem.objects.extra(
      where=["(description::pdb.alias('description_simple')) ||| 'Sleek running shoes'"]
  ).values('description', 'rating', 'category')

  # Query against `description`
  MockItem.objects.filter(
      description=ParadeDB(MatchAny('Sleek running shoes'))
  ).values('description', 'rating', 'category')
  ```

  ```python SQLAlchemy theme={null}
  from sqlalchemy import select
  from sqlalchemy.orm import Session
  from paradedb.sqlalchemy import pdb, search

  # Query against `description_simple`
  stmt_alias = (
      select(MockItem.description, MockItem.rating, MockItem.category)
      .where(search.match_any(pdb.alias(MockItem.description, "description_simple"), "Sleek running shoes"))
  )

  # Query against `description`
  stmt = (
      select(MockItem.description, MockItem.rating, MockItem.category)
      .where(search.match_any(MockItem.description, "Sleek running shoes"))
  )

  with Session(engine) as session:
      {
          "rows_alias": session.execute(stmt_alias).all(),
          "rows": session.execute(stmt).all(),
      }
  ```

  ```ruby Rails theme={null}
  # Query against `description_simple`
  MockItem.search(:description_simple)
          .matching_any("Sleek running shoes")
          .select(:description, :rating, :category)

  # Query against `description`
  MockItem.search(:description)
          .matching_any("Sleek running shoes")
          .select(:description, :rating, :category)
  ```

  ```cs EF Core theme={null}
  // Query against `description_simple`
  await dbContext
      .MockItems.Where(item =>
          EF.Functions.MatchAny(
              EF.Functions.Alias(item.Description, "description_simple"),
              "Sleek running shoes"
          )
      )
      .Select(item => new { item.Description, item.Rating, item.Category })
      .ToListAsync();

  // Query against `description`
  await dbContext
      .MockItems.Where(item =>
          EF.Functions.MatchAny(item.Description, "Sleek running shoes")
      )
      .Select(item => new { item.Description, item.Rating, item.Category })
      .ToListAsync();
  ```
</CodeGroup>

<Note>
  If a text field uses multiple tokenizers and one of them is [literal](/documentation/tokenizers/available-tokenizers/literal), we recommend aliasing
  the other tokenizers and leaving the literal tokenizer un-aliased. This is so queries that `GROUP BY`, `ORDER BY`, or aggregate the
  text field can reference the field directly:

  ```sql SQL theme={null}
  CREATE INDEX search_idx ON mock_items
  USING bm25 (
    id,
    (description::pdb.literal),
    (description::pdb.simple('alias=description_simple'))
  ) WITH (key_field='id');

  SELECT description, rating, category
  FROM mock_items
  WHERE description @@@ 'shoes'
  ORDER BY description
  LIMIT 5;
  ```
</Note>