`rau.vocab`¶

This module provides tools for mapping token types to integer IDs.

class rau.vocab.Vocabulary¶

Bases: object

An abstract base class that represents a mapping between token types and integer IDs.

class rau.vocab.VocabularyBuilder¶

Bases: Generic[V]

An abstract base class that can be used for constructing Vocabulary objects of a certain type.

catchall(token)¶

Build a vocabulary that maps all token types to a single token type. This implements the behavior of an UNK token.

Parameters:: token (str) – A token string used to represent the catchall token.
Return type:: TypeVar(V)
Returns:: A vocabulary that maps all token strings to the specified token.

content(tokens)¶

Build a vocabulary that assigns consecutive integer IDs to a list of token strings. These are “content” tokens in the sense that they come from a corpus and are not special tokens.

Parameters:: tokens (list[str]) – A list of token type strings.
Return type:: TypeVar(V)
Returns:: A vocabulary containing the specified tokens.

reserved(tokens)¶

Build a vocabulary that assigns consecutive integer IDs to a list of special reserved tokens. The token strings of these special tokens are for display purposes only and will never conflict with content tokens.