Tokenizer

See how text splits into tokens across popular model tokenizers.

Input

Choose a tokenizer, paste text, and inspect the resulting token pieces or IDs.

Tokenizer

Hugging Face repo URL

Switch between token text and token IDs.

Decode token text

Tokens will appear here.