Tokenization
Text tokenization using Byte Pair Encoding (BPE). Supports Indian language.
Try with samples:
Sample 1
Sample 2
Sample 3
or
Upload Text File
Original Text:
Encode Text
Tokens:
Decode Tokens
Decoded Text:
Reset