Po-Chuan Hsieh 6c34be9ab1
textproc/py-tiktoken: Add py-tiktoken 0.4.0
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
2023-08-22 01:01:58 +08:00

2 lines
63 B
Plaintext

tiktoken is a fast BPE tokeniser for use with OpenAI's models.