Hiroki Tagato 1f65a995d5 textproc/py-tokenizers: update to 0.20.1 ...

While here, add LICENSE_FILE.

Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.20.1

Reported by:	Repology

2024-10-18 14:12:30 +09:00

..

distinfo

textproc/py-tokenizers: update to 0.20.1

2024-10-18 14:12:30 +09:00

Makefile

textproc/py-tokenizers: update to 0.20.1

2024-10-18 14:12:30 +09:00

Makefile.crates

textproc/py-tokenizers: update to 0.20.1

2024-10-18 14:12:30 +09:00

pkg-descr

…