Hiroki Tagato 9f75adf237 textproc/py-tokenizers: update to 0.21.2 ...

While here, refactor test-related part so that it can execute both
python and rust tests.

Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.21.2

Reported by:	Repology

2025-06-26 18:16:07 +09:00

..

distinfo

textproc/py-tokenizers: update to 0.21.2

2025-06-26 18:16:07 +09:00

Makefile

textproc/py-tokenizers: update to 0.21.2

2025-06-26 18:16:07 +09:00

Makefile.crates

textproc/py-tokenizers: update to 0.21.2

2025-06-26 18:16:07 +09:00

pkg-descr

…