tiktoken
A fast BPE tokeniser for use with OpenAI's models
homepage ↗homepage ↗ pypi: tiktoken github: openai/tiktoken
Available in
| Overlay | Newest | Ebuilds | Last activity | |
|---|---|---|---|---|
| tatsh-overlay GitHub ↗ | 0.12.0 | 1 | 13 h | details › |
| guru gitweb ↗ | 0.13.0 | 1 | 47 h | details › |
| stuff GitHub ↗ | 0.13.0 | 2 | 2 d | details › |
Versions & arches
| Version | Overlay | amd64 | arm64 | Committed | |
|---|---|---|---|---|---|
| 0.13.0 | guru | amd64 testing | arm64 testing | — | view · download · history ↗ |
| 0.13.0 ≈ | stuff | amd64 testing | arm64 testing | 2 mo | view · download · history ↗ |
| 0.12.0 ≈ | stuff | amd64 testing | arm64 testing | 2 mo | view · download · history ↗ |
| 0.12.0 ≈ | tatsh-overlay | amd64 testing | arm64 untested | 2 mo | view · download · history ↗ |
Use flags of 0.13.0
- test Enable dependencies and/or preparations necessary to run tests (usually controlled by FEATURES=test but can be toggled independently)
- debug Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces
4 expansion flags (python targets, ABIs, cpu flags…)
- python_targets_python3_12
- python_targets_python3_13
- python_targets_python3_14
- python_targets_python3_15
Runtime dependencies of 0.13.0
show 14 lines
python_targets_python3_12?
(
)
python_targets_python3_13?
(
)
python_targets_python3_14?
(
)
python_targets_python3_15?
(
)