arabic-token-cost
Skill43 starsChanged 29 days agoMITView on GitHub
Use when estimating or comparing how many tokens Arabic text costs across LLM tokenizers, or when Arabic prompts seem expensive, truncated or slow compared with English.
- Installs packages
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 2 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 16
Use --file path for longer samples (at least a few hundred words gives stable numbers).
- Distinctive: line 31
Run the script on a representative sample of the user's real text, not a single sentence.
- Very specific: line 33
If cost matters, call the MCP tool search(query="tokenizer") and recommend Arabic-aware tokenizers or models the atlas returns, with …
- Distinctive: line 38
Do not recommend models that the atlas did not return.
≈ “Common” is an estimate: at least 1 in 5 of the 308,232 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Arabic Token Cost
Measures tokenizer fertility (tokens per word) on Arabic text against an English reference of the same word count.
Run
python ${CLAUDE_PLUGIN_ROOT}/skills/arabic-token-cost/token_cost.py --text "النص العربي هنا" [--tokenizers id,id]Use --file path for longer samples (at least a few hundred words gives stable numbers). Default tokenizers: gpt2, meta-llama/Llama-3.1-8B, Qwen/Qwen2.5-7B, inceptionai/jais-family-590m, google/gemma-2-2b.
It needs transformers. If missing it prints an install hint (uv pip install transformers) and exits 2. Tokenizers download from Hugging Face on first use; gated ones (such as Llama) may fail, and that row prints an error while the rest continue.
Reading the table
Columns: tokenizer | tokens | tokens/word | ratio vs English.
- Under 1.5 tokens/word: efficient.
- 1.5 to 2.5: acceptable.
- Over 2.5: expensive. Context fills about twice as fast and cost and latency rise accordingly.
- Ratio vs English shows the Arabic penalty for that tokenizer; compare it across rows, not in absolute terms.
Procedure
- Run the script on a representative sample of the user's real text, not a single sentence.
- Report the table and flag every tokenizer over 2.5 tokens/word.
- If cost matters, call the MCP tool
search(query="tokenizer")and recommend Arabic-aware tokenizers or models the atlas returns, with license and link.
Rules
- Fertility depends on the text (dialect and diacritics raise it). State the sample used.
- Do not recommend models that the atlas did not return.
In this repository6 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- dist/llms.txtllms.txt
- arabic-ai-advisorSkill
- arabic-dialect-promptsSkill
- rtl-bidi-lintSkill
- tashkeel-checkSkill
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.