huggingface-tokenizers
What this skill does
Fast tokenization for NLP tasks, with support for BPE, WordPiece, and Unigram algorithms. Designed for research and production use, with a Rust-based implementation for performance.
github/orchestra-research - general - 10.8k stars
Threat analysis
Skill info
pkg:github/Orchestra-Research/AI-Research-SKILLs@773a529?skill=huggingface-tokenizersAssessments
No risk patterns detected in this scan. A clean automated scan is a good signal, not a guarantee.
Badge
Add the Anomity scan badge for huggingface-tokenizers to your README.
How Anomity governs this at runtime
Scan-time vetting tells you what a skill says it will do. Anomity's Endpoint Sensor sees what agents actually do: it discovers skills alongside every other AI artifact on the endpoint, and runtime governance can allow, deny, or log the tool calls a skill triggers. Policy violations route to your SIEM, Slack, email, or Jira, backed by a queryable 90-day audit trail.
Book a 30-minute demo to see your own skill inventory.
Methodology and disputes
Every skill is assessed by the Anomity Skill Intelligence engine against its public source; findings indicate risk patterns, not confirmed exploitation. Maintainer of huggingface-tokenizers? Report an issue or request a rescan.
