Bringing Rust-based tokenisation into Go services without introducing a separate inference service.
Go
Rust
cgo
Tokenisation
I built splintr-go, Go bindings for the Rust splintr tokeniser through cgo. It exposes native tokenisation through a Go API, with explicit control over native-resource lifetime.
GitHub repositoryCapabilities#
- Encode and decode: convert text to token IDs and back, with bundled pretrained vocabularies or a supplied Hugging Face
tokenizer.json. - Parallel batches: encode and decode batches using native parallelism for larger inputs while preserving input order.
- Streaming decoding: incrementally decode generated token IDs, buffering split UTF-8 characters until they can be emitted correctly.
- Concurrency and cleanup: support concurrent tokeniser calls and explicit
Closemethods for releasing native resources. - Distribution: provide platform-specific native release archives and documented build and linking instructions.
Integration notes#
The bindings use cgo and require a native library and C toolchain. Bundled vocabularies can be loaded without runtime downloads; the upstream documentation lists their compatibility and separate licences. Streaming support is for decoding, not incremental encoding.