↓ Skip to main content
  1. Projects/

splintr-go

Bringing Rust-based tokenisation into Go services without introducing a separate inference service.
Go
Rust
cgo
Tokenisation

I built splintr-go, Go bindings for the Rust splintr tokeniser through cgo. It exposes native tokenisation through a Go API, with explicit control over native-resource lifetime.

GitHub repository

Capabilities
#

  • Encode and decode: convert text to token IDs and back, with bundled pretrained vocabularies or a supplied Hugging Face tokenizer.json.
  • Parallel batches: encode and decode batches using native parallelism for larger inputs while preserving input order.
  • Streaming decoding: incrementally decode generated token IDs, buffering split UTF-8 characters until they can be emitted correctly.
  • Concurrency and cleanup: support concurrent tokeniser calls and explicit Close methods for releasing native resources.
  • Distribution: provide platform-specific native release archives and documented build and linking instructions.

Integration notes
#

The bindings use cgo and require a native library and C toolchain. Bundled vocabularies can be loaded without runtime downloads; the upstream documentation lists their compatibility and separate licences. Streaming support is for decoding, not incremental encoding.