↓ Skip to main content
  1. Projects/

GLiNER model serving

Taking GLiNER extraction, classification and decision models from local inference to reusable APIs and Haystack pipelines.
Python
vLLM
Haystack
Docker

I built vllm-gliner to serve GLiNER2 and GLiNER2.5 extraction and classification models, including the GLiNER2.5 Decide decision model, through vLLM. It adds DeBERTa model, pooling and request-processing plugins, with Docker images for NVIDIA GPUs and x86-64 CPUs.

GitHub repository

Model serving
#

  • Entity extraction over HTTP: expose schema-based extraction through /pooling, including character spans, confidence scores and token usage.
  • Efficient inference: integrate DeBERTa attention kernels, model pooling and IO processing with vLLM’s serving infrastructure.
  • CPU and GPU deployments: provide model-preparation tooling and Docker Compose configurations for both environments.

These serving capabilities are supplied by the project’s vLLM plugins.

Decision models
#

GLiNER2.5 Decide turns text into structured decisions rather than generated prose. I added an opt-in endpoint plugin for the OpenAI-compatible /v1/decisions API and vLLM’s /v1/systemone API, answering multiple classification questions in a single model pass.

The Decisions API supports choices, predicate probabilities and scores—for example, identifying a support request’s intent, whether it asks for cancellation, and its urgency together. Decide uses a closed-set classification head, not an autoregressive language-model vocabulary, and also remains accessible through /pooling.

Companion project: gliner-haystack
#

gliner-haystack connects the inference server to Haystack pipelines. Its text and document extractors accept labels at runtime, return entity spans and confidence scores, and enrich new documents without changing their existing text or metadata.

For example, a Haystack indexing pipeline can tag entities before it writes documents to a store:

sequenceDiagram
    accTitle: Entity extraction in a Haystack pipeline
    accDescr: The document extractor sends text and entity labels to a separate GLiNER server, then returns copies of the documents with entity metadata.
    participant P as Pipeline
    participant H as Document extractor
    participant V as gliner via vLLM
    P->>H: Documents + labels
    H->>V: POST /pooling
    V-->>H: Spans + scores
    H-->>P: Enriched documents

GLiNERDocumentExtractor enriches documents; GLiNERTextExtractor returns entities for a text input without creating a document. Both are clients of the inference server. Document processing can also run asynchronously with bounded concurrency while preserving input order.