Measurement protocol

The figures on this site come from public datasets, not customer data. This page explains how they were obtained: results and method. The evaluation code is not published.

Datasets and licences

For the yes / no question, “yes” is MASSIVE's calendar scenario (creating, checking or removing an appointment); “no” covers the 17 other scenarios (alarm, weather, music, transport…), spread evenly in the test set.

A stance dataset (X-stance) was also evaluated, then discarded: its licence forbids commercial use.

Compared models

• Specialised model: mmBERT-base (multilingual encoder, MIT licence) fine-tuned for each task with a classification layer. Hyperparameters fixed before any measurement, with no search on the test set; last checkpoint kept. For serving, the model is exported to the open ONNX format and quantised to int8: 294 MB on disk, no GPU.
• Classic method: TF-IDF (words and character groups) followed by logistic regression, tuned on validation.
• API reference: a general-purpose classification service available through an API, queried with the same sentences and the list of categories. For choice tasks, the result is the average over two presentation orders of the categories.
• A second neural model (multilingual Laya, Apache 2.0) was also evaluated. It was not kept: its ONNX export failed and it is 7 to 45 times slower on CPU depending on the task (measured on Modal, decisionfactory report), for comparable accuracy.

Metrics

• Accuracy: share of test sentences classified correctly.
• Mean per-class accuracy: average of the accuracy of each category, so that frequent categories are not favoured.
• Calibration error (ECE): average gap between the stated confidence and the actual rate of correct answers (15 bins).

Non-inferiority test

The question is not “is the small model better?” but “is it at least as good as the API reference, within a margin?”.

• Paired bootstrap: 10,000 resamples of the test sentences with replacement, comparing both models on the same sentences.
• The model is declared non-inferior if the lower bound of the 95 % confidence interval of the difference is above −2 points, for both accuracy and mean per-class accuracy.
• Result (decisionfactory report, fp32 model trained on the full training set): non-inferior on Banking77, on the yes / no question in all three languages, and on MASSIVE in all four languages with the full test set. On 500-sentence samples the result was inconclusive for German, which is why MASSIVE is measured on the full test set.

Calibration

The probabilities shown in the playground are calibrated: a single temperature, fitted on validation, softens the model's scores so that “0.80” means roughly eight correct answers out of ten.

The same temperatures are applied to the served int8 model; its calibration has not been re-measured separately.

Served model and evaluation report

The decisionfactory evaluation report publishes the figures of the fp32 model (for MASSIVE, retrained for the full-test evaluation). This site shows those of the int8 model actually served, which we re-measured on the same test sets. The differences are one point at most:

The “classic method” and “API reference” columns come from the decisionfactory report (October 2026).

Latency and cost measurement

• The latency of the specialised model is measured on the server hosting this site (4 vCPU, no GPU), inside the service container, limited to 2 CPUs and 2.5 GB of memory. No latency of this model measured on another machine is shown.
• One decision = one sentence, processed alone (no batching), one request at a time. Fixed public sentences of 4 to 20 words, plus a 500-character text; warm-up runs excluded; median (p50) and 95th percentile (p95). Tokenisation is included in the measured time.
• Results on this server (measured on October 17, 2026): median latency of about 35 ms per short sentence (p95: 50 to 68 ms depending on the task), 177 to 202 ms for a 500-character text, and about 27 decisions per second single-stream (one request at a time, the deployed setting).
• The classic method is measured the same way, on the same server, in a container with the same limits, with a single thread. On this server, the classic method answers in ~12 ms for banking intents and ~2 ms for the yes / no question, but in ~56 ms on MASSIVE, i.e. more slowly than the neural model (~35 ms): it is also its largest model (97 MB, character groups in four languages).
• The latency shown for these two models excludes the network between the visitor and the server. In the playground, “latency of this decision” is the compute time measured by the server for your request.
• The latency of the API reference (about 220 ms) was measured client-side, network included, in October 2026. It therefore includes the network round trip, which the other two columns do not.
• Cost per million decisions of the specialised model: full monthly server price, in US dollars, divided by the number of decisions processed in a month at full load, using the single-stream throughput (one request at a time, the deployed setting). Price used: 6.00 USD per month, i.e. the full price of the server, attributed entirely to the playground although the server is shared with other services. This is a conservative estimate: the actual cost is lower. Measured throughput: 26.8 decisions per second.
• Comparison costs, in US dollars: classic method on a 2-core cloud instance, API reference from its per-token price and the number of tokens measured per request; public prices as of 15 October 2026.

Effect of the hardware

The same int8 model, measured in two environments. No cost column: the price bases are not comparable.

On the Modal containers, the model answers about four times faster. The gap does not come from the processor alone: those containers had 4 vCPU and 4 threads, against 2 CPUs and 2 threads here, and tokenisation was not timed there. It is therefore an order of magnitude of what a more powerful machine brings, not an isolated measure of the processor's effect.

Limitations

• Public data: your results will depend on your data. They will be measured on a sample of it before any decision.
• Banking77 only exists in English. The yes / no question is tested on a balanced sample (50 % “yes”), which is easier than most real situations.
• With 500 test sentences, the uncertainty on an accuracy is about ±2 points (95 % interval).
• The API reference was measured in October 2026; its results and prices may change.
• The cost assumes a server at 100 % load and includes neither integration nor maintenance.
• A specialised model does not decide alone: we recommend a confidence threshold and human review of doubtful cases.