CoreShieldCoreShield AI

How we measure

How the figures on our homepage are produced, and what they do not tell you.

بيانات اصطناعية · Synthetic data

The benchmark

Each benchmark is a set of synthetic documents (Word, PDF, Excel and, in the newest set, CSV and JSON; mostly Arabic) containing planted sensitive values: names, organisations, phone numbers, national IDs, iqamas, commercial registrations, employee numbers, IBANs, emails and amounts. Every document has an answer file listing exactly which values it contains and where.

A held-out set

Each set was built by a separate process that never saw the detector's rules, the earlier benchmark sets or their results. It uses its own templates, phrasing, layouts, name lists, organisation lists and phone formats, so the rules could not be tuned to it.

Frozen before the fixes, scored once

Each set was frozen with SHA-256 checksums and committed before the round of detector changes it measures, then scored exactly once for publication after those changes were merged. After a set has been seen it is used only for regression checks, and its published figure is never replaced by a later re-score. Each table below reports the detector version it was measured on. The fingerprint of each set's checksum file is shown with its table.

Which detector is in service

The detector in service is ⁨svc-detector v1.34⁩. ⁨bench-files-v7⁩ is the only set measured on it, so the homepage shows its figures alone. The older sets (⁨bench-files-v6⁩, ⁨bench-files-v5⁩) were measured on earlier versions; their tables below are kept as history, each with its own detector version and date, and are not mixed into the homepage figures.

Organisations: rules and a trained model

Organisation names are detected by the rules together with a trained Arabic/English model; every other value type by the rules alone. The model was trained only on synthetic data and adds a name only where no rule has already found something. The organisation figure was measured with the person-name model also switched on in the same run; their spans overlap rarely. The newest test set, the model's training data and our adversarial tests were written by the same family of AI models on the same day, so the gain the model shows on that set very likely overstates its gain on real documents.

What the 95% confidence interval means

A benchmark is a sample. The interval (Wilson score method) is the range the true detection rate very likely falls in: 95.0% (95% CI 93.2–96.3) means the measured rate was 95.0% and the true rate is very likely between 93.2% and 96.3%. Smaller samples give wider intervals.

The measurements

The set in service first, then the older sets as history, each measured on its own detector version. "Detected" is the share of planted values found, with its 95% confidence interval; "Precision" is the share of flagged values that really were sensitive.

bench-files-v7Detector in service

⁨540⁩ documents · ⁨32,754⁩ planted values · ⁨svc-detector v1.34⁩ · measured ⁨16 November 2026⁩

Value typeDetectedPrecision
Structured identifiers, combined99.0%95% CI ⁨98.8⁩–⁨99.2⁩–
National ID100.0%95% CI ⁨99.8⁩–⁨100.0⁩98.5%
Iqama100.0%95% CI ⁨99.7⁩–⁨100.0⁩98.5%
Commercial registration93.1%95% CI ⁨91.5⁩–⁨94.4⁩99.9%
IBAN100.0%95% CI ⁨99.8⁩–⁨100.0⁩100.0%
Email100.0%95% CI ⁨99.8⁩–⁨100.0⁩100.0%
Employee numbers61.9%95% CI ⁨59.1⁩–⁨64.6⁩99.6%
Names95.3%95% CI ⁨94.8⁩–⁨95.7⁩94.8%
Organisations85.3%95% CI ⁨84.2⁩–⁨86.3⁩98.0%
Phone numbers96.4%95% CI ⁨95.9⁩–⁨96.9⁩99.7%
Amounts outside the HR sector12.4%95% CI ⁨11.6⁩–⁨13.3⁩99.9%
All value types, general policy78.1%95% CI ⁨77.6⁩–⁨78.5⁩97.8%
All value types, HR policy93.3%95% CI ⁨93.0⁩–⁨93.6⁩97.9%

Rows for single value types use the general policy. National ID and Iqama share one precision figure: the detector flags both as an ID number.

Organisations in this table: rules + trained model. All other rows: rules only.

Fingerprint of the checksum file: 6cb3251356780d7ea9750eb770c41b9ae0ce5357ec71165673bfa91099f8256b

bench-files-v6

⁨402⁩ documents · ⁨14,514⁩ planted values · ⁨svc-detector v1.28⁩ · measured ⁨15 November 2026⁩

Value typeDetectedPrecision
Structured identifiers, combined95.1%95% CI ⁨94.3⁩–⁨95.9⁩–
National ID96.2%95% CI ⁨94.3⁩–⁨97.5⁩95.2%
Iqama100.0%95% CI ⁨99.1⁩–⁨100.0⁩95.2%
Commercial registration82.1%95% CI ⁨79.0⁩–⁨84.8⁩96.2%
IBAN100.0%95% CI ⁨99.2⁩–⁨100.0⁩100.0%
Email100.0%95% CI ⁨99.5⁩–⁨100.0⁩100.0%
Employee numbers83.4%95% CI ⁨80.8⁩–⁨85.7⁩100.0%
Names90.7%95% CI ⁨89.9⁩–⁨91.5⁩93.9%
Organisations62.9%95% CI ⁨60.9⁩–⁨64.8⁩98.9%
Phone numbers94.4%95% CI ⁨93.3⁩–⁨95.4⁩100.0%
Amounts outside the HR sector29.6%95% CI ⁨27.5⁩–⁨31.8⁩100.0%
All value types, general policy79.6%95% CI ⁨78.9⁩–⁨80.2⁩96.8%
All value types, HR policy86.7%95% CI ⁨86.1⁩–⁨87.2⁩97.0%

Rows for single value types use the general policy. National ID and Iqama share one precision figure: the detector flags both as an ID number.

Fingerprint of the checksum file: 35301a2cad1d1132970725aa7d6dc2eca75d484a565ddd902a0bd8e12f161da5

bench-files-v5

⁨300⁩ documents · ⁨18,342⁩ planted values · ⁨svc-detector v1.25⁩ · measured ⁨15 November 2026⁩

Value typeDetectedPrecision
Structured identifiers, combined97.6%95% CI ⁨97.1⁩–⁨98.1⁩–
National ID97.3%95% CI ⁨95.6⁩–⁨98.3⁩99.0%
Iqama96.3%95% CI ⁨94.7⁩–⁨97.4⁩99.0%
Commercial registration90.8%95% CI ⁨88.1⁩–⁨93.0⁩100.0%
IBAN99.6%95% CI ⁨98.9⁩–⁨99.9⁩100.0%
Email100.0%95% CI ⁨99.7⁩–⁨100.0⁩100.0%
Names87.0%95% CI ⁨86.2⁩–⁨87.8⁩97.1%
Organisations85.3%95% CI ⁨84.0⁩–⁨86.4⁩100.0%
Phone numbers93.2%95% CI ⁨92.1⁩–⁨94.2⁩100.0%
Amounts outside the HR sector33.6%95% CI ⁨31.5⁩–⁨35.8⁩100.0%
All value types, general policy84.6%95% CI ⁨84.0⁩–⁨85.1⁩98.8%
All value types, HR policy90.0%95% CI ⁨89.6⁩–⁨90.4⁩98.0%

Rows for single value types use the general policy. National ID and Iqama share one precision figure: the detector flags both as an ID number.

Employee numbers were not labelled as sensitive in this set.

Fingerprint of the checksum file: 729f7ee175851864b855d952b33f00290fd0f828b915b53ffeef692a9ce0afb3

Names and organisations by file format

Detection rate by the file the value was in, on ⁨bench-files-v7⁩, with its 95% confidence interval. Values in a file the extractor could not read fully count as missed.

File formatNamesOrganisations
CSV98.6%95% CI ⁨98.1⁩–⁨99.1⁩81.9%95% CI ⁨79.4⁩–⁨84.1⁩
DOCX91.8%95% CI ⁨90.2⁩–⁨93.1⁩85.1%95% CI ⁨82.7⁩–⁨87.2⁩
JSON92.8%95% CI ⁨90.7⁩–⁨94.5⁩78.8%95% CI ⁨74.1⁩–⁨82.9⁩
PDF90.5%95% CI ⁨88.7⁩–⁨91.9⁩84.5%95% CI ⁨81.8⁩–⁨86.8⁩
XLSX97.6%95% CI ⁨96.9⁩–⁨98.1⁩90.8%95% CI ⁨89.0⁩–⁨92.3⁩

Amounts outside the HR sector

Outside the HR sector, only amounts written with a currency (such as «ريال» or SAR) are cloaked, by design: a bare number in a contract or invoice may be a quantity, a reference or a year, and cloaking it would take context the model needs. In HR, amounts are cloaked in context (salaries, allowances, pay columns). The figure below is therefore low on purpose, and any organisation can turn on in-context amount cloaking for any sector.

12.4%

detected · 95% CI ⁨11.6⁩–⁨13.3⁩ · precision ⁨99.9⁩% · bench-files-v7

Limitations

  • The documents are synthetic. Results on real documents may differ.
  • The test sets differ from each other, and results move between them, most of all for organisations (see the tables above). Expect similar variation on your own documents.
  • Amounts: outside the HR sector only amounts written with a currency are cloaked, by design, so that figure is low on purpose.
  • Names and organisations are detected less reliably than structured identifiers, and are being improved.
  • Precision is reported in the tables above and beside the homepage figures: a value flagged by mistake is cloaked unnecessarily but is never exposed.
  • A new held-out benchmark is required before these figures are updated again.

Source

Results reports:

  • tests/accuracy/results/bench-files-v7-REPORT-2026-11-16.md
  • tests/accuracy/results/bench-files-v6-REPORT-2026-11-15.md
  • tests/accuracy/results/bench-files-v5-REPORT-2026-11-15.md

The accuracy figures come from running the detector code on synthetic test documents, outside the live service; the live counters come from evaluation workspaces. No customer data is included.