1,937,516,961 Tokens Evaluated with 100.00% Roundtrip Fidelity
An exhaustive empirical benchmark proving zero information loss, sub-millisecond core engine latency, and enterprise-grade SLA stability over TLS 1.3 across 9,334,805 prompts (~1.94 billion tokens) and 10,281,399 intercepted entities.
📦 1. The Unified Multi-Source Benchmark Dataset
Evaluated against a composite corpus of 9,334,805 real-world prompts (~1.94 billion tokens) stored in Apache Parquet format across 94 row groups.
⚡ 2. Full In-Memory Engine Benchmark (9,334,805 Prompts)
Direct multi-core worker evaluation of the standalone tokenization and rehydration engine. Every single prompt was tokenized, then rehydrated, and checked for strict byte-for-byte exact equality: rehydratedText === originalPrompt.
| Benchmark Metric | Observed Value | Auditable Standard / Meaning |
|---|---|---|
| Total Prompts Evaluated | 9,334,805 | 100% of entire unified parquet dataset across 94 row groups |
| Exact Roundtrip Matches | 9,334,805 | 100.00% Bit-for-Bit Exact Fidelity |
| Roundtrip Mismatches | 0 (Zero) | Zero token collisions, zero dropped characters, zero hallucinations |
| Total Entities Protected | 10,281,399 entities | Sovereign IDs, credit cards, emails, phone numbers, tracking numbers |
| Estimated Tokens Analyzed | 1,937,516,961 tokens | ~1.94 Billion tokens of text processed end-to-end |
| Total Execution Time | 526.35 seconds (8.77 min) | Continuous parallel multi-core execution |
| Overall Engine Throughput | 17,735.0 prompts / sec | V8 engine speed across multi-threaded workers |
| Average Processing Latency | 0.314 ms (314 µs) | Sub-millisecond latency per prompt |
| Median (p50) Processing Latency | 0.086 ms (86 µs) | Ultra-low compute overhead for production LLM pipelines |
🌐 3. Real-World HTTPS Socket SLA Benchmark (50,000 Prompts)
To measure production-grade network performance (including TLS 1.3 encryption handshakes, TCP socket pooling, HTTP headers, JSON serialization, and full roundtrips), a dedicated network benchmark was conducted on 50,000 random prompts sampled uniformly across all 94 row groups of the dataset.
2. Client POST /v1/detokenize over TLS 1.3 HTTPS socket → Server resolves tokens from session map and rehydrates the original text.
3. Integrity Assertion: Client verifies byte-for-byte that
rehydratedText === originalPrompt.
| Network Metric | Observed Performance | Operational Notes |
|---|---|---|
| Prompts Tested Over HTTPS | 50,000 prompts | Sampled uniformly across all 94 row groups (9.33M dataset) |
| Total HTTPS Network Requests | 100,000 requests | 50,000 Tokenize calls + 50,000 Detokenize calls |
| Total Elapsed Network Time | 37.15 seconds | Tested with 30 concurrent socket streams (undici.Pool) |
| Overall Prompt Throughput | 1,345.98 prompts / sec | End-to-end multi-stream client-to-API processing |
| Overall Network RPS | 2,691.95 HTTP req / sec | Continuous TLS 1.3 socket throughput |
| Token Throughput | 277,506 tokens / sec | Estimated over standard 4-char token density |
| Total Entities Sanitized | 54,071 entities | Detected and replaced across test set |
| Exact Roundtrip Matches | 50,000 / 50,000 (100.00%) | Zero mismatches over network sockets |
| Socket / HTTP Drop Errors | 0 | 100% network reliability under high concurrency |
📈 4. Interactive Benchmark Analytics & Visualizations
Explore interactive performance curves, throughput scaling across enterprise solutions, error elimination progression, and dataset entity distributions. Click legends or toggle controls to customize views.
⏱️ 5. Real-World Latency Percentile Matrix
Percentile distribution matrix of network calls over TLS 1.3 encrypted HTTPS connections under sustained 30-stream production load:
| Operation | Min | Average | p50 (Median) | p90 | p95 | p99 | p99.9 | Max |
|---|---|---|---|---|---|---|---|---|
| POST /v1/tokenize | 4.60 ms | 14.35 ms | 13.30 ms | 20.98 ms | 25.22 ms | 38.00 ms | 62.68 ms | 73.93 ms |
| POST /v1/detokenize | 4.50 ms | 7.92 ms | 7.20 ms | 10.91 ms | 12.81 ms | 18.31 ms | 26.96 ms | 64.22 ms |
| Total Roundtrip (End-to-End) | 12.06 ms | 22.27 ms | 20.81 ms | 30.86 ms | 36.36 ms | 51.55 ms | 79.22 ms | 104.20 ms |
🔧 6. Evolution to Absolute 100.00% Perfection
Achieving 100% roundtrip fidelity on a 9.33-million prompt corpus required diagnosing and resolving 5 distinct boundary-level edge cases:
| Iteration | Total Prompts | Mismatches | Fidelity Rate | Root Cause Addressed & Technical Fix |
|---|---|---|---|---|
| Run 1 (Initial Engine) | 9,334,805 | 339 (0.0036%) | 99.9964% | Alphanumeric tracking number suffixes matched phone patterns; UUID length caps; email gateway headers. |
| Run 2 (Post-Fix) | 9,334,805 | 9 (0.000096%) | 99.999903% | Added negative lookbehinds (?<![A-Za-z0-9]) to phone pattern; capped universal ID length; fixed 330 of 339 mismatches. |
| Run 3 (Hardened Engine) | 9,334,805 | 0 (0.000000%) | 100.00% | Fixed invoice pattern letter-boundary collisions (Ford-150, WORD-001) and Colombian national ID (CC992140300) prefix grouping. Clean sweep across all 9,334,805 prompts. |
🧪 7. Live Edge-Case Verifier
Test the exact edge-case prompts that previously failed in standard regex tokenizers and verify their deterministic roundtrip restoration in real-time:
💻 8. Reproduce the Benchmarks Locally
Both benchmarks are 100% reproducible on any multi-core machine using Node.js 18+ and the open-source repository:
• Full 9.33M Dataset Audit: multicore_benchmark_summary.json (100.00% fidelity)
• Real-World HTTPS Audit: http_latency_benchmark_summary.json (13.3ms median latency)