A text-first, LLM-friendly serialization format.
Up to 50% fewer tokens than JSON. Zero dependencies. Built for the AI era.
Same data structure, dramatically different token costs
Slide to see how token savings scale with data volume
Type hints enabled (+20 tokens)
Type hints (u32, str, bool) add ~20 tokens but enable schema validation, TypeScript generation, and better LLM understanding. Still 32% smaller than JSON!
More than just serialization - a complete data platform
32-50% fewer tokens than JSON. Reduces LLM API costs and speeds up processing with smart delimiter optimization.
users[2]{id,name}
JSONPath-like queries with filters, wildcards, recursive descent, and array slicing for powerful data extraction.
users[?(@.age > 18)]
Full CRUD operations with change tracking, atomic file saves, and rollback capabilities for safe edits.
doc.set('user.age', 31)
O(1) hash indices and O(log n) BTree indexes. 10-1600x faster than sequential scans on large datasets.
<0.1ms lookup times
Process multi-gigabyte files with constant memory usage. Perfect for large-scale data processing pipelines.
streamQuery('10GB.tonl')
Define schemas, validate data types, set constraints, and auto-generate TypeScript type definitions. Toggle "Show Types" above to see type hints.
age: u32 min:18 max:120
Traverse document trees, iterate over collections, and search hierarchies with intuitive APIs.
doc.walk(visitor)
Track all modifications with detailed diffs, timestamps, and rollback support for audit trails.
doc.getChanges()
Complete command-line toolkit for encoding, decoding, querying, validation, and formatting operations.
tonl encode --smart
See how TONL solves common data challenges
Optimize your AI workflows with massive token savings and faster processing times.
Query and analyze large datasets with lightning-fast performance and SQL-like syntax.
Store and manage application configs with human-readable format and validation.
Stream 50GB+ log files with <100MB memory. Perfect for ETL pipelines.
Query test data easily. No database needed. Fast test execution.
Track changes with diff engine. Rollback capability for safe migrations.
Compact cache storage with O(1) endpoint lookups and TTL support.
Fast SKU lookups, price range queries with BTree indexing.
Store articles, metadata. Query by tags, date ranges efficiently.
Snapshots, complete history, detailed change tracking with timestamps.
Efficient data sync. Change notifications. Fast queries for dashboards.
Copy this prompt to enable any LLM to parse TONL data
Add this to your LLM system prompt when sending TONL data
The following data is in TONL format. Parse it as follows:
• Lines with [count]{fields}: are array headers, data rows follow
• Lines with {fields}: are object headers, field: value pairs follow
• Indentation (2 spaces) indicates nesting levels
• Default delimiter is comma unless #delimiter header specifies otherwise
• Type hints may appear: field:type (e.g., id:u32, name:str, active:bool)
→ Ignore the :type part, just parse the values
• Value types: unquoted numbers/booleans, quoted strings, null
Examples:
Without types (compact):
users[2]{id,name,role}:
1, Alice, admin
2, Bob, user
With types (validation):
users[2]{id:u32,name:str,role:str}:
1, Alice, admin
2, Bob, user
Both represent: {"users": [{"id":1,"name":"Alice","role":"admin"}, {"id":2,"name":"Bob","role":"user"}]}💡 Pro tip: This prompt is ~150 tokens. Adding it to your system prompt is negligible compared to the 32-50% savings on your data!
Detailed LLM integration guide with more examples and edge cases
View on GitHubReal working code you can use today
// Filter users by condition
const admins = doc.query(
'users[?(@.role == "admin")]'
);
// Get nested values
const names = doc.query(
'users[*].name'
);
// Complex filters
const active = doc.query(
'users[?(@.age > 25 && @.active)]'
);
// Array slicing
const first = doc.query('users[0:5]');// Update single value
doc.set('users[0].age', 32);
// Add new item
doc.push('users', {
id: 1005,
name: 'Eve Green'
});
// Delete item
doc.delete('users[3]');
// Get changes
const changes = doc.getChanges();// Create hash index (O(1) lookup)
doc.createIndex('users', 'id', 'hash');
// Create BTree index (range queries)
doc.createIndex('users', 'age', 'btree');
// Ultra-fast indexed lookup
const user = doc.queryIndexed(
'users',
'id',
1001
); // ~0.05ms// Stream process huge files
await streamQuery(
'large.tonl',
'users[?(@.active)]',
(chunk) => {
// Process each chunk
console.log(chunk);
}
);
// Constant memory usage
// Works with 10GB+ filesReal numbers from production workloads
| Operation | Speed | Complexity | Details |
|---|---|---|---|
| Parse/Decode | ~1.2ms | O(n) |
1MB file, 10K records |
| Encode/Serialize | ~0.8ms | O(n) |
1MB data, smart optimization |
| Hash Index Lookup | ~0.05ms | O(1) |
10M records indexed |
| BTree Range Query | ~0.3ms | O(log n) |
100K results from 10M |
| Sequential Scan | ~45ms | O(n) |
10M records, no index |
| Stream Processing | ~50MB/s | O(1) memory |
Constant 10MB RAM usage |
| Modification | ~0.1ms | O(1) |
Single value update |
| File Save (Atomic) | ~5ms | O(n) |
1MB file with fsync |
Benchmark Environment: Node.js 20.x on Apple M1 Pro, 16GB RAM. Times shown are median of 1000 iterations. Hash index provides 10-1600x speedup over sequential scans for large datasets.
All commands you need for data operations
tonl encode data.json
Convert JSON to TONL format with smart optimization
tonl decode data.tonl
Convert TONL back to JSON format
tonl query file.tonl
Execute JSONPath queries on TONL files
tonl validate --schema
Validate files against schema definitions
tonl format file.tonl
Reformat files with specific delimiters
tonl stats --tokenizer
Compare token costs across LLM models
tonl generate-types
Auto-generate TypeScript type definitions
tonl --help
View detailed help and usage examples
# Convert JSON to TONL with optimization
tonl encode data.json --smart --stats
# Query for specific data
tonl query data.tonl 'users[?(@.age > 25)]' --output filtered.tonl
# Validate against schema
tonl validate filtered.tonl --schema user-schema.tonl
# Convert back to JSON
tonl decode filtered.tonl --out result.jsonTONL vs other popular data formats
| Feature | TONL | JSON | CSV | YAML |
|---|---|---|---|---|
| Token Efficiency | 32-50% smaller | Baseline | N/A | Similar to JSON |
| Nested Structures | ✅ | ✅ | ❌ | ✅ |
| Human Readable | ✅ | ✅ | ✅ | ✅ |
| Schema Validation | ✅ Built-in | Requires JSON Schema | ❌ | Limited |
| Query API | ✅ JSONPath | Library needed | ❌ | Library needed |
| Streaming Support | ✅ O(1) memory | Limited | ✅ | ❌ |
| Type System | 8 types | Dynamic | Strings only | Dynamic |
| LLM Optimized | ✅ Primary goal | ❌ | ❌ | ❌ |
| Change Tracking | ✅ Built-in diff | Library needed | ❌ | ❌ |
| Indexing (O(1) lookups) | ✅ Hash + BTree | ❌ | ❌ | ❌ |
Works everywhere - Node.js, Browser, Deno, Bun
<script type="module">
import { encodeTONL, decodeTONL }
from 'https://cdn.jsdelivr.net/npm/tonl@1.0.7/+esm';
const data = { users: [{ id: 1, name: 'Alice' }] };
const tonl = encodeTONL(data);
console.log(tonl);
</script><script src="https://unpkg.com/tonl@1.0.7/
dist/browser/tonl.umd.js"></script>
<script>
const tonl = TONL.encodeTONL({
hello: "world"
});
console.log(tonl);
</script>import { TONLDocument, encodeTONL, EncodeOptions } from 'tonl';
// Full IntelliSense support
const options: EncodeOptions = {
includeTypes: true,
delimiter: ',',
indent: 2
};
const doc = TONLDocument.fromJSON<UserData>(data);
// Type-safe queries and modifications
const admins = doc.query('users[?(@.role == "admin")]');Tested across 9 different data types
All benchmarks run on Node.js 20.x, Apple M1 Pro, 16GB RAM. Results are median of 1000 iterations.
Complete implementation guides and specifications available
Built for production, designed for efficiency
Save 32-50% on LLM API costs instantly. For apps processing millions of tokens daily, this translates to thousands of dollars saved monthly.
Pure TypeScript implementation with no runtime dependencies. No supply chain risks, no version conflicts, no security vulnerabilities from third-party code.
Hash indexes provide O(1) lookups - 1600x faster than sequential scans. BTree indexes enable sub-millisecond range queries on millions of records.
496 tests passing with 100% coverage. TypeScript strict mode. Semantic versioning. Comprehensive documentation. Used in production by multiple teams.
Join teams building efficient AI applications
"TONL has dramatically reduced our LLM API costs. We're processing the same amount of data but paying 32-50% less depending on the data structure. The query API is incredibly powerful, and the zero-dependency design means no supply chain risks."
Everything you need to know about TONL
TONL eliminates JSON's redundant syntax like repeated key names, excessive quotes, and bracket nesting. It uses a table-like format where column names are declared once in a header, then data rows follow with just values. The smart encoder also chooses optimal delimiters (, | ; or tab) based on your data to minimize escaping.
Yes! TONL provides perfect round-trip conversion with JSON. You can encode JSON to TONL for LLM prompts (saving tokens),
then decode back to JSON without any data loss. The CLI and API make integration seamless - just add
tonl encode/decode steps to your pipeline.
Unlike CSV, TONL supports nested objects, arrays, multiple data types, and maintains type information. Unlike JSON, it's optimized for token efficiency. Unlike YAML, it has a formal specification and fast parsers.
TONL also includes powerful features CSV lacks: JSONPath queries, schema validation, indexing for performance, streaming support for huge files, and built-in modification APIs.
Absolutely! TONL v1.0+ is production-ready with 496 passing tests, zero known bugs, and stable APIs. It has zero runtime dependencies, comprehensive documentation, and follows semantic versioning. The TypeScript implementation is fully typed and includes extensive error handling. Many projects are already using it successfully.
Getting started is simple:
npm install tonl
Then import and use in your code, or use the CLI for quick conversions. Check out our documentation and interactive playground to learn more.
Yes! TONL is a text format that works with any LLM that accepts text input - OpenAI GPT, Anthropic Claude, Google Gemini, Meta Llama, etc. The format is designed to be LLM-friendly and human-readable. Token savings are consistent across all major tokenizers (tested with GPT-5, Claude 3.5, Gemini 2.0, and Llama 4).
Join developers who are building smarter, more efficient AI applications with TONL