Install and start using TONL in 30 seconds
npm install tonl
import { TONLDocument } from 'tonl';
// Create from JSON
const doc = TONLDocument.fromJSON({
users: [{ name: 'Alice', age: 30 }]
});
// Query
const result = doc.query('users[*].name');
// Modify
doc.set('users[0].age', 31);
// Save
await doc.save('data.tonl');
encodeTONL(data)
Maximum token savings (38-50%), no type hints
encodeTONL(data, { includeTypes: true })
Schema validation enabled (~32% savings)
encodeTONL(data, { delimiter: '|' })
Use pipe, tab, or semicolon delimiters
encodeSmart(data)
Auto-selects best delimiter and options
Need more?
Check out the full getting started guide on GitHub.
Complete TONLDocument API and core functions
Create TONL document from JavaScript object
const doc = TONLDocument.fromJSON({ users: [...] })
Parse TONL text into document
const doc = TONLDocument.fromTONL(tonlText)
Load TONL file from disk
const doc = await TONLDocument.load('data.tonl')
JSONPath-like queries with filters
doc.query('users[?(@.role == "admin")]')
Get single value at path
doc.get('users[0].name') // "Alice"
Check if path exists
doc.has('users[0].email') // true/false
Update value at path
doc.set('users[0].age', 31)
Remove field or array element
doc.delete('user.tempField')
Append to array
doc.push('users', newUser)
Deep merge objects
doc.merge('config', updates)
Export as TONL string
const tonlText = doc.toTONL()
Export as JavaScript object
const obj = doc.toJSON()
Atomic file save with backup
await doc.save('output.tonl')
JSONPath-like query expressions
==, !=, >, <, >=, <=
&&, ||, !
contains, startsWith, endsWith
Command-line interface for TONL
tonl stats data.json --interactive
tonl stats data.json -i --theme neon
tonl stats data.json --compare
tonl stats data.json --tokenizer claude-sonnet-4.5
Experience the future of CLI with real-time analysis and beautiful themes
tonl encode data.json --smart
tonl decode data.tonl
tonl query users.tonl 'users[*]'
tonl get data.tonl "user.name"
tonl validate --schema schema.tonl
tonl stats data.json --tokenizer claude-sonnet-4.5
tonl stats your-data.json --interactive
Launch the interactive dashboard with real-time file analysis and beautiful visual feedback.
tonl stats data.json -i --theme neon # Bright
colors
tonl stats data.json -i --theme matrix #
Green terminal
tonl stats data.json -i --theme
cyberpunk # Futuristic style
tonl stats data.json
-i --theme default # Clean terminal
Choose from 4 stunning visual themes for personalized experience.
tonl stats data.json --tokenizer gpt-5 # GPT-5
tonl stats data.json --tokenizer
claude-sonnet-4.5 # Claude Sonnet 4.5
tonl stats data.json
--tokenizer gemini-2.5-pro # Gemini 2.5 Pro
tonl stats
data.json --tokenizer gemini-3-pro # Gemini 3 Pro
tonl
stats data.json --tokenizer llama-4 # Llama 4
tonl
stats data.json --tokenizer o200k # OpenAI o200k
tonl
stats data.json --tokenizer cl100k # OpenAI
cl100k
Compare token costs across different LLM models in real-time.
System prompt for teaching LLMs to read TONL data
Copy this into your LLM system prompt when sending TONL formatted data:
The following data is in TONL format. Parse it as follows:
• Lines with [count]{fields}: are array headers, data rows follow
• Lines with {fields}: are object headers, field: value pairs follow
• Indentation (2 spaces) indicates nesting levels
• Default delimiter is comma unless #delimiter header specifies otherwise
• Type hints may appear: field:type (e.g., id:u32, name:str)
→ Ignore the :type part, just parse the values
• Value types: unquoted numbers/booleans, quoted strings, null
• v2.0 Optimization: May contain #optimize directives (ignore these, they're metadata)
Examples:
Without types: users[2]{id,name,role}:
With types: users[2]{id:u32,name:str,role:str}:
With optimization: #optimize dictionary delta bitpack
All parse the same - just read the data values.
This represents: {"users": [{"id":1,"name":"Alice","role":"admin"}, {"id":2,"name":"Bob","role":"user"}]}
TONL v2.0 provides 60% additional compression while maintaining full LLM compatibility.
Define and validate data structures with TONL schemas
Schemas let you define data structure, types, and constraints. TONL validates data against schemas and can auto-generate TypeScript types.
@schema v1
@strict true
User: obj
id: u32 required
username: str required min:3 max:20
email: str required pattern:email
age: u32? min:13 max:150
roles: list<str> required
users: list<User> required min:1
import { parseSchema, validateTONL } from 'tonl/schema';
// Load and parse schema
const schema = parseSchema(schemaContent);
// Validate data
const result = validateTONL(data, schema);
if (!result.valid) {
result.errors.forEach(err => {
console.error(`${err.field}: ${err.message}`);
});
}
str, u32, i32, u64, i64, f32, f64, bool
obj, list, list<T>, optional (field?)
min, max, pattern, email, url
min, max, positive, negative, integer
Process multi-GB files with constant memory usage
Stream processing allows you to work with files larger than available RAM. TONL's line-based format is perfect for streaming - process records one at a time with <100MB memory usage.
import { createEncodeStream } from 'tonl/stream';
import { createReadStream, createWriteStream } from 'fs';
// Stream encode large JSON files
createReadStream('huge.json')
.pipe(createEncodeStream({ smart: true }))
.pipe(createWriteStream('huge.tonl'));
import { streamQuery } from 'tonl/stream';
// Query huge files efficiently
await streamQuery(
'large-dataset.tonl',
'users[?(@.active)]',
(chunk) => {
// Process each matching chunk
console.log(chunk);
}
);
// Memory stays constant ~10MB
10 optimization strategies for up to 60% additional compression
TONL v2.0 introduces 10 advanced optimization strategies that provide up to 60% additional compression beyond standard TONL encoding.
Dictionary, Delta, RLE, Bit Packing, Column Reorder, Quantizer, Schema Inheritance, Hierarchical Grouping, Tokenizer Aware, Adaptive
Additional compression beyond standard TONL with automatic strategy selection
One-line activation with automatic optimization
tonl encode data.json --optimize --verbose
Enable all optimization strategies with a single command
Compress repetitive values using lookup dictionaries
Compress sequential numeric data (timestamps, IDs, counters)
Compress repeated consecutive values
Optimized binary encoding for booleans and small integers
Optimize field order for better compression
Reduce decimal precision safely
import { AdaptiveOptimizer } from 'tonl';
// Create optimizer
const optimizer = new AdaptiveOptimizer();
// Analyze dataset
const analysis = optimizer.analyzeDataset(data);
console.log(`Estimated savings: ${analysis.estimatedSavings}%`);
// Apply optimization
const result = optimizer.optimize(data);
console.log(`Directives: ${result.directives.length}`);
Build TONL libraries in any language
95KB of implementation documentation with algorithms, pseudo-code, and test requirements.
Powerful data aggregation with fluent API - count, sum, avg, min, max, groupBy and more
Full-featured aggregation system with 15+ functions, fluent chaining, and deep integration with TONLDocument queries.
import { TONLDocument } from 'tonl';
const doc = TONLDocument.fromJSON({ users: [...], orders: [...] });
// Count users
doc.count('users[*]'); // 42
// Sum order totals
doc.sum('orders[*]', 'total'); // 15420.50
// Average age
doc.avg('users[*]', 'age'); // 29.5
// Group by country
doc.groupBy('users[*]', 'country'); // { TR: [...], US: [...] }
// Chained operations
doc.aggregate('users[*]')
.filter(u => u.active)
.orderBy('age', 'desc')
.take(10)
.toArray();
count(), sum(field), avg(field), min(field), max(field)
groupBy(field), distinct(field), frequency(field)
stats(), median(), percentile(n), variance, stdDev
filter(), map(), reduce(), flatten()
first(), last(), at(n), take(n), skip(n)
orderBy(field, 'asc'|'desc'), partition(fn)
const stats = doc.aggregate('products[*]').stats('price');
// Returns:
// {
// count: 150,
// sum: 7499.50,
// avg: 49.99,
// min: 9.99,
// max: 999.99,
// variance: 12500.25,
// stdDev: 111.80
// }
Levenshtein, Jaro-Winkler, Soundex algorithms for approximate string matching
Find similar strings even with typos, variations, or phonetic similarities. Perfect for search, name matching, and data cleanup.
import {
levenshteinDistance,
fuzzyMatch,
soundsLike,
fuzzySearch
} from 'tonl/query/fuzzy-matcher';
// Levenshtein distance
levenshteinDistance('kitten', 'sitting'); // 3
// Fuzzy match with threshold
fuzzyMatch('John', 'Jon', { threshold: 0.8 }); // true
// Phonetic matching
soundsLike('Smith', 'Smyth'); // true
// Search with ranking
fuzzySearch('JavaScrpt', ['JavaScript', 'TypeScript', 'Python']);
// [{ value: 'JavaScript', similarity: 0.9, index: 0 }]
Edit distance - insertions, deletions, substitutions
Optimized for short strings with prefix bonus
Bigram overlap for longer text comparison
Phonetic encoding for name matching
Filter and compare dates with natural syntax - @now-7d, @today, before, after, sameWeek
Query data by date ranges, relative times, and calendar periods. Perfect for logs, events, and time-series data.
// Events in the last 7 days
doc.query('events[?(@.date > @now-7d)]');
// Orders from today
doc.query('orders[?(@.createdAt sameDay @today)]');
// Logs before a specific date
doc.query('logs[?(@.timestamp before @2025-01-01)]');
// Tasks due this week
doc.query('tasks[?(@.dueDate sameWeek @now)]');
// Records older than 3 months
doc.query('records[?(@.date daysAgo 90)]');
before, after, between
daysAgo, weeksAgo, monthsAgo, yearsAgo
sameDay, sameWeek, sameMonth, sameYear
s(sec), m(min), h(hour), d(day), w(week), M(month), y(year)
import {
parseTemporalLiteral,
isBefore,
isAfter,
isSameDay
} from 'tonl/query/temporal-evaluator';
// Parse temporal literal
const lastWeek = parseTemporalLiteral('@now-7d');
console.log(lastWeek.timestamp); // Unix timestamp
// Compare dates
isBefore(new Date('2025-01-01'), new Date('2025-12-31')); // true
isAfter(new Date('2025-12-31'), new Date('2025-01-01')); // true
isSameDay(new Date(), new Date()); // true