The Linter For
Fine-Tuning Data
Parallelogram validates your dataset before you spend GPU hours on it. If it exits 0, your training run won't fail because of data.
Works with
- Claude Code
- Codex
- Cursor
- Windsurf
- Cline
- GitHub Copilot
- Gemini CLI
- Zed
- JetBrains
- Warp
- TRAE
- Hugging Face
- GitHub Actions
Catch It Before The GPU Does
Find the bad examples. Fix what you can. Check the clean file.
Here’s what that looks like in a real workflow.
Three examples. Two issues. One clean file.
Parallelogram finds a duplicate record and a private email in train.jsonl. The fix writes two records to clean.jsonl: the duplicate is removed and the email becomes [REDACTED_EMAIL]. The original file stays untouched. A second check passes with no errors or warnings.
❯ parallelogram check train.jsonl i (global) [context-window] Context-window counts are approximate → no tokenizer specified — pass --tokenizer <model> (e.g. gpt-4o, Qwen/Qwen2.5-7B) for an exact count ✗ train.jsonl:2 [duplicates] Duplicate of line 1 → 2 copies total at lines: [1, 2] ! train.jsonl:3 [pii] Message 1 contains an email address: m***@***.com → The model can learn and repeat it; --fix replaces it with a placeholder ╭───────────── parallelogram ──────────────╮ │ 3 records 2 clean 1 errors 1 warnings │ ╰──────────────────────────────────────────╯ ❯ parallelogram check train.jsonl --fix --output clean.jsonl ✓ duplicates · 1 fix ✓ pii · 1 fix ✗ dropped: train.jsonl:2 → duplicates (dropped as fix) ╭─────────────────── parallelogram --fix ───────────────────╮ │ 3 records 1 unchanged 1 fixed 1 dropped 0 unparseable │ ╰───────────────────────────────────────────────────────────╯ → Wrote 2 records to clean.jsonl ❯ parallelogram check clean.jsonl i (global) [context-window] Context-window counts are approximate → no tokenizer specified — pass --tokenizer <model> (e.g. gpt-4o, Qwen/Qwen2.5-7B) for an exact count ╭───────────── parallelogram ──────────────╮ │ 2 records 2 clean 0 errors 0 warnings │ ╰──────────────────────────────────────────╯
name: Validate training data
on: [push, pull_request]
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: Thatayotlhe04/Parallelogram@v1
with:
path: data/train.jsonl
fail-on: error # errors (2) and regressions (3) always fail
# eval: data/eval.jsonl # fail on train/eval leakage
# baseline: baseline/report.json
# The health report lands in the job summary. A dataset PR that breaks
# role order fails the same way a code PR with a failing test does.$ claude mcp add parallelogram -- parallelogram mcp # Claude Code
$ codex mcp add parallelogram -- parallelogram mcp # Codex
→ check_dataset {"path": "train.jsonl", "eval_path": "eval.jsonl", "limit": 3}
{ "summary": { "errors": 4, "warnings": 2 }, "issues_truncated": true,
"by_rule": { "secrets": …, "preference": …, "duplicates": …, "eval-overlap": … },
"issues": [
{ "rule": "secrets", "line": 2, "message": "…OpenAI API key: sk-pro…", "fixable": true },
{ "rule": "preference", "line": 3, "message": "…answers are identical", "fixable": false },
{ "rule": "duplicates", "line": 4, "message": "Duplicate of line 1", "fixable": true } ] }
→ fix_dataset {"path": "train.jsonl", "eval_path": "eval.jsonl", "output": "clean.jsonl"}
{ "summary": { "fixed": 2, "dropped": 3, "emitted": 3 } } clean.jsonl rechecks cleanTHE CASE FOR CHECKING FIRST.
Your advantage starts before training.
A fine-tuned model learns your data—including its mistakes.
Catch checkable defects before you spend GPU hours.
Same model. Different data. Different result.
Exact-match accuracy · higher is better · controlled small-model test
Flag empty turns, malformed examples and invalid roles before training.
Catch train/eval overlap and redact secrets before they reach model weights.
In a separate test: 18 of 23 defect types flagged at 98–100% recall.
Test setup & limits
Accuracy: locally built 383K-parameter GPT-style model, 2,000 fine-tuning examples, 600 held-out extraction examples, five seeds. At 30% injection, empty answers reduced mean accuracy by 22.4 percentage points (95% paired t-interval: 18.8–26.0); systematic errors by 31.7 (30.3–33.1). At 10% the reductions were 4.6 (3.3–5.9) and 12.2 (10.6–13.8). Systematic errors actually affected 12.1% and 31.9% of rows.
Detection: separate test of Parallelogram 1.6.1, 100 injected rows per defect type, all rules, approximate token counts with a 2,048-token limit. The 98–100% range includes warnings. The base was filtered by all detectors, so false positives on natural data are not estimated. Both tests were conducted by Parallelogram’s author; these results do not measure accuracy recovered by the linter, production-model quality, or performance against frontier APIs.
One workload. Four ways to run it.
Estimated monthly spend · USD · lower is cheaper
Change the volume.
Watch the economics shift.
OpenAIGPT-6 AstraAPI · standard rate
AnthropicClaude Fable 5.2API · standard rate
GoogleGemini 4 CarbonAPI · standard rate
Self-hostedYour fine-tuneLlama-based · scenarioPricing & assumptions
Snapshot: 12 November 2026. API cost = tasks × (1,500 × input rate + 200 × output rate) ÷ 1,000,000. The 200 output tokens include any billed reasoning tokens; more reasoning increases cost. Tokenization varies by provider. Standard uncached text rates; no batch discounts, tools, taxes, retries or application-operation costs. Selected models are examples, not a complete market survey.
USD per million input/output tokens: GPT-6 Astra 10/50; Claude Fable 5.2 8/40; Gemini 4 Carbon 2/10. Efficient alternatives: GPT-6 Luna 0.10/0.50; Claude Haiku 5.5 0.10/0.50; Gemini 3.1 Flash-Lite 0.25/1.50. Gemini 4 Carbon’s promotional rates end 30 November 2026; this is a dated snapshot.
The self-hosted bar is an editable budget, not a measured Llama deployment or a vendor quote. The default assumes $2,800 hosting, $2,500 maintenance and $700 amortized training per month. It stays fixed as volume changes: increase it if extra GPUs or staffing are needed. Matching the APIs’ task quality and meeting throughput targets require your own evaluation. No Parallelogram savings or model-quality uplift is assumed.
Buy capability. Build specialization.
Different approaches to the same job—not a universal winner.
| What matters | | Your fine-tuneFor example, a self-hosted Llama adapter |
|---|---|---|
| Where to start | Broad reasoning & unfamiliar tasks | Stable, repeated tasks with examples |
| Behavior | Prompts, tools & retrieval | Training examples + your adapter |
| Model weights | Managed by the provider | You hold the adapter and licensed base |
| Serving location | Available provider regions | You choose the infrastructure |
| Operations | Provider runs the model | Your team serves, monitors & updates |
| Proof of usefulness | Evaluate on your actual workload | Evaluate on the same held-out tasks |
They can work together. Route familiar work to your specialized model and escalate the harder cases to a frontier API.
“In-house” here means a self-hosted, fine-tuned open-weight model—not training a foundation model from scratch. Capability depends on the base, data and task.
Whichever model you choose, start with better data.Check structure, duplicates, leakage and secrets before you train.
See how Parallelogram fitsEleven Checks Your Trainer Won't Run
Each one crashes the run, quietly trains the wrong thing, or leaks something it shouldn't.
All rules-
Tool Calls New in 1.6
Calls to tools the record never declares, call ids with no matching reply, arguments that aren't a JSON object. Covers OpenAI
tool_calls, legacyfunction_calland ShareGPTfunction_call/observationturns. Argument values are never printed.tool-callserror -
Secrets New in 1.1
API keys, tokens and private keys a model could repeat. Redacted by --fix, never printed in full.
secretserrorfixable -
Personal Data New in 1.1
Email addresses, phone numbers and card numbers. Replaced with placeholders by --fix.
piiwarningfixable -
Eval Leakage New in 1.1
Train records whose content or prompt also sits in your eval set, inflating the score.
eval-overlaperrorfixable -
Preference Pairs New in 1.1
DPO pairs whose chosen and rejected answers are the same, so they teach nothing.
preferenceerror -
Schema
Malformed JSONL, missing fields, wrong types, in any of six formats. Can't be disabled.
schemaerror -
Role Order
Doubled turns, misplaced system prompts, conversations that end on the user.
roleserror -
Empty Content
Blank or whitespace-only turns that teach the model silence.
empty-contenterrorfixable -
Duplicates
Repeated records that push the model toward memorizing instead of generalizing.
duplicateserrorfixable -
Context Window
Records over your token budget, which TRL truncates without telling you.
context-windowerrorfixable -
Encoding
BOM markers and mojibake that end up baked into the weights.
encodingwarningfixable
Check. Fix. Gate.
Every issue gets a line and a rule. Mechanical problems are repaired, the rest are dropped with a reason, and the re-check decides.
- 01 · check
parallelogram check data.jsonlEvery issue with its line number and rule. Exits 0, 1 or 2.
- 02 · fix
parallelogram check data.jsonl --fix --output clean.jsonlMechanical repairs, then a re-check. Anything still broken is dropped, with the reason.
- 03 · gate
uses: Thatayotlhe04/Parallelogram@v1The GitHub Action fails the pull request when dataset quality regresses.
- 1.6.0
Tool-calling datasets,
parallelogram integratefor eight agents, and a pre-commit hook - 1.5.1
Every field scanned for keys and personal data, and a CI gate that can't be crashed open
- 1.5.0
For coding agents: an MCP server, a Python API and
--json --limit - 1.1.1
Fixes from an independent review, including secrets quoted in diagnostics
- 1.1.0
Alpaca and DPO formats, secrets and PII checks, train/eval leakage with
--eval
What does "exits 0" actually guarantee?
With all rules on: every record parses, has a valid role order ending on the assistant, has no empty turns or exact duplicates, and fits your token budget (counted exactly with --tokenizer, estimated otherwise). It checks that your data will train, not that it's good data.
Does my data leave my machine?
No. Nothing is uploaded and there is no telemetry. The only network request is opt-in: --tokenizer may download a tokenizer vocabulary once, then caches it.
Which formats are supported?
OpenAI/Qwen chat, ShareGPT, prompt/completion, Alpaca and preference (DPO) pairs, detected per line and mixable in one file. --fix writes each record back in the shape it came in.
Will it catch a leaked API key?
Yes. The secrets check finds keys and tokens for OpenAI, Anthropic, AWS, GitHub, Hugging Face, Slack, Google and Stripe, plus private key blocks. It never prints them in full, and --fix redacts them.
Are token counts exact?
With pip install 'parallelogram[tokenizer]' and --tokenizer, yes (tiktoken or Hugging Face). Without, counts are estimated and overflows are warnings, never errors.
How do I use it in CI?
Add uses: Thatayotlhe04/Parallelogram@v1 to a workflow. It writes a report to the job summary and fails on errors, or on regressions against a --baseline.
Can my coding agent run it?
Yes. parallelogram mcp gives Claude Code, Codex and Cursor three tools: check, fix and list rules. Agents using the shell can run check --json --limit 50, which keeps every count exact but lists only the most important issues. Setup is in the agent guide.
Is it free?
Yes. Apache-2.0, on PyPI and GitHub.