Open Source AI
Handwriting JSON
Turn handwritten forms, notes, and scanned paperwork into automation-ready JSON with a Python API and CLI.
Customer
Open-source developers automating handwritten document workflows
Timeline
May 2026 · v0.1.1
Status
Published · GitHub and PyPI
Capability
Stack
Outcome
Customer Context
Who they are and what world they live in
Many teams still collect operational data on paper: signup forms, field inspection notes, school permission slips, donation forms, clinic intake forms, KYC forms, surveys, maintenance reports, and old scanned records. OCR can return text, but automation usually needs structured JSON with predictable keys, validation behavior, and a shape that matches downstream software.
The Problem
The fuzzy ask, translated
The original OmmSai project solved one urgent healthcare use case involving roughly 15,000 handwritten prescription files. Handwriting JSON takes the reusable idea and generalizes it: forms, notes, and scanned paperwork should become automation-ready JSON for CRMs, spreadsheets, compliance workflows, school systems, clinic intake pipelines, and review queues.
The Constraints
Time · Budget · Regulatory · Technical · Organizational
Keep the public API small enough for developers to understand in one sitting
Support handwritten PDFs and images without making the product medical-only
Accept schema guidance through JSON Schema or example JSON so output can match automation needs
Do not ship a hosted service, Docker wrapper, REST API, or OCR fallback before the package core is stable
Make the CLI useful for experiments while keeping the Python API clean for production pipelines
Architecture Decisions
What I chose. What I rejected. Why.
Provider strategy
Chosen
LiteLLM-backed adapter for vision LLM calls
Rejected
Hard-coding one model vendor
Why
Extraction quality, cost, latency, and availability vary across vision LLM providers. LiteLLM gives Handwriting JSON a thin routing layer while keeping the package focused on document normalization, schema guidance, JSON parsing, and validation warnings.
Orchestration
Chosen
Simple package pipeline: input normalization → prompt builder → provider call → JSON parser → optional validation
Rejected
LangChain and LangGraph in v0.1
Why
The v0.1 release is not a multi-step agent workflow. Adding LangChain or LangGraph before retries, repair loops, human review routing, OCR fallback, or multi-model routing are needed would make the API heavier without improving the core handwritten forms-to-JSON use case.
Output guidance
Chosen
Optional JSON Schema or example JSON
Rejected
Free-form text-only extraction
Why
The goal is not just to read handwriting. The goal is to turn handwritten documents into structured JSON that software can reliably use. Schema guidance makes the expected shape explicit.
Distribution
Chosen
Python package plus Typer CLI on PyPI
Rejected
REST service or hosted SaaS for the first release
Why
A package and CLI fit the developer audience and keep adoption simple: install it, point it at a file, choose a vision model, and get JSON. Service deployment concerns can come later if the use cases demand them.
The Hard Problem
The one thing that almost broke the deployment
The hard product boundary was not letting the OmmSai prescription origin story define the whole product. A useful launch needs to make the broader automation pattern obvious: handwritten signup forms, inspection notes, permission slips, donation sheets, KYC forms, surveys, maintenance reports, clinic intake forms, and scanned records all have the same core problem.
The Fix
Positioned Handwriting JSON around handwritten workflow automation: normalize local paths, URLs, base64 strings, bytes, and file-like objects; detect PDF, PNG, JPG/JPEG, and WebP inputs; build schema-guided prompts; call a vision LLM through LiteLLM; parse JSON; and surface validation warnings. The package is intentionally a foundation, not a platform.
Production Reality
What I had to fix in week 2
The healthcare lessons still shaped the package: handwritten document automation needs schema discipline, explicit uncertainty, and outputs designed for downstream systems. The public package keeps OmmSai as proof that the problem is real while positioning Handwriting JSON as the reusable open-source tool for many paper-heavy workflows.
Lessons Carried Forward
What this taught me that I apply to every deployment
A real one-off deployment can produce a reusable open-source package when the core primitive is separated from the event-specific workflow
Handwritten workflow automation is a better product boundary than prescription extraction or generic OCR
LiteLLM is enough abstraction for v0.1 because provider routing matters, but agent orchestration does not yet
LangChain and LangGraph should enter only when the workflow grows into retries, repair loops, routing, or human review
Roadmap features like Docker, REST, and OCR fallback are useful only after the package contract is stable
Related Deployments