
# Handwritten Markdown: From Paper to Structured Text

I take a lot of handwritten notes — on tablets, in notebooks, on whatever is nearby. Writing by hand feels natural, and there are no distractions. But when I want to turn those notes into something searchable, editable, or shareable, the experience falls apart.

Cloud OCR services require subscriptions. Third-party tools chain together multiple services (Mathpix, Tesseract, LLMs) in fragile pipelines. None of them output structured Markdown. And none of them learn how I write.

This project is an attempt to solve all three problems at once: convert handwritten notes into clean Markdown, locally, with personalization built in. The initial prototype uses reMarkable tablet exports as its primary input source, but the approach works equally well with scanned paper documents.

## The Problem

Handwriting recognition is a solved problem — for printed text on clean backgrounds. For actual handwriting, the challenges multiply:

- **Messy input.** Real notes have crossed-out words, arrows, margin scribbles, and varying sizes.
- **No structure.** OCR tools produce flat text. But notes have headers, quotes, lists, and diagrams — losing that structure defeats the purpose.
- **Generic models.** Everyone's handwriting is different. A model trained on general handwriting data will consistently misread your specific quirks.
- **Privacy.** Sending personal notes to cloud APIs is not always acceptable.

Existing solutions each solve one of these, but none solve all of them:

| Approach                            | Private | Learns Your Writing | Structured Output |
| ----------------------------------- | ------- | ------------------- | ----------------- |
| Cloud APIs (Mathpix, Google Vision) | No      | No                  | No                |
| Local LLMs (Ollama + Mistral)       | Yes     | No                  | Basic             |
| Research libraries (TrOCR, Nephi)   | Yes     | Possible            | No                |
| **This project**                    | **Yes** | **Yes**             | **Yes**           |

## Markup Rules: Giving the Model Hints

The key insight is that you can make handwriting recognition dramatically easier if you add a few simple conventions to how you write. I call these **markup rules** — lightweight symbols you add while writing that tell the system what structure to produce.

**Structure markers** (at the start of a line):

- `#` → Heading level 1
- `##` → Heading level 2
- `>` → Block quote
- A drawn rectangle around a region → Image

**Text formatting:**

- **Bold** letters → converted to `**bold**`
- _Italic_ letters → converted to `*italic*`
- Lists → simply start each item with a dash

**Text mode markers:**

- Normal text (no marker) → dictionary correction is applied. If the model reads "teh", it corrects to "the".
- Text wrapped in `'single quotes'` → literal mode. No correction. Useful for technical terms, code, or foreign words that would otherwise be "fixed" by the language model.

These rules are minimal enough that they don't interfere with natural note-taking, but they give the recognition pipeline strong structural signals. Instead of trying to infer whether something is a heading from its visual size (unreliable), the `#` marker makes it unambiguous.

## How It Works

The system can read from multiple sources: reMarkable tablet exports, scanned documents, or photographed pages. No cloud sync, no API calls — just local file processing.

### Example

![Handwritten notes and their converted Markdown output](/public/images/posts/handwritten_example_converted.png)

## Why Not Just Use an LLM?

Large language models can do handwriting recognition — you can feed a page image to a vision model and ask it to transcribe. But there are trade-offs:

- **Cost and speed.** Running a large vision model on every page is slow and expensive, even locally.
- **No personalization.** The LLM doesn't learn your handwriting over time.
- **Inconsistent structure.** LLMs are unpredictable about formatting. Sometimes they add headers, sometimes they don't. The markup rules give deterministic structure.
- **Privacy.** Cloud LLMs send your data to external servers. Local LLMs require significant hardware.

A specialized CRNN model trained on handwriting is smaller, faster, and — once fine-tuned on your data — more accurate for this specific task.

## Current Status

The project is in its early stages. The data loading infrastructure works and the MNIST baseline validates the training pipeline. The markup rules are defined and documented.

<div data-upvote="handwritten-markdown"></div>


Handwritten Markdown Converter


# Building Multilingual Dictionaries for E-Readers

If you read books on an e-reader in a foreign language, you know the built-in dictionary is a lifesaver. Tap a word, get a definition, keep reading. But for less common language pairs, you're on your own — no e-reader manufacturer offers a Dutch-Hungarian dictionary, and neither does anyone else. The alternative is switching to a phone app every time you encounter an unfamiliar word, which breaks the reading flow entirely.

The problem goes deeper than just finding one translation. As a language learner, seeing a word translated into multiple languages you already speak helps you memorize faster. If I look up a Dutch word and see both the Hungarian and the English translation side by side, the meaning sticks better. That's a feature you won't find on any e-reader today.

That's the problem this project solves. It's a pipeline for collecting words, translations, and example sentences using modern tools — including LLMs and open databases — to build custom **multilingual** dictionaries that work on Kindle, Kobo, and other e-readers. Not just source-to-target, but source-to-multiple-targets. The pipeline is currently being developed and refined on a Dutch-Hungarian-English dictionary — a combination that matters to me personally — but it's designed to work with any language combination you choose.

## What Makes a Good Dictionary

A dictionary isn't built in one pass — it grows gradually, entry by entry, source by source. Each entry should aim to include what you'd expect from any proper dictionary: the word itself, its translations, pronunciation (IPA), part of speech (noun, verb, adjective), and example sentences showing how the word is used in context.

Since entries come from many different sources — curated databases, text corpora, LLMs — reliability varies. Every translation carries a **quality score** from 1 to 5. When the same translation is confirmed by multiple sources, the score increases. If Wiktionary says "huis" means "house" and OpenSubtitles agrees, that translation is more trustworthy than one generated by an LLM alone.

When a new source is imported, the system doesn't create duplicate entries. It merges intelligently: new translations are added to existing words, existing translations get their quality boosted if confirmed, and manual edits are never overwritten by automated imports. This way the dictionary gets richer and more reliable over time, without losing work that's already been done.

## The Pipeline

Building a dictionary is a four-step process. Each step adds a layer of depth and quality to the final result.

### Step 1: Collect Words

The foundation is a word list. Open databases like Wiktionary provide a rich, human-curated starting point — with definitions, pronunciation (IPA), and parts of speech already structured. A parser reads the Wiktionary data dump and extracts entries for the source language. These form the authoritative base of the dictionary.

### Step 2: Collect Sentences

A dictionary entry without context is hard to learn from. The next step is collecting real-world sentences where each word appears. Sources like **OpenSubtitles** — which aligns movie and TV subtitle files across languages — provide thousands of natural sentence pairs. Book corpora, news archives, and parallel text datasets all contribute too, each with different strengths: subtitles give colloquial language, news gives formal vocabulary, books give literary context. The system is designed to plug in new corpus sources as they become available.

### Step 3: Analyse and Structure

Raw sentences and word lists need processing before they become a dictionary. This is where text mining comes in: analysing grammar in each sentence, removing inflections, identifying whether a word is a noun, verb, or adjective, and mapping inflected forms back to their base form. Transformer-based models help here too — they can disambiguate words based on context, so a word like "bank" gets the right translation depending on whether the sentence is about finance or a riverbank. Hungarian, for example, adds many suffixes to words, so a single Dutch word might map to dozens of inflected forms — the pipeline detects and filters these. Translations found across multiple sources get a higher quality score, giving the reader a clear signal of reliability.

### Step 4: Fill the Gaps with LLMs

After curated sources and corpus analysis, there are still words without translations in every target language. This is where LLMs come in. They can generate translations, suggest example sentences, and fill in missing entries — all without being tied to a single translation provider. LLM-generated content gets a lower quality score than human-curated or corpus-derived translations, making it a useful fallback while keeping transparency about the source.

### Web Interface

The project includes a web UI that serves two purposes. First, it works as an online dictionary — you can search for any word just like you would on any dictionary website. Second, it allows human curation: you can review, correct, and enrich entries generated by the pipeline. You can add words by hand in `word - translation` format, add usage examples, and adjust quality scores. This human-in-the-loop step is what turns a raw data pipeline into a polished dictionary.

Once you're happy with the result, you can export the dictionary to e-reader formats — **Kindle (.mobi)**, **Kobo (dicthtml.zip)**, and **StarDict (.ifo/.idx/.dict)** for PocketBook, KOReader, and GoldenDict — and sideload it onto your device. All three formats are generated from the same underlying data. Or simply keep using the web interface as your go-to online dictionary.

## Are You Interested?

The architecture is language-agnostic — adding a new language combination means providing the right import sources and running the pipeline. Whether it's 2, 3, or 4 languages, the web interface, quality scoring, and export formats all work without changes.

<div data-upvote-feedback="inkdict"></div>


Building Multilingual Dictionaries for E-Readers


# Case Study: Ondash Architecture

Ondash is a financial analytics platform that takes accounting XML files and turns them into interactive dashboards, reports, and AI-powered insights. Behind the simple user experience — upload a file, get a report — is a microservices architecture with seven services, two databases, a message bus, and a real-time WebSocket layer.

This post walks through the architecture: what each service does, how they communicate, and the design decisions that shaped the system.

![Ondash Architecture](/public/images/posts/ondash-architecture.png)

## System Overview

The platform is built as a collection of independent services, each responsible for a specific capability:

| Service | Language | Purpose |
|---------|----------|---------|
| **Frontend** | React/TypeScript | Web UI with dashboards, charts, chat |
| **Backend** | Python/FastAPI | GraphQL API, business logic, data orchestration |
| **Finan** | Python/FastAPI | Hungarian accounting XML processing |
| **Agent** | Python/LangChain | AI task decomposition and scenario execution |
| **Update Broker** | Go | Real-time WebSocket gateway |
| **Report Engine** | Bun/TypeScript | PDF generation via headless Chrome |
| **Mail** | Bun/TypeScript | Email notifications and report delivery |

Services communicate through two channels: **GraphQL** for synchronous request-response, and **Redpanda (Kafka-compatible)** for asynchronous events. The distinction is deliberate — commands that need a response use GraphQL; notifications where the producer doesn't care about consumers go through Kafka.

## Data Flow: From XML to Dashboard

The end-to-end flow from file upload to interactive dashboard:

1. **User uploads XML** — The frontend sends the accounting file to the backend, which stores it in AWS S3
2. **Finan processes the XML** — Extracts company data, accounts, journal entries, and partner information from the Hungarian accounting format
3. **Embeddings are generated** — Account names are sent to OpenAI's text-embedding model for semantic search capability
4. **Data persists to PostgreSQL** — Each company gets its own schema (multi-tenancy via schema isolation)
5. **Standard codes are assigned** — An LLM matches accounts to standardized accounting codes using embeddings for context
6. **KPIs are calculated** — Financial metrics (liquidity, solvency, profitability) are computed from the processed data
7. **Dashboard queries execute** — The backend runs parameterized SQL queries against the company's data
8. **Real-time updates propagate** — The update broker pushes changes to connected clients via WebSocket
9. **Charts render** — The frontend visualizes results with ECharts
10. **PDF export** — The report engine captures the dashboard as a PDF using Puppeteer

## Frontend

**Stack:** React 18, TypeScript, Vite, Chakra UI, Apollo Client

The frontend is a single-page application built around a **dashboard system**. Each dashboard contains widgets (charts, tables, metrics) and parameters (filters like month, year, account type). When a user changes a parameter, the system identifies which widgets are affected, re-runs their SQL queries with the new parameter values, and updates the display.

Parameters use a template syntax in SQL queries:

```sql
WHERE DATE_TRUNC('month', e.date) = @month::date
AND account_type = @account_type
```

The `@parameter_id` placeholders are substituted server-side before execution. This keeps the SQL readable while making dashboards dynamic.

Real-time updates come through the Update Broker's WebSocket connection. The frontend uses a React Context with hooks (`useWebSocket`, `useTopicData`) to subscribe to relevant topics and filter incoming messages client-side.

## Backend

**Stack:** Python 3.9+, FastAPI, Strawberry GraphQL, MongoDB, PostgreSQL

The backend is the central orchestration layer. It exposes a **GraphQL API** (via Strawberry) that handles all data operations: CRUD for dashboards, projects, users, and chat; parameterized queries against company databases; and file management via S3.

**MongoDB** stores flexible documents — dashboard configurations, chat histories, project metadata, user settings. **PostgreSQL** stores structured financial data where ACID transactions and complex queries matter.

The repository pattern abstracts database access. Each adapter (MongoDB, PostgreSQL) implements a common interface, making it possible to swap storage backends without touching business logic.

Multi-tenancy is handled at two levels: MongoDB uses project-based isolation (each user sees only their projects), while PostgreSQL uses schema-per-company (the finan service creates a new schema for each uploaded company file).

## Finan: Financial Data Processing

**Stack:** Python, FastAPI, PostgreSQL (with pgvector), SQLModel

The finan service is specialized for Hungarian accounting XML files (Fökonyvi format). It runs a processing pipeline:

1. **Parse XML** — Extract company info, partners, journal entries, accounts
2. **Generate embeddings** — Send account names to OpenAI's `text-embedding-3-small` model
3. **Merge into PostgreSQL** — Upsert logic handles re-uploads gracefully
4. **Build monthly aggregates** — Roll up journal entries into account-level monthly totals
5. **Assign standard codes** — Match accounts to standardized accounting codes using LLM + embedding similarity
6. **Calculate KPIs** — Compute financial health indicators

The embedding step enables semantic search: even if two companies use different names for the same account type, the vector similarity finds the match. This is what allows the standard code assignment to work across different accounting conventions.

## Agent: AI-Powered Analysis

**Stack:** Python, FastAPI, LangChain, OpenAI/Anthropic

The agent service handles natural language queries about financial data. When a user asks a question in the chat, the agent:

1. **Decomposes the query** into atomic tasks
2. **Maps tasks to scenarios** — chart generation, dashboard creation, data lookup, etc.
3. **Executes scenarios** with access to the company's data
4. **Streams results** back via Kafka events

The event schema uses three Kafka topics:
- `ondash.llm.status` — Progress updates (with a completion flag)
- `ondash.llm.error` — Error notifications
- `ondash.llm.answer` — All content responses (text, charts, tables, widgets, dashboards, forms)

This separation means the frontend can show a progress indicator while the agent works, display errors immediately, and render results as they arrive — all through the same WebSocket channel.

## Update Broker: Real-Time WebSocket Gateway

**Stack:** Go, Gorilla WebSocket, Kafka-go, Protocol Buffers

The update broker bridges the event-driven backend with the browser-based frontend. It consumes messages from Redpanda (Kafka), deserializes them from Protocol Buffers, converts to JSON, and broadcasts to subscribed WebSocket clients.

The hub-and-spoke architecture keeps things simple: one Hub manages all client connections, routes messages based on topic subscriptions, and handles filtering. Providers (Kafka, MQTT) run as goroutines, auto-starting when a client subscribes and auto-stopping when the last subscriber leaves.

Messages are sent once per client regardless of how many subscriptions match, preventing duplicates. The frontend handles fine-grained filtering client-side.

## Report Engine and Mail

**Report Engine** (Bun/TypeScript) generates PDFs by rendering dashboards in headless Chrome via Puppeteer. This guarantees visual fidelity — the PDF looks exactly like the screen. Text remains selectable and copyable.

**Mail** (Bun/TypeScript) sends reports and notifications via SMTP, with support for attachments and inline charts.

Both services are intentionally simple — they do one thing and do it well.

## Infrastructure

**Kubernetes (EKS)** orchestrates all services. **ArgoCD** provides GitOps-based continuous deployment — push to main, and changes deploy automatically. Each service builds as a multi-architecture Docker image (amd64 + arm64) via GitHub Actions and pushes to AWS ECR.

**Monitoring** uses Grafana with Loki for log aggregation and Better Stack for centralized logging. Loki parses logs in `logfmt` format, making it easy to filter by service, level, or custom fields.

**Message format** across services uses Protocol Buffers (not JSON) for type-safe, compact inter-service communication. A shared `ondash-schemas` repo holds all `.proto` definitions, ensuring services agree on message structure at compile time.

## Key Design Decisions

**Commands vs. Events.** Synchronous operations (GraphQL mutations) are commands — the caller needs a response. Asynchronous notifications (Kafka) are events — the producer doesn't know or care who consumes them. This distinction keeps the architecture clean and prevents accidental coupling.

**Schema-per-company multi-tenancy.** Instead of a `company_id` column on every table, each company gets its own PostgreSQL schema. Queries are simpler (no WHERE clause needed), data is physically isolated, and dropping a company is just `DROP SCHEMA`.

**Protobuf-first messaging.** JSON is flexible but error-prone at scale. Protobuf schemas catch mismatches at build time and produce smaller payloads. The schema registry ensures all services speak the same language.

**Two databases, each for its strength.** MongoDB for flexible, evolving documents (dashboards, settings). PostgreSQL for structured, queryable data (financial records). Trying to use one for both would mean compromises in either direction.


Case study: Ondash Architecture


# Ondash: Financial Intelligence for Accounting

Accountants and auditors spend a surprising amount of their time not on analysis, but on preparing the analysis. They receive audit files, open them in spreadsheet tools, manually build charts, write summaries, compare year-over-year figures, and assemble everything into a presentable report for their clients. The actual expertise — interpreting the numbers, spotting risks, advising the client — gets squeezed into whatever time is left.

Ondash was built to flip that ratio. It takes standardized audit XML files as input and produces complete financial reports: analyses, interactive charts, and forward-looking projections. The accountant's job shifts from building the report to reviewing and refining it.

## What Problem Does It Solve?

Every accounting firm that works with annual accounts deals with the same workflow. A client submits their financial data, typically as a structured XML file following national or EU standards. The accountant then needs to:

1. **Parse the data** — extract balance sheet items, profit and loss figures, and metadata from the XML structure
2. **Run analyses** — calculate ratios (liquidity, solvency, profitability), compare against prior years, identify trends
3. **Visualize results** — create charts that make the numbers accessible to non-financial stakeholders
4. **Generate projections** — estimate where the company is heading based on historical trends
5. **Compile a report** — combine everything into a coherent document ready for the client meeting

Each step is well-defined and repetitive. The logic doesn't change much between clients — what changes is the data. This makes it a perfect candidate for automation.

## How Ondash Works

The core workflow is deliberately simple from the user's perspective:

**Upload** — The accountant uploads one or more audit XML files through the web interface. The system recognizes the file structure and extracts all relevant financial data automatically.

**Analyze** — Within moments, the platform runs a battery of standard financial analyses. Balance sheet composition, revenue trends, cost structure breakdowns, key ratios, and year-over-year comparisons are all generated without manual configuration.

**Review** — The results appear as an interactive dashboard. Charts are clickable and explorable. The accountant can drill into any number, verify the calculations, and add their own commentary or interpretation where needed.

**Export** — The final report can be exported as a polished document ready for client delivery. It includes all the analyses, visualizations, and projections in a professional format.

The pricing model reflects this simplicity: pay per dossier, starting from one euro per analysis. No subscription, no hidden fees. For firms processing many dossiers, bulk credit packs bring the cost down further.

## Financial Monitoring: The Core Component

At the heart of Ondash is the financial monitoring engine. It doesn't just display numbers — it watches them. The system understands what healthy financial indicators look like and flags deviations automatically.

When a company's liquidity ratio drops below industry norms, or when a cost category grows disproportionately fast, the report highlights it. This turns the output from a passive data dump into an active diagnostic tool. The accountant arrives at the client meeting not just with numbers, but with a prioritized list of topics worth discussing.

The monitoring covers the standard dimensions of financial health:

- **Liquidity** — Can the company meet short-term obligations?
- **Solvency** — Is the capital structure sustainable long-term?
- **Profitability** — Are margins healthy and stable?
- **Activity** — How efficiently is capital being deployed?
- **Growth** — Are revenues and assets trending in the right direction?

Each dimension gets its own section in the report with appropriate visualizations: bar charts for comparisons, line charts for trends, composition charts for structure analysis.

## Agentic Workflows: The Next Step

The current version of Ondash automates the mechanical parts of report generation. But the roadmap goes further. We are building agentic workflows — AI agents that don't just crunch numbers but reason about them.

An agentic workflow means the system can:

- **Interpret patterns** — Not just flag that revenue dropped 12%, but suggest possible causes based on which cost centers or product lines changed
- **Generate narratives** — Draft the explanatory text that accompanies each chart, in the accountant's professional language
- **Ask follow-up questions** — If the data is ambiguous or incomplete, the agent can identify what additional information would clarify the picture
- **Adapt to context** — Adjust the depth and focus of the analysis based on the company's industry, size, and history

The accountant remains in control. The agent produces a draft; the professional reviews, adjusts, and approves. Think of it as having a well-prepared junior analyst who does the groundwork and presents a first version, freeing the senior professional to focus on judgment and advice.

## Who Is It For?

Ondash is designed for accounting firms and audit teams, particularly those who:

- Process dozens or hundreds of annual accounts per year
- Want to deliver more insightful reports without proportionally increasing headcount
- Need consistent report quality across the team, not just from senior partners
- Value speed — turning around a client report in minutes rather than hours

The platform doesn't replace the accountant. It replaces the spreadsheet gymnastics, the manual chart building, and the copy-paste report assembly. The professional's expertise — knowing what the numbers mean for this specific client — remains irreplaceable.

## Where It's Heading

The vision for Ondash is to become the standard tool accounting teams reach for when they need to turn raw financial data into client-ready intelligence. The immediate focus is on processing Hungarian audit XML files. But the underlying engine — parse structured financial data, analyze it, visualize it, explain it — applies wherever standardized financial reporting exists.

As the agentic workflow capabilities mature, the platform will handle increasingly complex scenarios: multi-entity consolidations, sector-specific benchmarking, and predictive modeling that goes beyond simple trend extrapolation.

The goal is not to automate accounting. It's to automate the tedious parts so that accountants can do more of the work that actually requires their expertise.

---

Interested in trying Ondash or learning more? Visit [ondash.ai](https://ondash.ai/en/).


Ondash: Financial Intelligence for Accounting


# Python Microservice Starter

Python as a general purpose language can be used in many ways. Here I describe how I scaffold it as a microservice starter. I have used this scaffolding for more than 10 microservices and it works well.

Microservices don't need to do a lot usually — just a few things. Load data from somewhere and send it to another place, keep something in memory, control a device with an exposed API for another service, and so on.

So why bother with structure if it's a "micro" service? The answer is simple: if you build dozens of microservices, it helps if they all look the same. Every service needs to load config — should it be YAML, TOML, or JSON? Every service exposes an API — it would be confusing to use different technologies across the fleet.

In a microservice environment each service is small and easy to understand on its own. But if you have dozens and they all look different, you start to wish for consistency. This Python starter solves that problem.

# Architecture

First we need a good architectural structure. The picture below shows the hexagonal architecture we follow:

![Architecture](../public/images/posts/hexagon.excalidraw.svg)

**Service Core** — This is where the business logic lives. The code should echo the business language using correct terms and expressions. During meetings we build up a shared vocabulary and regularly check it. It is important that the developer speaks the same language as the business person to avoid misunderstandings.

**Adapters** — The service core is surrounded by adapters (or ports) that connect to external services, subscribe to events, reach persistent data, or talk to devices. Adapters can also handle monitoring and logging. Ideally these are reusable across services — a developer should recognize familiar patterns at a quick glance.

The layered structure looks like this:

```
┌─────────────────────────────────┐
│   API Layer (Declarative)       │
│   FastAPI routers, middleware,  │
│   request/response models       │
└──────────────────┬──────────────┘
                   │
┌──────────────────▼──────────────┐
│   Service Layer (Business Logic)│
│   Domain models, business rules,│
│   orchestration                 │
└──────────────────┬──────────────┘
                   │
┌──────────────────▼──────────────┐
│   Adapter Layer (External Deps) │
│   Database, MQTT, device APIs,  │
│   logging, monitoring           │
└─────────────────────────────────┘
```

The project structure reflects this separation — with one pragmatic adjustment. In a strict hexagonal layout, the API and GUI would live inside the adapters folder. But in practice, the API is the main entry point of the service and the GUI is an operational tool for monitoring and quick interactions. Burying them two levels deep makes the codebase harder to navigate without adding any real architectural benefit. So we keep them at the top level, where developers expect to find them:

```
src/
├── api/              # FastAPI endpoints, middleware, models
├── gui/              # NiceGUI monitoring dashboard
├── service/          # Business logic (domain language)
├── adapters/         # External integrations
│   ├── db/           # PostgreSQL adapters (3 patterns)
│   ├── mqtt/         # MQTT broker integration
│   └── device/       # Device communication
├── config/           # Configuration management
├── main.py           # Application bootstrap
└── loggerhelper.py   # Custom logging utility
```

# Tech Stack

## Config Files

A microservice runs in different environments — unit tests, local development, staging, production — and each one needs different settings. Unit tests might use fake adapters instead of real databases. Local development connects to a different database than production. Kubernetes injects secrets through environment variables rather than files on disk. And a single test case might need to override one specific value without touching anything else.

To handle all of this cleanly, we use **Pydantic Settings** with TOML files and a layered override system. TOML (Tom's Obvious Minimal Language) is a configuration file format designed for high human readability — similar in purpose to JSON or YAML, but cleaner and less error-prone for config files. Each layer overwrites the previous one:

1. **TOML files** (lowest priority) — `settings.toml` holds non-sensitive defaults. A second file, `.secrets.toml` (gitignored), adds credentials for local development. The later file overwrites the earlier, so secrets override defaults without duplicating them.
2. **Environment variables** — overwrite everything from the TOML files. This is how Kubernetes passes configuration through ConfigMaps and injects sensitive values from Secrets.
3. **Code initialization** (highest priority) — overwrites everything. This is how a unit test sets a specific value for a single test case without touching any file.

```toml
# settings.toml
[default.database]
host = "localhost"
port = 5432
name = "mydb"
password = "<placeholder>"

# .secrets.toml (gitignored)
[default.database]
password = "the-real-password"  # overwrites the placeholder above
```

The config is validated at startup by Pydantic. If a required field is missing or has the wrong type, the service fails fast with a clear error message instead of crashing later at runtime.

## FastAPI with Pydantic

**FastAPI** is the web framework. It gives us automatic Swagger documentation, request validation via Pydantic models, and async support out of the box.

Every service exposes its API through routers. The starter includes an example router with proper error handling and a custom exception handler that returns consistent error responses across all services.

```python
@router.get("/users/{user_id}", response_model=UserResponse)
async def get_user(user_id: int):
    user = service.get_user(user_id)
    if not user:
        raise HTTPException(status_code=404, detail="User not found")
    return user
```

A logging middleware tracks request/response timing on every call — useful for spotting performance issues without adding instrumentation to each endpoint. Certain paths (like `/monitor`) can be excluded from logging to reduce noise.

## Database

The starter includes three database adapter patterns, so you can pick what fits your use case:

### SQL Files Pattern

SQL queries live in separate `.sql` files and are loaded at initialization. Good for complex queries where you want syntax highlighting and easy review.

### Inline SQL Pattern

SQL is embedded directly in Python code using SQLModel/SQLAlchemy. More concise for simple CRUD operations.

### ORM Model Pattern

Pure SQLModel with Pydantic integration. No raw SQL — the ORM generates everything. Best for straightforward data models where you want type safety end-to-end.

All three patterns have corresponding tests included in the starter.

## Monitoring

Every microservice gets a built-in monitoring dashboard via **NiceGUI**, mounted at `/monitor`. It provides a web-based UI for checking service health, viewing real-time metrics, and inspecting state — without needing external monitoring tools during development.

This is especially useful in industrial environments where Grafana might not be set up yet, but you still need visibility into what the service is doing.

## Logging

The starter includes a custom `LoggerHelper` that wraps Python's logging module with:

- File and stream handlers
- Consistent formatting with timestamps and log level indicators
- Custom tags for filtering (e.g., `[DB]`, `[MQTT]`, `[API]`)
- Output to both console and `logs.txt`

## Graceful Shutdown

Microservices need to clean up properly — close database connections, finish processing messages, unsubscribe from MQTT topics. The starter uses a pub/sub pattern with `pypubsub`: when a SIGTERM or SIGINT arrives, a `"terminate"` message is broadcast. Each adapter subscribes to this message and handles its own cleanup.

This matters in Kubernetes where pods get a grace period to shut down before being killed.

# Testing

The starter follows the test pyramid:

- **Unit tests** — Fast, no external dependencies. Test business logic in isolation.
- **Integration tests** — Test adapters against real services (PostgreSQL, Redis, MQTT via Docker Compose).
- **End-to-end tests** — Full scenario testing using generated API clients from the OpenAPI schema.

```ini
# pyproject.toml
[tool.pytest.ini_options]
testpaths = ["src", "test"]
python_files = ["*_test.py", "test_*.py"]
```

The CI pipeline runs unit and integration tests on every push, with PostgreSQL and Redis as GitHub Actions services.

# Deployment

## Docker

The Dockerfile uses multi-stage builds:

- **Development stage**: includes code-server, git, debugging tools
- **Production stage**: minimal image running `python src/main.py`

```yaml
# docker-compose.yml includes:
- PostgreSQL (port 5335)
- Redis (port 6379)
- Mosquitto MQTT broker (ports 1883, 9001)
- Ignition (for industrial projects)
```

## CI/CD

The GitHub Actions pipeline has three reusable workflows:

1. **Test** — Linting (Flake8 + Ruff), pytest with service containers
2. **Build** — Docker buildx, push to Azure Container Registry with version tags (`v{YYYYMMDD}.{RUN_NUMBER}-{BRANCH}`)
3. **Release** — Git tagging, Kubernetes manifest updates, deploy via ArgoCD

The release workflow supports multiple environments (dev, production) and only triggers on `main` and `develop` branches.

## Kubernetes

Deployment uses Helm charts with Kustomize overlays for per-environment configuration. ArgoCD watches the repository and automatically syncs changes — true GitOps.

```
kubernetes/
├── base/          # Helm chart templates
└── overlays/      # Per-environment configs (dev, staging, prod)
```

# Task Runner

The starter uses **Invoke** as a task runner for common operations:

```bash
invoke test       # Run all tests
invoke build      # Build Docker image with version tag
invoke push       # Push to container registry
invoke testdata   # Initialize test database
```

This keeps the frequently used commands short and consistent across all services built from this starter.


Python Microservice Starter


# TimescaleDB Historian Module for Ignition

In industrial automation, time-series data is everywhere. Temperatures, pressures, flow rates, motor speeds — all sampled every second, stored for months or years, and queried for trending, reporting, and analysis. The platform that manages this data in many factories is **Ignition** by Inductive Automation.

Ignition's built-in SQL Historian works well at moderate scale. But when you push past 100 GB of historical data and start requesting aggregated views — hourly averages over a week, daily totals over a month — performance degrades in ways that are hard to work around. This module replaces the default historian with a direct connection to **TimescaleDB**, a PostgreSQL extension purpose-built for time-series data.

## What Is Ignition?

Ignition is an industrial automation platform used for SCADA (Supervisory Control and Data Acquisition), HMI (Human-Machine Interface), and data integration. It runs on Java, connects to PLCs and sensors on the factory floor, and provides tools for building dashboards, storing historical data, and running automated processes.

At the center of Ignition is the **tag system** — a hierarchical namespace where every sensor reading, calculated value, and status flag lives as a "tag." Tags can be configured to log their values to a **historian**, which stores every change with a timestamp for later retrieval.

**Ignition modules** are Java plugins that extend the platform's functionality. This TimescaleDB module implements a custom historian that plugs into Ignition's historian framework, replacing the default storage and query engines.

## The Problem with the Default Setup

When people try to use TimescaleDB with Ignition's standard SQL Historian, they hit several pain points:

**No server-side aggregation.** This is the biggest issue. When you request hourly averages over 7 days for 5 tags, Ignition's SQL Historian fetches *all* raw data points — potentially hundreds of thousands of rows — transfers them over JDBC to the Java gateway, and aggregates them in memory. TimescaleDB has `time_bucket()`, a native function that computes aggregations directly in the database. But the SQL Historian doesn't know about it.

**Schema mismatch.** Ignition's `sqlth_*` tables use `BIGINT` for timestamps and split values across `intvalue`/`floatvalue` columns. This doesn't align with TimescaleDB's expectations. Converting these tables to hypertables requires manual SQL, and the conversion can break if Ignition's schema changes.

**Partition conflicts.** Ignition manages its own partition system via `sqlth_partitions`. TimescaleDB has its own chunking. Running both creates conflicts that require careful workarounds.

**Performance cliff at scale.** Community workarounds (BRIN indexes, partition flags) help, but they're fragile and undocumented. Beyond ~100 GB, query performance becomes unpredictable.

## What This Module Does Differently

The module bypasses Ignition's SQL Historian entirely. It implements Ignition's Historian Extension Point API (introduced in Ignition 8.3) with its own storage engine, query engine, and schema — all designed for TimescaleDB from the ground up.

### Server-Side Aggregation

The single biggest performance improvement. When Ignition requests aggregated data, the module generates a SQL query using `time_bucket()` and the appropriate aggregate function (`AVG`, `SUM`, `COUNT`, `MIN`, `MAX`, etc.), and sends it directly to TimescaleDB.

The difference is dramatic. Requesting daily averages for 10 tags over 30 days:

| Approach | Rows transferred over JDBC |
|----------|---------------------------|
| SQL Historian (fetches all raw data) | ~25,900,000 |
| This module (server-side aggregation) | ~300 |

That's a 99.998% reduction in data transfer. Queries that take seconds with the default historian complete in milliseconds.

The module advertises support for 17 aggregation types: Average, Min, Max, Sum, Count, LastValue, Range, Variance, StdDev, and more. For each, it maps to the corresponding SQL function and lets TimescaleDB do the work.

### Optimized Schema

Instead of Ignition's legacy `sqlth_*` tables, the module creates two tables:

**`tags`** — An enrollment table mapping tag paths to integer IDs. Stores metadata (data type, creation time, engineering units) so the tag path string isn't repeated on every value row.

**`tag_values`** — A TimescaleDB hypertable partitioned on the `time` column. Stores `tag_id`, `value_double`, `value_string`, `quality`, and a proper `TIMESTAMPTZ` timestamp.

The schema is created automatically on first startup. No manual SQL, no conversion steps.

### Automatic Compression

After 7 days, chunks are automatically compressed with `segmentby=tag_id`. This means:
- Data is stored in columnar format — queries reading only `value_double` skip all other columns
- Only segments for requested tags are decompressed
- `COUNT`, `MIN`, `MAX` can use pre-computed metadata without decompressing rows at all

### Chunk Pruning

TimescaleDB automatically partitions data into time-based chunks (7 days by default). When a query asks for the last 24 hours, chunks from other weeks are never touched. Combined with the `(tag_id, time DESC)` index, this makes both raw and aggregated queries fast regardless of total data volume.

## Architecture

The module consists of five main components:

```
Ignition Tag Changes
        |
TimescaleDbHistorianGatewayHook (module entry point)
        |
TimescaleDbHistoryProvider (owns both engines)
    |                    |
StorageEngine         QueryEngine
    |                    |
    +---- TimescaleDbClient (shared JDBC pool + tag cache) ----+
                         |
                    TimescaleDB
```

**StorageEngine** receives tag value changes from Ignition, resolves tag paths to integer IDs (auto-enrolling new tags), and writes data in configurable batches (default: 100 points or every 5 seconds).

**QueryEngine** handles `system.tag.queryTagHistory()` calls. For raw queries, it runs a simple `SELECT` with time range filtering. For aggregated queries, it generates `time_bucket()` SQL and pushes the computation to the database.

**TimescaleDbClient** manages a HikariCP connection pool (10 connections) and an in-memory cache mapping tag paths to integer IDs. The cache is loaded on startup and updated as new tags are enrolled, eliminating a database lookup on every write.

## Configuration

Setting up a historian profile takes about a minute:

1. Install the module in Ignition's gateway
2. Go to **Config > Tags > History > Historians**
3. Create a new **TimescaleDB Historian** profile
4. Enter the JDBC URL, username, and password
5. Configure tags to use this historian as their history provider

The module handles schema creation, index management, and compression policies automatically.

Advanced settings include batch size, batch interval, debug logging, and optional Store & Forward integration for write buffering during database outages.

## Who Is This For?

This module is built for Ignition users who:
- Store large volumes of historical data (100 GB+)
- Need fast aggregated queries for dashboards and reports
- Want TimescaleDB's compression and retention features without manual schema management
- Run Ignition 8.3+ with Java 17

It's particularly relevant for manufacturing environments where trending dashboards show hourly or daily aggregates over weeks or months — exactly the queries where server-side aggregation makes the biggest difference.


Handwritten Markdown Converter

Building Multilingual Dictionaries for E-Readers

Case study: Ondash Architecture

Ondash: Financial Intelligence for Accounting

Python Microservice Starter

TimescaleDB Historian Module for Ignition

Backend State Manager: Tag Engine