imtoanle

Elasticsearch for search, ClickHouse for analytics

Tue, 27 Oct 2026 00:00:00 GMT

From October 2025 to October 2026 I built a B2B search platform on my own: contact discovery and lead generation over hundreds of millions of searchable records. I designed it end to end, and I also deployed and operated it. That last part shaped every architecture decision, because any component I added was one I’d be running.

This post is about one of those decisions: which datastore does what.

Two workloads that look alike

The product asks two kinds of questions of the same data.

Interactive search and filtering. A user narrows a large dataset by indexed fields and expects results while they’re still looking at the screen.
Analytics and reporting. Aggregations, rankings, grouped counts and reporting scans over the same records.

Both are “queries over records”, which makes it tempting to answer both from one place. But they optimise for different things. Search wants fast discovery over indexed fields. Analytics wants to scan and aggregate many rows cheaply.

What was already working

Elasticsearch already handled the core search and filtering workload, and handled it well enough. The dataset was large, but large by itself wasn’t a reason to replace anything.

That matters, because the easy story here would be “Elasticsearch couldn’t cope, so I brought in ClickHouse”. That isn’t what happened, and I’d rather not tell it that way.

The decision

I kept the boundary simple:

Elasticsearch → interactive search / filtering
ClickHouse    → analytics / aggregation / ranking / reporting

Elasticsearch stays wherever the existing search and index model is enough. ClickHouse is used only for analytical access patterns that justify an extra pipeline. Its column-oriented storage suits aggregation and ranking, and its cost profile for those workloads was better. I’m keeping that cost point qualitative, since I don’t have a retained benchmark to put a number on it.

What I didn’t do

Keep everything in Elasticsearch. This was a real option. One datastore is simpler to run, and Elasticsearch can do aggregations. I chose the split because analytics was a distinct enough responsibility to deserve its own path, and because it kept reporting concerns from shaping the search design. It wasn’t because Elasticsearch couldn’t do it.

Replace Elasticsearch with ClickHouse. This would have mixed a storage migration into an analytics decision, to solve a search problem I didn’t have.

One generic datastore for everything. Minimising the number of technologies is a good default, but it shouldn’t be the guiding rule when two workloads clearly differ.

What the second datastore costs

A second datastore is not free, and on a solo project the person paying was me:

data has to be projected into an analytical shape and kept moving;
schema changes may need coordinating across the search and analytics projections;
freshness expectations have to be explicit, since the two can lag each other;
there’s more to monitor, deploy and recover when something fails.

So the rule I took from this isn’t “two databases are better than one”. It’s that a second datastore is justified when the workload boundary is worth its operating cost. Here it was. On a smaller dataset, or with lighter reporting, I’d have stayed with one.

Notes

The product is anonymised on purpose, so I call it “a B2B search platform” and describe its scale only in round terms.
This is a reconstructed decision record, written after the fact from my own production ownership. It shows how I reasoned, but it isn’t an independent audit.

I keep the general version of this rule in the garden: A second datastore has to earn its operating cost.

An AI-first OCR pipeline, with people for the exceptions

Tue, 06 Oct 2026 00:00:00 GMT

This project took a document workflow for vehicle dossiers that was done almost entirely by hand and made it automated by default.

Before

The core flow was manual:

receive images and documents;
classify them;
type in or extract the fields;
check the data;
prepare the submission.

Every step depended on a person, so every document waited for one.

After

I built the workflow end to end, along with the review tooling:

Intake from two places. Submissions arrive through a Zalo webhook, and staff can also create records in the admin.
Automatic classification of each document.
OCR field extraction.
Duplicate detection, so the same document doesn’t get processed twice.
Validation rules that check the extracted data.
Exception review for anything automation couldn’t handle.
Submission package generation at the end.

AI-first, with people for the exceptions

People don’t check every document before it moves on. They aren’t removed from the process either.

Documents are classified and extracted automatically. A person steps in only when automated processing fails or needs correcting. Human attention goes to the cases that need it, not to re-checking work that’s already right.

I’d describe it as AI-first with exception-based human review. It isn’t human-first, and it isn’t fully autonomous: the review step is where failed or wrong extractions get corrected.

The stack

Cloudflare Workers for the application and processing steps;
Queues to move documents through the stages;
D1 for structured data;
R2 for the images and files;
React for the admin and review tools.

Queues suit a staged pipeline like this one, where each document moves from intake through classification, extraction and validation.

What I can’t tell you

I have no published accuracy rate, throughput figure, document volume or time-saved number for this project. Without them, the honest claim is about the shape of the workflow: a manual pipeline became an automated one, with human review as the exception path. Any percentage I quoted would be one I made up.

The general idea is in the garden: Exception-based human review.

Reputation scoring over 64 million wallets

Tue, 15 Sep 2026 00:00:00 GMT

From September 2022 to January 2025 I worked at Octan Network as a blockchain developer, building backend services and internal tools in Go and TypeScript. The project I’m writing about here is reputation scoring: ranking on-chain wallets and serving analytics about them.

The project’s repository is public: Octan-Labs/Reputation-scoring.

My part

I didn’t own all of Octan, and this isn’t a “I built everything” post. On this project my scope was:

database design;
backend APIs;
the ClickHouse analytical workloads;
deployment, on Kubernetes and AWS;
performance optimisation.

The data stack combined TimescaleDB and ClickHouse.

The scale

Production data processing scanned approximately 64 million wallets. That’s the total number of wallets scanned. It isn’t a count of users or customers, or of wallets indexed at any one moment. Blockchain numbers get inflated easily, so I want to be precise about what this one means.

Why ClickHouse

Reputation scoring is mostly aggregation and ranking: summarise a lot of activity per wallet, then order wallets against each other. Those are analytical workloads, and ClickHouse fits them well. There was also pressure on infrastructure cost, and ClickHouse helped there too. I’m keeping that benefit qualitative because I don’t have a cost figure I can stand behind.

The results

After the optimisation work, roughly:

indexing went from hours to minutes;
search went from minutes to under five seconds.

These were production observations, not a controlled benchmark. I no longer have the logs or screenshots, so I give them as approximate first-person results and nothing more precise. If I quoted “a 37× speed-up” you’d be right to ask where the number came from, and I wouldn’t have an answer.

What I took from it

When one workload is mostly aggregation and ranking, give it a store built for that, rather than stretching the transactional or search side to cover it. I applied the same thinking later on a B2B search platform: Elasticsearch for search, ClickHouse for analytics.

The general rule, and when not to follow it, is in A second datastore has to earn its operating cost.

Moving parts of a Rails monolith to Go, not all of it

Tue, 25 Aug 2026 00:00:00 GMT

I worked at JobReady as a Ruby developer from November 2014 to October 2019, and as technical leader for a team of 5–7 engineers from July 2017 to October 2019. The work was workflow-heavy business software in Ruby on Rails: features, data changes, background processing, integrations and production support.

The leadership part

“Tech lead” means different things in different companies, so here’s what it meant there:

making architecture decisions, and owning the database design that came with them;
estimating work;
reviewing code;
mentoring engineers;
planning releases.

That’s technical leadership, not people management. I’m not claiming I hired or managed anyone.

The problem

Parts of the Rails monolith had become bottlenecks. We wanted certain capabilities to be faster and more reliable, and to be deployable on their own instead of only as part of the whole application.

The tempting answer

When a monolith hurts, “rewrite it in something faster” is always on the table. A full rewrite also means running two systems for a long time and re-discovering business rules that only exist in the old code.

We didn’t do that.

What we did

We extracted selected capabilities into Go services and left the rest in Rails. Something moved out of the monolith only if it was causing pain:

Performance: the capability was a bottleneck.
Reliability: it needed to be more dependable than it was inside the monolith.
Independent deployment: it needed to ship on its own schedule.

Everything else stayed in Rails, where the domain logic already lived.

What changed

The extracted capabilities got faster and more reliable, and could be deployed on their own. I don’t have numbers I can publish, so I’ll leave it at that rather than make some up.

The rule

The rule was “move what hurts, keep what works”. Each extraction had a specific reason behind it, which made it easy to explain and review, and the rest of the application kept running as it was.

The general version of this lives in the garden: Extract what hurts, keep what works.