Building analytics from scratch

At my previous workplace, when we launched our new website, we were lacking analytics. All we had were the surface-level numbers Cloudflare provides. The launch had been a quick one on purpose, and it was meant to push us forward despite our not being completely ready. We had scrapped PostHog at the last minute because of cost concerns, and after realizing the blatant hole, I quickly added Google Analytics back into the stack. However, I knew PostHog provided a key feature Google didn’t: user journeys. And with the rising capabilities of LLMs, I had a burning question in my head: “Could I just build the analytics myself?”

There were multiple important issues that were kindling this flame, and the core of it was that we didn’t have the ground-truth numbers that told us what we needed to optimize. If the conversion rate is low one day, how do we track down the issue? And is the issue a product issue, natural variance, or a bug in the code? Without knowing the answers to these questions, we were in the dark, and to me that was unacceptable.

There were various circumstances that fueled my anxiety, and the crux of it was that our business was at a critical point. As a result, I decided to take action quickly, a couple of days after we launched. Actually, this idea of building analytics from scratch wasn’t a fresh one. I had already played with it the previous year using Tinybird, a hosted ClickHouse solution, but the difference this time was that I wanted to set up our own server for ClickHouse. This also wasn’t without basis: I had previously tested using LLMs to do server management on Hetzner with great success.

Before I go deeper into the analytics, there are actually two pieces to this issue that I solved separately, and the foundation for the analytics was mostly in place because of a different problem. That problem was observability. We had long been plagued with issues hosting Svelte on Cloudflare. The problem was multi-dimensional: we had been experiencing page-load slowdowns, intermittent crashes, and even outages. These problems were very difficult to track down, as they did not occur locally, and the feedback loop using logs from Cloudflare was slow and inconsistent. Therefore, I knew I needed to build a system that made this whole situation much easier to deal with.

As with many good solutions, my idea didn’t start from scratch. I had long been aware of the Effect library as well as OpenTelemetry. A lot of the content creators I was following made videos on Effect around the same period, and they had piqued my curiosity, although I didn’t entirely understand the point of it. However, even with my limited understanding, I knew that it could probably be applied to our problem; in particular, this video opened my eyes to how well it applied.

So I set about refactoring most of our codebase, learning Effect and OpenTelemetry in the process. The short version is that Effect can be used to create programs, effectively wrappers around our uncertain logic, and these can also be chained together with spans that can be used to track down errors. This is useful on its own, but the real beauty comes from the scale of adoption. With a completely refactored codebase, we would be able to track down every user request, get timing info for each span and child span, and find the exact code locations of errors.

I clearly remember the moment I first saw the working system on a Grafana dashboard. It was like magic. I could see real-time error rates, average request latency, and a sorted list of the slowest spans in our codebase. Additionally, with the server being our own, I set up the AGENTS.md with the observability stack details, meaning any outage or heightened error rate was just a question away from being traced to its root cause. But this still wasn’t enough.

Tracing spans takes some time. For tracking user journeys and product analytics, it doesn’t cut it. For this, we needed a tool designed specifically for the job: ClickHouse.

Yet adding ClickHouse was not as difficult as it may seem. The process had already been used for Grafana and Prometheus; we just needed to create a new server for ClickHouse. Once it was set up, all that was left was to add event reporting at key points in the product journey, meaning every part of the sales funnels. Did the user click the buy now button? How far did they scroll down the page? How long did they spend on the page?

For example, I was able to locate multiple key points of friction in our checkout process, which had been a black box beforehand: guests would leave before logging in, the country switcher didn’t have every country, and multiple users left due to a checkout bug. I’m sure you can see the value. This is not just important; it is a vital part of making both business decisions and coding decisions. And when our sales dipped, it provided a feeling of security: the data was now there, and we just had to pinpoint what we were looking for.

Because we now owned the analytics and observability stack, we had the freedom to choose how it was displayed. A fast solution I used was Grafana, but we also had the freedom to put an analytics dashboard directly on our website, visible to admins. That is exactly what I did. If the analytics dashboard was slow, I could optimize it. If there was a particular metric important to us, I could visualize it. I could use whatever graph I pleased and position it wherever made sense.

Not long ago, a year to be exact, I had discarded the idea of hosting our own ClickHouse database. Everything I read pointed to it being complicated, and the query language was not something I wanted to tangle with. Advances in the capabilities of LLMs completely changed the equation, and they enabled me to build a different kind of foundation for our business: one driven by facts and precise details. If there is one lesson I would take from this, it’s to try something ambitious. After all, it might just work out better than you can imagine.