How Uptime Cairn Stores Years of Monitoring Data — and Why p95 Was Surprisingly Difficult

When you build an uptime monitoring system, the obvious problem is checking whether a website is up. The less obvious problem comes later: what do you do with all the data?

A monitor checking every 20 seconds produces 4,320 heartbeats every day. Multiply that by hundreds or thousands of monitors and the numbers become enormous very quickly. Yet users still expect to open a dashboard months later and answer questions like:

• How reliable was this website last month?
• What was its average response time?
• What was the worst response time?
• Did performance degrade compared to the previous month?
• What did we promise this client, and did we meet it?

This is one of the problems Uptime Cairn was designed around from the beginning. The project is not intended to be just another uptime dashboard. Reporting is a first-class part of the product, and that means the underlying data architecture has to support long-term history without turning every query into a database archaeology project.

This article explains how that works, and the interesting problem we hit when we tried to make p95 latency fit into the same architecture.

The raw heartbeat problem#

Every check performed by a monitor produces a heartbeat. A simplified heartbeat contains a timestamp, a status, and a response_time.

Imagine a monitor checking a website every 20 seconds. That's 3 checks per minute, 180 per hour, 4,320 per day. Now imagine 5,000 monitors.

Keeping every raw heartbeat forever would have several problems:

1. Storage would continuously grow.
2. Long-range queries would become expensive.
3. Charts would have far too many points.
4. Generating reports would require processing huge amounts of unnecessary data.

But throwing the data away isn't acceptable either. Long-term uptime history is one of the most important pieces of an uptime monitoring system.

So Uptime Cairn uses tiered rollups. The raw data is temporary. The summaries become the history.

From raw heartbeats to years of history#

The rollup pipeline is deliberately simple. Raw heartbeats roll up into 1-minute buckets, then 5-minute, then hourly, then daily.

Each level is built from the level immediately before it. Five-minute buckets are built from one-minute buckets, hourly buckets from five-minute buckets, daily buckets from hourly buckets. Nothing ever reaches back past its own parent tier.

This matters because each stage doesn't need to go all the way back to raw data. Rebuilding one day means processing 24 hourly rows rather than thousands of individual heartbeats.

The retention policy follows the same structure:

• Raw data: 7 days
• 1-minute data: 30 days
• 5-minute data: 90 days
• Hourly data: 1 year
• Daily data: retained indefinitely

This gives Uptime Cairn detailed recent history while still allowing years of historical reporting. The underlying design is described in the project's architecture plan as one of the foundations that makes long-range reporting possible.

Why we store sums instead of averages#

Here's a subtle but important design decision. Suppose four one-minute buckets have response-time averages of 120 ms, 190 ms, 110 ms and 140 ms.

It might seem reasonable to calculate the five-minute average by simply averaging those numbers, giving (120 + 190 + 110 + 140) / 4 = 140 ms.

But that's wrong. Each bucket may contain a different number of measurements, and a failed check doesn't have a response time at all.

So rather than storing only the average, Uptime Cairn stores four values per bucket:

• rt_sum
• rt_count
• rt_min
• rt_max

The average can then always be reconstructed as rt_sum / rt_count. In the example from the architecture, the true result is 1490 / 11 = 135.5 ms rather than 140 ms.

This property survives every level of the rollup hierarchy. Counts can be added. Sums can be added. Minimums can be reduced to the smallest minimum, and maximums to the largest maximum. The resulting statistics remain mathematically meaningful regardless of how many aggregation levels we pass through.

No data is not downtime#

Monitoring systems need to distinguish between "the website was down" and "we didn't collect any data". Those aren't the same thing.

If a monitor is paused, a server is offline, or a probe doesn't execute any checks during a period, there may simply be no observations. Uptime Cairn treats that as a gap rather than automatically turning it into downtime. A period with no checks can produce null uptime rather than 0%, and on a status page that can be represented as a gap rather than falsely claiming an outage.

There is also a subtle case where a check exists but its result is unknown or skipped. That bucket can still exist while having zero successful and zero failed checks, resulting in the same null uptime semantics.

This distinction becomes particularly important when generating reports. A missing observation shouldn't silently become an incident.

Late data changes the design#

Monitoring data doesn't always arrive exactly when expected. Imagine a probe temporarily loses its connection. It might buffer results locally and send them later after reconnecting.

That creates a problem for rollups. A bucket that looked complete at 10:01 might receive additional data at 10:06. If the aggregation was performed only once, the stored result would become incorrect.

Uptime Cairn handles this by deliberately reprocessing recent history. The rollup worker periodically goes back and rebuilds recent ranges. The important part is that it doesn't incrementally add to an existing aggregate: it rebuilds the bucket from its source data and overwrites the result. That makes the operation idempotent. Run the same aggregation twice and you get the same result.

The system also waits briefly after a bucket closes before processing it, and limits how many chunks a single pass can process, so a long period of downtime doesn't suddenly result in one enormous database operation.

So far, everything fits nicely#

At this point the architecture is pleasantly straightforward. Every metric we care about can be represented using compact, composable values:

• Uptime is total_up / (total_up + total_down)
• Average response time is total_response_time / response_count
• Minimum is MIN(child_minimums)
• Maximum is MAX(child_maximums)

Then we reached p95. And that's where things got interesting.

The p95 problem#

Percentiles don't compose the way sums, counts, minimums and maximums do.

Suppose we have two buckets. One contains the values 100, 110, 120, 130 and 2000. The other contains 100, 110, 120, 130 and 140.

Knowing the p95 of each bucket doesn't give us enough information to calculate the exact p95 of the combined dataset. You can't take average(p95_a, p95_b). You can't take max(p95_a, p95_b) and call that the actual p95 either.

The fundamental problem is that a percentile describes the distribution of the underlying observations. Once those observations have been reduced to a percentile, most of the information needed to reconstruct the combined distribution is gone.

The project design document describes p95 as the one statistic in the rollup schema that doesn't naturally compose.

The tempting solution: MAX(p95)#

The current architecture initially handled this by taking the maximum p95 when creating coarser buckets. At first glance this sounds reasonable, and it has one mathematically useful property: it is an upper bound. If each sub-bucket has a p95, then the maximum of those p95 values is guaranteed to be greater than or equal to the true p95 of the combined dataset.

But there's a major catch. The approximation gets progressively worse as we move up the rollup hierarchy. A daily p95 eventually becomes the maximum of hourly p95s, which are themselves the maximum of five-minute p95s, and so on.

Eventually the daily number answers a very different question:

What was the worst p95 among all the smaller buckets during this day?

That's not the same as:

What was the p95 of all requests during this day?

The architecture document gives a good example: a single short GC pause can cause a daily p95 to look extremely high, even though the service was perfectly healthy for almost the entire day. At that point, calling the value simply "p95" becomes misleading.

Why not use a histogram?#

The mathematically cleaner solution is to store a mergeable distribution sketch or histogram. Instead of storing only p95 = 250 ms, we could store a compact representation of the response-time distribution. Two histograms can then be merged, and from the merged distribution we could calculate p50, p90, p95, p99 and anything else.

A DDSketch-style logarithmic histogram is particularly interesting, because its buckets can be merged using simple count addition while providing bounded relative error.

So why not just implement it? Because engineering is about trade-offs, not just mathematical correctness.

The cost of making p95 exact#

1. Storage#

A simple p95 value is tiny. A histogram requires multiple bucket and count pairs. The storage cost is manageable but no longer negligible, especially at the finest rollup levels where there can be hundreds of millions of rows.

2. More complicated writes#

The existing rollup operation can largely be expressed as straightforward SQL: SUM(), MIN(), MAX(). A mergeable histogram requires more specialised aggregation logic, which means introducing custom aggregation machinery into a system deliberately designed to remain lightweight and easy to deploy.

3. Cross-database consistency#

Uptime Cairn is designed around a path from embedded SQLite for simple installations towards PostgreSQL/Timescale-style infrastructure for larger deployments. The same inputs should produce the same results regardless of backend. A custom statistical structure creates another implementation that must remain consistent across environments.

4. The finest tier gets worse#

This is perhaps the most interesting trade-off. The current one-minute tier can calculate an exact p95 from raw observations. Replacing that with an approximate histogram means introducing approximation exactly where users are most likely to care about precision: during an incident.

The architecture analysis therefore concluded that immediately introducing a sketch would trade away accuracy at the finest resolution to solve a problem that primarily affects long-range aggregation.

The pragmatic decision for Uptime Cairn#

After looking at the alternatives, I decided not to pretend that we have an exact long-range p95 when we don't.

Instead, Uptime Cairn provides p95 for the most recent seven days, where raw heartbeat data is still available. That gives us an important property: when Uptime Cairn shows a p95 for the recent period, it calculates it from the underlying observations rather than repeatedly approximating it through multiple rollup levels.

For long-term reporting, we don't need to force p95 into the rollup architecture just because other metrics fit there. The reports focus instead on statistics that compose cleanly across the entire retention period.

Reports use average and maximum for long-term history#

For monthly reporting, Uptime Cairn provides:

• Average response time
• Maximum response time
• Uptime
• Other report-level availability information

Average response time remains exact because we retain the response-time sum and count. Maximum response time remains exact because maximum values compose perfectly across rollups.

So a monthly report can confidently answer "what was the average response time this month?" and "what was the worst response time recorded this month?" without pretending that an approximate statistic is an exact percentile. This is an intentional product decision rather than a missing feature.

Why this fits Uptime Cairn's purpose#

There is an important distinction between an uptime monitoring platform and a full APM platform. In an APM system, percentiles such as p95 and p99 are fundamental. Teams may need to know what percentage of requests took longer than their latency SLO.

That's a very different problem from "did my client's website stay online this month?", and Uptime Cairn is primarily concerned with the latter. For uptime monitoring and client reporting, an exact average, maximum and uptime figure can be considerably more useful than a misleading long-range p95. And when p95 really matters, during a recent incident, the raw data is still there for seven days.

Reporting is bigger than a graph#

This decision fits into the larger direction of Uptime Cairn. The project's goal is not simply to keep a graph on screen. Reporting is intended to become a first-class subsystem, and the roadmap includes scheduled reports, branded reports, multiple export formats, SLA/SLO reporting, incident post-mortems, comparative reports, custom report builders, historical browsing and expiry reporting.

For an agency, the useful monthly question isn't "what was the p95 latency?" It is more likely:

Can I send my client a professional report showing that their website was available 99.98% of the month, averaged 180 ms response time, and had a worst-case response of 2.4 seconds?

That's a much more directly actionable report.

A small architectural compromise can be the right architecture#

There's a temptation when building infrastructure software to solve every problem in the most theoretically complete way possible. Sometimes that's exactly what you should do. But sometimes the better engineering decision is to identify where precision actually matters and avoid adding complexity where it doesn't.

For Uptime Cairn, that splits the system in two.

Recent history, covering the last seven days, is served from raw observations. That gives an exact p95.

Long-term history, running from 30 days to 90 days to a year and then indefinitely, is served from composable rollups. That gives an exact average, maximum and uptime, and no p95 at all.

This keeps the storage model simple, keeps the rollup pipeline deterministic, and avoids introducing an approximation simply to satisfy the expectation that every time-series metric must exist at every resolution.

The bigger lesson#

The p95 problem turned out to be a useful architectural test. It forced us to ask a fundamental question: what information actually needs to survive aggregation?

• To reconstruct uptime, counts are enough.
• To reconstruct averages, a sum and a count are enough.
• To reconstruct extremes, a minimum and a maximum are enough.
• To reconstruct percentiles, you need the distribution.

Once you understand that distinction, the storage architecture becomes much clearer. And that is the philosophy behind Uptime Cairn's monitoring data pipeline: keep the raw data while it is valuable, compress it intelligently when it isn't, and never make a number look more precise than the underlying data allows.

The result is a monitoring system that can keep years of history without keeping years of individual heartbeats, while remaining honest about what its metrics actually mean.

That's a trade-off I'm comfortable making.