Akumuli

Analyzing 500 Billion Rows Using Akumuli

Tue, 10 Mar 2020 12:00:00 +0000

It’s been a while since Scylla team demonstrated that their highly capable database can read 1B (1.000.000.000) rows per second on an 83-node cluster. This is roughly 12M rows per second per server.

The task was simple: store temperature measurements from one million homes for one year with one-minute readings. That about a half of a trillion data points - 1.000.000 * 60 * 24 * 365 = 525.600.000.000 rows! Moreover, as if the sheer volume was not enough they introduced an outlier to the dataset. The outlier is a single reading that’s supposed to be found by the query. This setup was meant to show how well their consistent hashing works.

It didn’t end there though. Folks from Altinity decided to repeat the test with Clickhouse using Intel NUC as their server! The NUC despite being a tiny PC with inadequate cooling was outfitted with 4-core Intel i7 CPU, 32G of RAM, and most importantly 1Tb NVMe SSD. Clickhouse was able to load that dataset in 17 hours and find the ‘needle’ in 8.6 seconds (using the pre-built index). Their compression algorithm was able to trim every data point to just 1.6 bytes on average. This benchmark is interesting because it demonstrates that with the right software you can outmaneuver a big expensive setup using cheap commodity hardware. Kudos to the Altinity team!

I decided to build own version of the test. I don’t have an Intel NUC so I ended up using 12 vCPU ‘Standard’ DigitalOcean droplet (which is DO’s word for VM) because it’s the cheapest option that offers 1TB of SSD storage. This instance has 12 shared vCPUs which means three things. First, the CPUs are shared between different VMs on the same host so you can expect inconsistent performance. The CPUs are hyper threads, not real CPUs. Third, this setup is nowhere near as good as the real dedicated machine so it’s cheap! My goal was to spend less than $50 on hosting while doing this test to demonstrate that Akumuli is cost-effective.

Input Data

I wrote a script that generates the test data. I wanted to use 8 vCPU ‘Standard’ droplet to run the script and to load the data into Akumuli. This droplet has 640GB of storage. I had to tweak the input to fit this droplet. It still has 525.6B unique data-points and 1M homes but everything was divided evenly between 10 metrics (‘temp0’ - ‘temp9’). It allowed to batch up to 10 values together in the input file and as result ‘gzip’ was able to compress everything way better.

The data was generated using a random walk within reasonable bounds for temperature (19C to 35C). Instead of introducing a single outlier the script introduced a series of outliers. It selected 100 homes and for each of them, it generated a series that was growing randomly up to 100C.

It took around two days to generate the input files. The script that does this is called ‘gen.sh’. The output was around 200GB compressed and around 4TB uncompressed.

Loading the Data

Because I used the shared VM I repeated the experiment several times. The best load time I’ve seen was 6 hours, the worst one - 7. The variance was because both VMs had shared vCPUs.

You can see that this run was affected by other VMs in the cloud by looking at this graph. I’ve been using 8 connections to load the data. This means that the Akumuli server could process the data using 8 threads. Akumuli doesn’t use all available CPUs for ingestion. It leaves some CPU power for queries so ingestion never makes the database unresponsive. On a 12 CPU machine, it uses only 8 CPUs for ingestion by default. The best-case ingestion speed was 24.319.822 elements / second. The network was quite busy too.

You can see that after 3 am something happened on the host machine. I never saw uneven distribution like this on dedicated hardware. The disk stats are also interesting:

The disk usage at the end of the test was 554GB. This means that Akumuli used 1.1 bytes/reading on average. Akumuli uses a novel time-series compression algorithm which leverages next value prediction.

Curiously, you can see this 3 am dent on every graph. This graph tells us that akumulid was using just a tad above 50Mbps of disk bandwidth to sustain that 24M elements per second write rate. This is why we need good time-series compression in our lives!

Aggregate Queries

Akumuli can find a single ‘needle’ without much trouble because it supports aggregation on the storage level out of the box. The ‘aggregate’ query can give you the outlier and its timestamp but you have to run this query for every series. This query can run faster if the series was compacted. Because of that, I ran it for both compacted and not compacted storage.

Also, Akumuli runs every query in a single thread by design. To leverage parallelism you have to run several queries in parallel. Because of that, I have single aggregate queries that run in one thread using 10% of data, and parallel queries that run in ten threads using all data. All queries were performed on a full year time range.

Single Aggregate

This query was performed on non-compacted storage. You can see that it took around 5 minutes to aggregate 52B data-points. This translates to a whooping 175M elements per second scan speed or 3ms execution time per series. Of cause it’s not a brute force scan.

Only a small portion of the storage was scanned to compute the aggregates. You can read about the algorithm that makes this possible here. This query wasn’t limited by disk bandwidth or CPU speed but by the disk IOPS. This is a network-attached SSD and the IOPS throttled at around 3K.

Compacted Single Aggregate

If the storage is compacted aggregate query works faster. Long story short, it took 3 seconds to aggregate 52B data points.

Run aggregate query in one thread @ 20200303T085219
11305509        
                           
real    0m2.512s                                  
user    0m0.046s
sys     0m0.126s  
Completed @ 20200303T085222

The graph doesn’t have enough resolution to show that. Of cause, the reason for that performance is caching. Index nodes were compact and hot in the cache.

Parallel Aggregate

Let’s run the same query using 10 threads and 10 times more data!

You can see that it took around 45 minutes to aggregate 525B data points. This was because the query hit that 3K IOPS wall.

But it’s still good considering that we’re aggregated 1M series one by one wasting less than 3 ms per series.

Compacted Parallel Aggregate

On compacted storage, the same query was way faster since it needed to touch a smaller number of disk pages.

It took only 13 seconds to complete. The memory requirements for the query were quite moderate BTW.

You can see that the aggregate query does the job pretty quickly, but the downside is that it returns an outlier for every unique time-series. In our case, it returns 1M values which is a 100MB data transfer. This may not be what you want since your app has to parse all this data to find the top outlier among them. To get rid of this last step we can use a filter. The filter can be added to any other query (except aggregate) and it will filter out all data points that don’t match the filter requirements. Akumuli has some tricks under a sleeve to make this filter queries fast.

Single Filter

Let’s see how a single filter query performs. I’d set the threshold to 95C for this workload.

The query completed in 204 seconds. It streamed the outliers it found to the client.

Filter query doesn’t depend that much on compaction so there is no need to run it for both compacted and not compacted states.

Parallel Filter

Let’s try the same approach using 10 queries.

The filter query found every temperature reading above 95C in 30 minutes. The database transferred back around 3GB of data at 15Mbps.

The bottleneck here was once again a 3000 IOPS limit. This workload can run up to 10 times faster on dedicated hardware with normal SSD (tens of thousands IOPS) or up to 100 times faster on NVMe SSD (hundreds of thousand IOPS).

Conclusion

This benchmark wasn’t meant to represent real-world performance. Most likely you won’t be running queries like these and you won’t be loading the data the same way I did. If you want to benchmark Akumuli more realistically, try the TSBS benchmark suite. It supports Akumuli now.

Nevertheless, this shows what the database can do. Also, note that typical monitoring workloads don’t require the database to run such high-cardinality queries. And if you will run queries used in this test on single series you’ll often end up with sub-millisecond latencies. Also, note that Akumuli was never optimized for heavy queries that end up scanning the entire dataset. Given that, this is a great result. You can indeed load a huge dataset and analyze it. And most importantly, you can build interactive applications (e.g. monitoring dashboards) without much trouble.

Runtime Cost of Write-Ahead Logging

Wed, 13 Mar 2019 12:00:00 +0000

Recent major Akumuli release added support for write-ahead logging. This is a huge milestone for several reasons.

It enables durable writes. Every time-series has a separate write buffer that could be lost in case of crash. Originally, Akumuli was developed with high-frequency data in mind. In this case it’s more important to restart the database after the crash as soon as possible. But if every series is updated rarely write buffers would contain hours of monitoring data.
Previously, Akumuli used to open for writing every time-series in the database. Even the ones that wasn’t updated. With new WAL individual time-series could be opened for writing only when needed and also closed when they no longer needed. This helps to deal with situation when new series are created on the fly and the cardinality grows but only a subset of series are being updated.
In the future it will allow to implement data replication via log shipping.
Also, it will allow to implement backfill. Akumuli have a means to write data to the past but the process can be efficient only if the data is batched and WAL provides the batching mechanism.

There are some cons though. The most important one is performance degradation. Akumuli in WAL mode have to do some extra job. Every data point have to be logged and WAL volumes have to be rotated from time to time. I don’t want to go into details to much (wait for the next post) but you should know that WAL have to be updated more frequently than main storage and the data in WAL is less compact.

I decided to measure the difference between Akumuli with and without WAL. Here is my test setup!

Load generator is a standard DigitalOcean droplet with 6-cores, 16GB of RAM and 320GB of disk space.
Target server runs on general purpose DigitalOcean droplet with 8-cores, 32GB of RAM and 100GB of disk space.

Both droplets have 6TB of transfer which is enough for the test.

Test Data

All benchmark code lives in this repository. To generate the data you have to clone it and run bash gen.sh. This script generates 6 gzip archives with test data in Akumuli format (the one that is normally used for ingestion). It takes many hours to finish and uses 222GB of disk space. This dataset should contain ~60-billion data-points from ~350000 unique time-series.

Individual series are generated using random walk algorithm. Every series name has six tags.

Target

On the target machine I installed Akumuli from the package repository. Then I configured it by running akumulid –init and editing ~/.akumulid configuration file. First of all I increased number of volumes to 20 (to use 80GB of disk out of available 100GB) and commented [WAL] section and all its elements. After that I created the database by running akumulid –create and started the server by running akumulid.

At this point I also setup the monitoring for both load generator and the target. I installed and configured Akumuli on load generator (all parameters left default). Also, I installed netdata on both machines. Both netdata instances was configured to send metrics in OpenTSDB format to the second Akumuli instance on load generator box. I also used apps plugin to generate metrics for the akumulid process. Also, I set up Grafana and installed Akumuli datasource plugin.

Running The Tests

To run the test ssh to the load generator and run bash run.sh. This script does two things. First of all it send data from every generated file to the Akumuli instance on the target machine in parallel. This is why 6-core droplet is needed. The data is sent via a bash itself using this syntax:

cat test-file.gz | gunzip > /dev/tcp/$IP/8282 &

Note that this command send uncompressed data so the actual transferred volume was not 222GB but close to 775GB. The second thing script does happens when all data is sent. It queries Akumuli instance on the target machine trying to fetch last data-point it had sent. It’s needed to make sure that all input was processed by the database server.

When run.sh finished I recorded the results, reconfigured target machine by resetting the database via

akumulid --delete
akumulid --create 

and uncommenting the [WAL] section in the configuration file. After that I started akumulid on the target machine and run.sh on load generator.

Here is how the first run looked like.

And here is the second one (with WAL).

You can see that all graphs are somewhat close. Test duration was tad longer in the second run. And the Disk I/O graph is the most interesting one. It clearly shows that second run was writing more data.

Result

	No WAL	WAL
Duration	11090 seconds (3:04:50)	13160 seconds (3:39:20)
Write rate	5.410.244 datapoints/sec	4.559.241 datapoints/sec
Relative speed	100%	82%

The results are very predictable. I had the same close to 20% slowdown on my laptop with WAL enabled. But considering the overall performance the impact is not that big. In software engineering, performance was always traded for variety of good things like security or durability. Given the fact that Akumuli scales with added CPU-cores almost linearly and that every CPU-core gives you extra +1M writes per second the choice could be easy.

On the other hand, WAL uses extra disk throughput. This means that with WAL Akumuli will wear SSD faster. Also, WAL significantly increases recovery time.

High-cardinality support

Fri, 03 Aug 2018 12:00:00 +0000

Every TSDB have to deal with vast data volumes. But it’s not only writes per second. There is a different dimension to the problem which is not tackled by most TSDB vendors. This problem is a dataset cardinality, the number of individual series in the database. It plays a really huge role.

It’s safe to assume that any TSDB will work reasonably well if the cardinality is small (100K of metrics). But when cardinality is high (millions of metrics) we have to deal with the hard problem. Most TSDBs will start to generate write timeout errors or eat up the RAM. For instance, Akumuli will use a lot of RAM because it allocates some memory for every time-series stored on disk.

Akumuli’s memory requirements depend on cardinality. To handle 1M unique time-series it needs around 10GB of RAM. 2M unique time-series will need 20GB and so on.

Where did this numbers came from? First of all, Akumuli writes data in fixed-size blocks. Each block is 4KB. When you create a time-series and write data into it, Akumuli allocates a memory block and fills it with compressed data. When it gets full it writes it to disk and allocates the next one. It have to write in 4KB blocks to minimize wear leveling and write amplification of the SSD. It will always allocate 4KB of memory for every time-series for this purpose. Even if only small fraction of this memory is used. So, the memory y expenses for all series will be cardinality * 4KB. If the database stores one million series the memory resident pages will eat around 3.8GB of RAM.

This problem was greatly reduced in latest release. New algorithm allocates 4KB pages partially by 1KB chunks! When you start writing it allocates first 1KB chunk (so almost empty block will use only 1KB of RAM, not 4KB). When the first chunk gets full it allocates the second one and so on, until all four chunks (4KB) will be allocated and filled with data. Now it’s time to write it to disk.

The problem is that we have four disjoint 1KB memory chunks that we need to write into one 4KB page on disk. We can’t just perform four 1KB writes because they won’t be atomic. If something will happen in the middle we will end up with partially updated block in the database! The database will be corrupted. Also, multiple updates are bad for SSDs. SSD can’t update pages in-place. It’s controller uses read-update-modify protocol to fetch old page, update it and write it to the new place. If we will write data this way it increase the wear leveling and will introduce a lot of work for the garbage collector inside the SSD controller.

So, we need to write this four 1KB chunks as one 4KB block. We can actually do this using the vector I/O (aka scatter/gather I/O). Using one writev call on Linux we can pack all four chunks into one iovec array which will be written atomically. Using only one write operation. This method is as efficient as ordinary write operation given that it involves only one syscall. But the most important, it provides atomicity so there is no in-between state. The page will be written completely or not written at all.

This simple trick allows to lower memory requirements by 50%. This is due to the fact that on average every series will have just two 1KB chunks allocated. I’m not going to bore you but it’s easy to prove if you curious.

This trick is already used by recent version of Akumuli but it’s not the entire solution. I’ll write about different memory optimizations later.

Scaling TSDB-specific operations

Sat, 28 Apr 2018 12:00:00 +0000

The point of using a specialized time-series database is to have an edge over conventional databases in time-series specific operations. Most often, TSDB’s are judged by their write speed. In my opinion, the read performance is as important if not the most. Moreover, not only plain reads but filtering, aggregation, and downsampling operations which are used in every monitoring dashboard or alerting system.

Many TSDB’s use rollup aggregation mechanism to enable fast data resampling and aggregation. It works the following way. The user needs to create a set of rollup aggregation rules in advance. For instance, for every time-series in the database, she may want to store raw data, average values with 5-minute step, and average values with 1-hour step. The database will update these 5-minute and 1-hour rollups periodically. The user may set different retention policies for different rollups.

The problem with this approach is that the user must know in advance, what rollups she will need. Regenerating rollups can be costly and in some circumstances, impossible. Different time-series will need different aggregation function. For instance, for latencies we may want to use max but for free disk-space min is more suitable. Another problem is that the user must use proper rollup table in queries. Otherwise, the query will turn into a full table scan.

Another often ignored problem is that not all aggregations are composable. For instance, if you have averages with the 5-minute step but you need averages with 1-hour step you can’t just use these first 5-minute averages to compute the result. You need to compute it from raw data or you will get an incorrect result.

The final downside is the cardinality growth. Most modern TSDB’s struggles with high cardinality datasets (the ones that have a lot of unique time-series).

Akumuli solves this problem by resampling/aggregating the data on the fly. It uses some tricks to make this fast.

Akumuli stores each time-series in a B+tree. B+trees has two types of nodes. The leaf nodes are used to store compressed time-series data. Leaf nodes have fixed size. Because of that, each leaf has different capacity that depends on randomness and precision of the actual data. The inner nodes are used to store the links to other nodes. Akumuli stores the set of aggregates (like min, max, and sum) along with child address. Each aggregate corresponds to the entire subtree. The storage engine can use these values to your advantage.

Aggregation

This aggregates can be leveraged for various TSDB specific computations. The most simple use case is a data aggregation. Suppose that we need to find the largest value inside the time interval.

In the simplest case, the interval will span a whole number of leaf nodes. We just need to find the root nodes of all subtrees over which the time interval spans. No matter how large the time-span is and how many data-points it contains. We will read just a handful of pages from disk and use them to compose the result.

In this example, the time interval spans over nodes 10, 11, 12, and 13. To calculate an aggregate for it we need to read the root node 1 and proceed to inner nodes 2 and 3. Both of which has the links to subtrees that have the data we need (subtree 5, 10, 11, and subtree 6, 12, 13). The algorithm can extract the aggregates from 2->5 link, and from 3->6 link. After that this aggregates can be combined into the final result.

In the more complex scenario, the interval will span a fractional number of leaf nodes. The beginning and the end of the interval will be located in the middle of the corresponding leaf nodes. But here we may use divide and conquer approach. We can split the task into three parts:

Read the first page that partially crosses the time-interval. This will read the actual leaf node from disk, decompress it and compute max for every data-point that sits inside the interval.
Compute the max for the subtrees that sits inside the interval entirely, as described above.
Read the last page that partially crosses the time-interval, as in 1).

After that, we can compute final result by composing three max values (we should just get the largest one).

For instance, let’s look at figure 2.

First, we need to read page 9 (by following the path 1->2->4) and find the max value.
After that, we need to compute the max value in pages 10..13 by using pages 2 and 3 (the same way as in the previous example).
And finally, we need to read page 14 (by following the 1->3->7 path) and find the max value.

Downsampling

First of all, ‘group-aggregate’ query in Akumuli is not supposed to outperform pre-computed rollups. It supposed to be used in conjunction with Grafana and similar interactive tools. Because of this, some assumptions have to be made. Let’s state that the user usually reads the same number of data points with varying step and interval. This makes perfect sense for Grafana. If the size of the graph doesn’t change Grahana will vary the step and time interval in such way the number of points will be about the same. For instance, if the interval is 30 minutes, the step will be 1 second, but if the interval is 3 hours the step will be increased to 6 seconds or so. In both cases, the database should return about the same number of points.

Let’s look at naive approach first. The naive algorithm scans the series and calculates the downsampled series. For small steps, it’s a pretty good algorithm. But when the step becomes larger the amount of I/O needed growth linearly.

The optimized version utilizes aggregation as a subroutine. For every step interval, we should compute an aggregate. It’s done using the algorithm described above. The only trick is that the step 3) of the first interval and step 1) of the next step are combined. This algorithm requires the same amount of I/O as naive algorithm if the step is small. But the amount of I/O operations wouldn’t grow linearly. It have an upper bound that depends on the number of extracted data-points.

Let’s perform some back of the envelope calculations. Let’s state that 1 block always has 1300 data points with 1-second step (around 3 bytes per data-point) and the query should return 400 data-points.

	1 sec	30 sec	5 min	30 min	1 hour	12 hour	1 day
Optimized	4KB	37KB	370KB	1.5MB	1.5MB	1.5MB	1.5MB
Naive	4KB	37KB	370KB	2.1MB	4.3MB	52MB	104MB

The naive approach will require the database to read around 37KB if the step is 30-seconds but if the step is 1-day the algorithm will read more than 100MB of data from disk. For the same query, the algorithm used by Akumuli will read only about 1.5MB. This is quite expensive compared to the rollup query. But on the other hand, it’s comparable with simple image download from a server.

Filtering

Akumuli also supports filter statement that can be added to select, join, and group-aggregate queries. This statement allows filtering time-series by value. This operation also utilizes pre-computed aggregates stored inside the B+tree nodes. The optimization is pretty simple. As the algorithm traverses the tree it may check does the link has any values that match the filter. If the subtree doesn’t have any it can be omitted. For instance, if we filtering out all values below 100 and the link to subtree has max=90 than we can safely prune the subtree from the search.

This approach works really well if we need to filter out everything but outliers. For instance, if no data point matches the filter the query will touch only root node of the tree. If all data points that match the filter are located in one leaf-node, the query will load only a couple of nodes. It will start from the root node and will follow the path until it reaches the leaf-node that has all the data we need.

Conslusions

These optimizations make a huge difference for many types of queries. An aggregate query execution lies within a millisecond range no matter the dataset. This means that it can be used as a subroutine in more complex algorithms. The filter query performance depends only on the number of returned data-points and works really fast if you need to search for outliers. And the downsampling aggregation can finally be used in interactive applications without any prior knowledge about the data and configuration.

Inverted Index

Fri, 17 Nov 2017 22:00:00 +0000

Tag support is very important for any modern time-series database. The world from which time-series data is coming is complex. Time-series data is not just a time-ordered values (measurements), this time ordered values form individual series and different series can relate to each other in numerous ways. The simplest example is an object that produces many measurements of different types. E.g. the server can have hundreds of different metrics like “CPU User”, “CPU System”, but more interestingly, it can have series names like “Number of software interrupts/sec of type X on core=Y” metric.

The particular series may look like this: proc.softirqs host=dev cpu=0 type=SCHED. The type tag may have tens of possible values as well as the cpu tag. This can be expressed using hierarchical naming scheme of Graphite in several ways, e.g. proc.softirqs.dev.0.SCHED or proc.softirqs.dev.SCHED.0 but here is the problem, the database user should know what each level of the name means! The hierarchical naming scheme is not self-descriptive. There should be a schema somewhere, it should be available to both readers and writers and it’s not stored in the database! So each agent on every machine should use proc.softirqs.dev.0.SCHED and not proc.softirqs.dev.SCHED.0 otherwise, you’re doomed. And when you see this proc.softirqs.dev.0.SCHED name you may only guess what this bloody ‘0’ means. The most critical problem is that this schema enforces hierarchy even when there is no hierarchy at all. In our example, both type and cpu names are equally important.

In Akumuli the tags are supported from the start. There is no fields or tables since everything can be modeled using tags. Tags not only can but should be redundant. There is a hard limit on series name length thought (4KB). Akumuli uses some modern techniques to enable this multi-dimensional queries and joins using tags. I’ll try to explain them in this article.

Forward Search

At the beginning, there was no index at all. Akumuli had been storing all series names in RAM in the string pool. There was a hash table that mapped ids to offsets inside the string pool, and the other hash table that mapped offsets to ids. New names were simply added to the string pool. Each series name was converted to the canonical form. In the canonical form, all unnecessary whitespace characters were removed and tags were sorted alphabetically.

To query this data structure regular expressions were used. Since all series names were stored contiguously it was possible to search the entire set of names using one regex search. This is quite dumb but it worked really well and it was quite fast. Regular expression libraries are fast and you can search through hundreds of megabytes of text in milliseconds. In IR this is called a forward search. Normal search engines used to deal with large text documents so forward search is not an option but time-series names are short so, one can use forward search with acceptable results even in interactive applications.

Inverted Index

However, some time ago I started to work on Grafana data source plugin. This plugin needed an autocomplete functionality from the database. The autocomplete is implemented using three queries:

suggest metric names by prefix (proc.cpu.???);
suggest tag name by tag prefix and metric name (proc.cpu.user hos???);
suggest tag value by value prefix, tag name, and metric name (proc.cpu.user host=dev-???);

It turned out that it’s problematic to implement this queries using regular expressions so I decided to implement an inverted index. In normal search engines, inverted index maps words to documents so all documents containing particular word combinations can be found without scanning each and every one of them. In Akumuli I’m using tag=value pairs instead of words and series names instead of documents. Check this example:

Let’s say that the data inside the index looks like this:

Id	Series Name
0	proc.softirqs host=dev cpu=0 type=SCHED
1	proc.softirqs host=dev cpu=1 type=SCHED
2	proc.softirqs host=dev cpu=0 type=TIMER
3	proc.softirqs host=dev cpu=1 type=TIMER
4	proc.softirqs host=test cpu=0 type=SCHED
5	proc.softirqs host=test cpu=1 type=SCHED
6	proc.softirqs host=test cpu=2 type=SCHED
7	proc.softirqs host=test cpu=3 type=SCHED
8	proc.softirqs host=test cpu=0 type=TIMER
9	proc.softirqs host=test cpu=1 type=TIMER
A	proc.softirqs host=test cpu=2 type=TIMER
B	proc.softirqs host=test cpu=3 type=TIMER

In this case these posting lists will be built:

Tag-Value pair	Posting list
host=dev	0, 1
host=test	3, 4, 5, 6, 7, 8, 9, A, B
cpu=0	0, 2, 4, 8
cpu=1	1, 3, 5, 9
cpu=2	6, A
cpu=3	7, B
type=SCHED	0, 1, 4, 5, 6, 7
type=TIMER	2, 3, 8, 9, A, B

So, suppose we need to find all the metrics from host ‘test’ that has a ‘SCHED’ type. We should read ‘host=test’ and ‘type=SCHED’ posting lists and find the intersection:

Tag-Value pair	Posting list
host=test	3, 4, 5, 6, 7, 8, 9, A, B
type=SCHED	0, 1, 4, 5, 6, 7

Query result is 4, 5, 6, 7:

Id	Series Name
4	proc.softirqs host=test cpu=0 type=SCHED
5	proc.softirqs host=test cpu=1 type=SCHED
6	proc.softirqs host=test cpu=2 type=SCHED
7	proc.softirqs host=test cpu=3 type=SCHED

That’s some basic IR stuff.

Optimizations

The main problem that I encountered was a high memory usage. Each series has a dozen of tags. Each tag-value pair has an associated postings list (an array that contains id’s of all series that contains this tag-value pair) so a value should be added to a dozen of posting lists. The value is 64-bit wide and a cardinality can be huge so these posting lists grow big. Apart from posting lists, I used series names as keys in a hash table that been used to map them to ids.

The first problem was solved by compressing posting lists in memory. Each posting list is just a sorted array of 64-bit integers. I started to build positing lists using delta encoding combined with LEB-128 and get 4x improvement as a result.

The second problem was solved using a string pool. I started to use the pointer to the value inside the string pool as a hash table key.

High Cardinality

The most complex problem that I have only appeared with high-cardinality data sets. There is quite a lot of unique tag-value pairs inside high-cardinality data sets. As result, a lot of small posting lists should be created. Compression won’t be effective in this situation (since every posting list will contain small number of ids) and memory fragmentation will become a problem.

This problem was solved using Count-Min Sketch. The normal CM-sketch used to count events. It consists of several arrays of counters. When the new event arrives, we should calculate N different hash values (N is a number of arrays we have), locate N counters, and increment them. To estimate the count for the event we should do almost the same - calculate the hashes, locate the counters but instead of incrementing them we should get the smallest one. This guarantees that hash collisions will affect the result the least.

In Akumuli’s index, the counters were replaced by the posting lists. When new series shows up, we need to compute N=3 hashes, locate 3 posting list, and add the new element to every one of them. As result, each posting list will have values from many tag-value pairs, not just one. To read the posting list we should do the same trick. Calculate 3 hashes, read 3 posting list, and then find the intersection between all of them. In a case of hash collision, the id of the series will end up in one posting lists out of three (way less likely in two or all of them). To eliminate the collisions further we should check that every series name that we’ve found contains the tag-value pair that we need using simple substring search.

This trick saves the day when we have a high cardinality set of tags. In case of Akumuli, the size of the table is fixed so there is no unneeded allocations and fragmentation. This inverted index can store really high number of series names and query it very efficiently.

Storage engine design (part 2)

Tue, 01 Aug 2017 13:00:00 +0000

In the previous article, I wrote about the reasons that made me choose B+tree based data-structure for Akumuli. In this article, I want to tell about another advantage of the B+tree based design compared to LSM-tree.

A shallow introduction into the LSM-tree algorithm

LSM-tree was designed for the world where the sequential read and write throughput was relatively high and the random read and write throughput was relatively low. In a nutshell, it trades increased I/O bandwidth use for reduced random I/O. LSM-tree is composed of several components (C0, C1, …, CK). The C0 component is usually kept in memory, new records are added to the C0 component until it gets full. When this happens the C0 component can be sorted by key and written to disk into C1 component. The C1 components can be merged into C2 components and so on. All this operations are sequential.

(source: http://www.cs.umb.edu/~poneil/lsmtree.pdf)

To read the data we may need to scan a range of keys inside the C0, C1, …CK components and merge the results. Each scan is a sequential read operation. To locate the begining of the range inside each component we can use binary search because each storage component is sorted by key.

The Problem

What’s the problem here, one may ask. The problem is that we need to read old data to be able to write the new! The same data gets written to disk many times during every merge. The key spaces of different storage components can overlap, thus we need to actually read the data and perform the 2-way merge and write the result! E.g. it’s quite possible that we may need to read much more than 1GB from disk (the existing storage component and the new one) to add 1GB of data to the database. In addition to this, we may need to run compaction procedure from time to time.

The typical TSDB workload looks like this, the writers are always active and the write rate is about the same (and quite high) all the time, but the readers are also active. It’s quite possible that the TSDB should send data to Grafana, and BI tools, and some background batch jobs, all at the same time. The database built on top on top of LSM-tree often needs to perform large sequential scans to read the required data. This processes can consume all read throughput of the storage device!

But the lack of the read bandwidth can stop merge and compaction procedures from keeping up with the writing process. Because we need to read data during the merge and during the compaction the writing process can be quite slow if the readers are too active. If compaction is not keeping up with writes, we may run out of disk space. If merge process is not fast enough, we may run out of file descriptors. The other possible problem is that the reads and the startup time can become slower if the merge is not keeping up because we’ll need to read from many unmerged files.

Another incarnation of the problem is ZFS compaction. Some time-series databases are relying on ZFS for compression. They write uncompressed data to disk in a format that can be easily compressed by the general purpose compression algorithms threefold. At some point, the ZFS will kick in and compress this data down to several bytes per data point but before that each data point will use 16 or even 24 bytes. I never saw this behavior in real life but I suspect that it’s quite possible that ZFS compression can be unable to keep up with the write rate if the readers are too active. In this case, we may also run out of disk space pretty soon.

Is there anything we can do about it?

Let’s see how Akumuli improves this situation. First and foremost, Akumuli doesn’t need to read anything to be able to write new data. Well, it needs to read old data if you want to insert something or update or delete (and these operations is not fully implemented yet) but if you’re adding new data with increasing timestamps there will be no reads at all.

Another property that greatly reduces the hassle between readers and writers is that all I/O is aligned. If you’re appending new data to a file unaligned, the system will read old data first. As result, the read throughput will be wasted even if your algorithm doesn’t need to read anything at all. I can recommend this fantastic paper if you want to dig deeper into the problem of the unaligned writes.

In the past the seek time of the hard drives was the main limiting factor but today the limiting factor is a bandwidth of the storage device. Modern SSD and NVMe drives may have very small latency and great random I/O throughput but the bandwidth is still limited. Because of this, the old optimization strategies (linearize all the things!) is not that effective anymore. Being able to read and write data only sequentially doesn’t hurt at all but the ability to read and write data precisely is more important because it saves the I/O bandwidth. Also, it worth to mention that this design makes the database prepared for the next gen byte-addressable non-volatile memory.

No free lunch

There are downsides of cause. There is a RUM conjecture that states that we can’t build the system that doesn’t have any read overhead, no update overhead, and no memory overhead at the same time. Akumuli sacrifices memory overhead to have great read and write performance. It needs to store some information in memory to enable this “blind” writes. As a result, the memory footprint grows with the cardinality of the data set (number of metrics). Just to be honest, you may need up to 16GB of RAM to work with 1M unique time-series, 32GB for 2M and so on. I’m planning to add an ability to disable “blind” writes to support higher cardinalities and limit memory use in the future. But this will come at the expense of decreased write throughput. This is a trade off to think about.

Storage engine design

Sat, 29 Apr 2017 18:00:00 +0000

In time-series databases the querying pattern differs from the write pattern. We usually write data in time order by updating many series every second. But querying is a different story. We usually want to read only a handful of series leaving most of the data behind. Here lies the biggest problem, which is we don’t know how data will be read and can’t put the data that will be read together close on disk. However, at least we can put each series to the separate file. Graphite does this and it somewhat works (with a fair amount of batching and in-memory caching).

Some other TSDBs use LSM-tree or similar structures. In these databases, the LSM-tree partitions the key space in the time dimension and the data points are stored in chunks in some column-oriented format that allows locating individual series quickly. Each chunk stores data points from many series (not necessarily from all of them). This design leads to high read amplification because it’s impossible to read only the data we need without reading and decompressing all the other data in the chunk. Another problem is data in the chunk is not aligned. This is a fundamental limitation; the data of the individual series in the chunk can’t be properly aligned by block boundary because everything is partitioned by time.

Akumuli’s goal is to maintain separate disk backed data structure for each series in the database and to make everything aligned on a page-sized boundary to minimize read amplification. This is not feasible with LSM-trees for many reasons:

LSM tree needs lots of RAM for memory resident part to batch up many values into one large write operation.
High write amplification. Each data element gets written to disk several times. When dealing with time-series data, this seemed unnecessary because most of the data is never changed.
Each SSTable is stored in its own file. Because of this, the database must open many files simultaneously. Storing each time-series in its own LSM-tree is not feasible. We will need a handful of files for each time-series and we might want to have millions of them.
Merging SSTables is expensive. We must read two or more files and write another one. If we want to store each time-series in its own tree, we will need to perform this operation frequently.
This design was created for spinning disks, not for modern era SSD and NVMe drives. Random wires are not that slow anymore.

This is the reason for the LSM-tree based design described above.

Meet the Numeric B+tree

Akumuli is based on novel data-structure called “Numeric B+tree”. It’s somewhat close to the LSM tree but has some nice properties that the latter doesn’t have. BTW, it’s called “numeric” only because it’s supposed to store numeric data (timestamps and values). The data structure itself can be described as B+LSM-tree. Think about LSM-tree but with B+trees instead of SSTables.

Akumuli uses timestamps as keys and keys should be inserted in increasing order (backfill can be implemented but this goes far beyond the subject of this article). This means that timestamps in each B+tree don’t overlap and that we can build full trees incrementally without node splitting.

Each NB+tree instance is a series of extents, each extent is a B+tree instance.

The first extent is a single leaf node of the B+tree stored in memory;
The second extent is a B+tree of height 2 (an inner node that stores up to 32 references to leaf nodes);
The third extent is a B+tree of height 3 (an inner node that stores up to 32 references to inner nodes that stores references to leaf nodes);
Etc.

Each extent can be seen as a single inner node that stores references to extents of the previous level (e.g. level 3 extent is a single inner node that stores references to level 2 extents, whiile level 2 extent is a single inner node that stores references to level 1 extents/leaf nodes). This property allows us to build extents easily.

The picture shows B+trees with three extents and a fan-out ratio of 4 but real B+tree used by Akumuli have a fan-out ratio of 32. Note that each B+tree can be incomplete only from the right.

Tree construction

The tree construction algorithm is simple:

Write new values to the level-1 extent (memory resident leaf node);
If level-1 extent is full - write it to disk and add reference to it to the level-2 extent;
If level-2 extent is full - write its root node to disk and add reference to it to the level-3 extent;
Etc.

On the picture above, when the level 1 extent will be full, the leaf node will be committed to disk and take its place in level 2 extent (showed by dash lines), this will make the level 2 extent full and cause it to be merged with level 3 extent (showed by dash lines as well). After that, the level 4 extent will be created and extents 1-3 will become empty.

This process is an equivalent of the SSTable merge in LSM-tree. The number of extents is expected to be small because each leaf node contains many values (it contains a variable number of values because of compression) and the fan-out ratio is 32. Let’s do some back of the envelope calculations! If leaf node can store about 1000 data points than level 2 extent will be able to store 32000 data points and level 3 will be able to store more than 10 million. Akumuli limits the number of extents in each tree by 10. This is more than enough because with 10 levels we can store a series with more than 10^16 data points, e.g this is enough to store more than 300 years of data with microsecond precision.

Here come the advantages of the NB+tree over the LSM-tree:

Memory resident part of the tree is small (single leaf node + several inner nodes);
Many trees can live in one file and blocks of different trees can be interleaved on the storage level;
Merge is very efficient. It’s enough to just add an address of the root node of the one extent to the other extent; no need to read the data and merge anything.
We can store some information inside inner nodes to speed up aggregation and resampling of the time-series without maintaining some external data-structures (rollups) or indexes.
Parallel operation. It is possible to write to the database from many writer threads (and Akumuli is doing this).
Good for modern SSDs. All writes are aligned and performed in parallel. All tree nodes have the same size.

This data-structure allows Akumuli to maintain separate NB+tree instance for every time-series in the database. The primary disadvantage of this design is the complexity of the crash recovery process. It’s impossible to maintain WAL or command log per NB+tree instance. Akumuli uses a different approach to crash recovery that’s not discussed in this article but you can find more details about it in this article.

Ability to build separate NB+tree instance for every time-series is truly important. It enables queries in column-oriented fashion and, at the same time, it enables really fast parallel writes with very small write amplification. My recent experiment showed that it can write about 16 million data points per second on a c3.8xlarge instance.

Update

Hacker News discussion.

Benchmarking Akumuli on 32-core machine

Fri, 10 Mar 2017 18:00:00 +0000

Recently I tested Akumulil on the m3.2xlarge EC2 instance. Write throughput was around 4.5 million elements/second. This number might look unrealistic at first but in fact, that’s less than 20MB/s of disk write throughput because each data point is tiny (less than five bytes in that particular case) and all data is compressed in real time.

Next step was to try it on a bigger machine. I chose c3.8xlarge EC2 instance with 32-core Intel Xeon E5-2680 v2 and SSD. This is what I did:

I created test data beforehand (32 gzip archives, 16GB compressed) and uploaded it to S3.
Then I created 4 m3.xlarge EC2 instances to generate load.
From each m3.xlarge instance, I downloaded the test data.
Then I started a single c3.8xlarge instance, build and installed Akumuli, set TCP.pool_size parameter to 32, created the database and started the network daemon.
After that I downloaded and edited the run.sh script on each m3.xlarge box. I set TARGET_HOST variable correctly and decreased the number of archives from 32 to 8 (each host had been sending the unique subset of test data).
Then I started run.sh on all machines simultaneously using parallel-ssh.

During the test run, each m3.xlarge instance had been sending data in parallel using 8 threads (32 threads total). All this data was written to disk in parallel on the c3.8xlarge instance. This is how everything looked in htop:

The dataset contained 2764832000 data points in 32000 series. All nodes finished sending data in less than 3 minutes and write throughput was above 16 million elements/second. Resulting database size was 11GB. SSD drive was underutilized, it can write far more than 64MB/s that was demanded by this test run.

This puts Akumuli in the same ballpark as BTrDB. As far as I know, these are the only two open source time-series databases that can write data in parallel. All other solutions still struggle with single writer solutions.

Understanding Akumuli Performance

Mon, 13 Feb 2017 18:00:00 +0000

Akumuli was designed with performance in mind from the very beginning. I set the lower bound for the write throughput at the 1M writes/second level as one of the project goals. Every version so far delivers this performance, that’s why this number is mentioned on the project page. But this is only a lower bound. It would be interesting to see what level of performance is achievable with today’s hardware!

To answer this question I tested Akumuli on AWS and it managed to sustain 4.5 million write operations per second on a dedicated m3.2xlarge node. This isn’t a peak but steady write throughput over the network. And my tests showed that it scales almost linearly on a multicore machine.

Methodology

Akumuli has a configuration option called TCP.pool_size. It controls the server side parallelism. By default, this option is set to one. But it can be increased on multicore machine. I ran the benchmark with different TCP.pool_size values on the eight core machine and here are the results:

Number of threads	Throughput datapoints/sec
1	693 984
4	2 433 831
8	4 547 421

The input was constructed using this script. All test data was generated beforehand and was divided into eight independent archives (each archive had its own set of time-series). Each archive contained exactly 86401000 data elements. All eight archives had around 691M data elements (3.5G compressed). I started Akumuli with different TCP.pool_size values and measured the time it took to write all the data. After each run, the resulting database size was 2.7GB (4.2 bytes per element).

Single m3.xlarge instance was used to generate load. I had been using 8 writer processes working in parallel each process transmitting its own set of series to feed the data to akumulid daemon.

What makes Akumuli so fast?

Storage in Akumuli is based on append-only B+tree. Each series is represented using a separate B+tree instance and can be modified independently. All B+trees reside in the same file.

Code in write-path is based on the observation that different TCP-sessions usually write data to the different time-series. When session writes something to the series first time the corresponding B+tree that stores series data get assigned to that session. After that, the session will be able to write data to this B+tree without any synchronization. If B+tree is already owned by another thread, additional synchronization with the owner thread will be needed.

Final notes

Using a single-node instance of Akumuli running on the m3.2xlarge instance I managed to write 4.5M data points / second. This is not a high-end machine by any means. I’m looking forward to running this test on more performant hardware and see the results. That would be very interesting to try.

Time-series compression (part 2)

Sun, 05 Feb 2017 18:00:00 +0000

In the previous article I discussed timestamps compression, now it’s time to talk about floating point data compression. This problem is not new, there are some good papers about it, e.g. Gorilla paper, and also this, and this one too. None of them is a good fit for Akumuli because the algorithms described in these papers are too specialized. The paper authors usually have a very specific dataset in mind. For example, Gorilla is optimized for slowly changed time-series and for integer time-series, and FCM algorithm will work best for scientific data.

So, I had to come up with a new algorithm. Simple delta encoding wouldn’t work at all. If you subtract one floating point number from the other, you won’t end up with the number that can be represented using a smaller number of bits. The solution here is to use bitwise XOR and get a binary diff instead of arithmetic delta.

Let’s get a closer look at IEEE 754 double-precision floating point number representation:

Fractional part occupies low 52 bits and sign and mantissa are located in high 12 bits. This means that if two numbers have close absolute values their high bits will be the same so when we XOR this values together, we will get a lot of leading zeroes in the result. Example: binary representation of 1.0 is 3ff0 0000 0000 0000 and binary representation of 1.0000000000000004 is 3ff0 0000 0000 0002. We will get 0000 0000 0000 0002 by XOR-ing this two values. This value occupies only two bits.

But some numbers will have very small (if any) number of leading zeroes when XOR-ed. E.g. 1.0 is represented as 3ff0 0000 0000 0000 and 2.0 is 4000 0000 0000 0000. By XOR-ing this two numbers we will get this result 7ff0 0000 0000 0000. This happens a lot in practice because many sources generate integers (e.g. number of bytes process is using, number of packets sent by the peer, etc).

So, we need to XOR not any but close values together. To achieve this Akumuli uses predictive coding. The idea is that we can try to predict next value and XOR it with the actual one. Basically, we will store prediction error. The predictor can be stateful, it can store some context information about the predicted signal. The signal usually is not totally random but stationary thus it can be predicted with some accuracy.

If our predictor is good prediction error will be small and we will be able to store it using a small number of bits. But what predictors should be used?

I’ve evaluated three predictors:

Last value predictor.
Finite Context Method (FCM) predictor.
Differential FCM predictor.

(Actually, I’ve evaluated compositions of different predictors, but this is too much for the blog post)

Last value predictor (or delta modulator) assumes that the next value will be the same as the previous one. So, to use this predictor we can just XOR adjacent values of the array and we’re done.

FCM predictor is more complex. It predicts next value based on a finite number of preceding values. FCM predictor contains fixed sized table. This table maps contexts to values that were observed after particular contexts. The data structure is very simple:

struct FCM {
  double table[TABLE_SIZE];
  int last_hash = 0;
};

The update procedure should be used to add new values to the table:

double predict_next(FCM* fcm) {
  return fcm->table[fcm->last_hash];
}  

To understand how it works we should know how these values are calculated:

void update(FCM* fcm, double value) {
  fcm->table[fcm->last_hash] = value;
  fcm->last_hash = hash(value) % TABLE_SIZE;
}

To understand how it works we can imagine how FCM predictor of size two will be dealing with this input sequence: [-1, 1, -1, 1, -1, 1].

Obviously, last value predictor will fail here. Let’s define the hash function first. We need to extract some context from each floating point number, for FCM predictor of size two we can use sign bit:

int hash(double value) 
  if (value < 0.0) return 1;
  return 0;
}

Now let’s look how predictor will do its job:

FCM initialized, table == [0, 0], last_hash == 0;
predict_next(fcm) returned 0;
update(fcm, -1), table == [-1, 0], last_hash == 1;
predict_next(fcm) returned 0;
update_next(fcm, 1), table == [-1, 1], last_hash == 0;
predict_next(fcm) returned -1;
update_next(fcm, -1), table == [-1, 1], last_hash == 1;
predict_next(fcm) returned 1;
udpate_next(fcm, 1), table == [-1, 1], last_hash == 0;
predict_next(fcm) returned -1;
…

So, FCM predictor can learn simple patterns like that. Basically, it remembers that some context (hash value) preceded some value and then when it meets this context it assumes that the value will follow. All we need to do is to give it some memory and proper hash function. It will be able to predict more complex patterns using more memory.

Famous Gorilla paper uses the algorithm that can be seen as a special case of the FCM compression with the table of size one. We already know it as “last value predictor”.

In Akumuli I’m using this hash function:

(last_hash << 5) ^ (value >> 50)

It extracts the sign, exponent, and two mantissa bits out of floating point number and blends it with some bits from the previous hash value. Blending previous hash bits is very important because it makes context depend on several adjacent values, not only the last.

DFCM works almost the same as FCM, but it works with differences, not the actual values.

I created this script to evaluate different predictors in different situations. It runs different predictors on several classes of input. Some of this classes are:

Random walk;
Steady growth trend;
Pattern;

Random walk

This class of input is very important because it mimics a stationary process. This is the most important and general case. DFCM certainly has some advantage here, not overwhelming but noticeable.

This graph shows number of bits needed to store each value using different predictors. X axis is a value index and Y axis is a number of bits needed to represent prediction error so, the smaller the value the better compression.

Pattern

Sometimes we need to represent the state of the system using discrete values (e.g. use 1 and 0 to represent on and off states). In many cases, this discrete values will form stable patterns because each value represents a state of some finite state machine and the state machine often goes through the same sequence of state transitions over and over again. Predictor should learn this patterns.

This is how three predictors compress repeating pattern composed of three discrete values:

After just six iterations DFCM was able to predict the input perfectly. Both FCM and last value predictors performed poorly.

Steady growth trend

Some data sources are producing time-series with increasing values (e.g. number of packets sent, electricity consumption, etc). This is how our predictors performed on simple linear trend:

After several iterations, DFCM predictor started to predict input perfectly. Other two predictors couldn’t comply. We can see the same picture with integer time-series:

What if the trend will be random?

You can see that DFCM still outperforms other predictors but there are some spikes. We definitely will need more space to store this.

Final thoughts

As result of this evaluation, I’ve chosen DFCM predictor for Akumuli. It performs relatively good on monitoring data as well as scientific data. You can see that compression ratio depends on dataset heavily and predictive compression algorithm do make a difference in some cases. This article covers only small part of the algorithm, but you can check this design document for more details.