Using SIMD acceleration in rust to create the world’s fastest tac

NeoSmart Technologies’ open source (MIT-licensed), cross-platform tac (written in rust) has been updated to version 2.0 (GitHub link). This is a major release with enormous performance improvements under-the-hood, primarily featuring handwritten SIMD acceleration of the core logic that allows it to process input up to three times faster than its GNU Coreutils counterpart. (For those that aren’t familiar with it, tac is a command-line utility to reverse the contents of a file.)

This release has been a long time in the making. I had the initial idea of utilizing vectorized instructions to speed up the core line ending search logic during the initial development of tac in 2017, but was put off by the complete lack of stabilized SIMD support in rust at the time. In 2019, after attempting to process a few log files – each of which were over 100 GiB – I decided to revisit the subject and implemented a proof-of-concept shortly thereafter… and that’s when things stalled for a number of reasons.

Continue reading

A high-performance, cross-platform tac rewrite

If you haven’t heard of tac, it’s a pretty nifty command-line utility that ships with the GNU utils and it’s used to print a file backwards, line-by-line. It’s especially useful when analyzing things like log files, and judicious use of tac can speed up commands considerably.

Take the example of a 30GiB webserver access log and you want to see the last request to a certain resource or that triggered a particular HTTP status code. You could run the following to get the last such request… which would take quite awhile on anything larger than a few hundred MiB:

> egrep "GET /path/to/resource " access.log | tail -n1

Or you could be smart about it and use tac instead, and not even have time to blink before the result comes back:

> tac access.log | egrep "GET /path/to/resource " | head -n1

Continue reading