Log files are like an internet of things (IoT) for IT. Log data is vital for all IT organisations – it tells us everything that has happened in our environment and is a valuable source of information for troubleshooting issues, such as in backup.
Often, it lies neglected until needed, but for some organisations log data is the lifeblood of artificial intelligence (AI) and analytics at the core of their business.
It is these log file data specialists that are targeted by Quickwit, which provides search and indexing software for petabyte-scale log datasets. It claims its point solution is 10 times more cost-efficient than similar functionality in existing products such as Elasticsearch, Splunk, Datadog and Google Chronicle.
Quickwit targets customers where often the most important data is log data. For them, the key datasets are not customers, products or transactions, but the activity logged in IT systems.
Such customers – centred on security and observability – run AI and analytics on huge log files for use cases that can range from security workloads to specialist provision of search tools for dark net content. Also targeted are financial services scenarios, where transactions are so frequent that log files form the vast bulk of data.
Here lies Quickwit’s key advantage for the customer, said co-founder Francois Massot at a recent IT Press Tour event in Rome. Namely, Quickwit aims at workloads where existing solutions are expensive and complex, and where indexing can be very slow.
He points to customers where the deployment of Quickwit has slashed search and indexing times. That allows much larger datasets to be retained, enables more comprehensive analytics to be carried out, and for new products to be developed.
Quickwit is built in the Rust programming language and uses the Tantivy search engine, which Massot said is the fastest open source search engine library available.
He said Quickwit can make an impression in the market, mostly due to its point solution advantages compared to the size and established nature of its competition.
“The companies are quite big so it’s harder for them to innovate,” said Massot. “Their products are already developed and have lots of features, so it’s hard for them to keep up with innovation.”
Massot said Quickwit is deployed as greenfield or in cases where it replaces existing search and indexing solutions.
“Customers might have Loki or ElasticSearch, but they’re innovating and they don’t have an event store and want to add new products,” he said.
Quickwit generally sells to customers that want to store data on their own hardware because of security concerns.
Massot cited a number of customers and the benefits they had gained. One went from OpenSearch to Quickwit and cut the number of CPUs needed by five, halved storage capacity and was able to retain 10 times the amount of data.
Another, US-based cloud provider Fly.io, was able to reduce by 20 times the number of compute nodes needed to handle large-scale, multi-tenant log files and boost data retention from three days to 30. Another customer, OwlyScan, uses Quickwit to provide a dark net search tool product.
Quickwit is provided by direct sales and partnerships with cloud providers, as well as being available as a community version via its open source distribution. It claims “more than 200 customers”.
Roadmap items for 2024 and beyond include enabling seamless migration from OpenSearch (Q2), Splunk SPL-like pipe-based queries (Q3) and a new storage engine for time series data (Q4 and 2025).