thatDot

Scaling Quine Streaming Graph to Process 1 Million Events/Second

September 27, 2022 · by thatDot · 13 min read

Note: If you want to reproduce this test, we have published the test details on Github so that you can understand and run it yourself.

Solving the Unsolvable: Graph that Scales Past 1 Million Events/Second

This is not a blog post about benchmarking Quine streaming graph. This is a post with an operational focus that explains how Quine solves the previously unsolvable: scaling graph data processing past a million events per second. In conventional terms, that means millions of simultaneous writes and multi-node graph traversals per second -- an unprecedented achievement.

The tests this post covers also demonstrate Quine Enterprise's resilience in the face of common failure scenarios.

Most importantly, this blog is about the new use cases for graph this performance makes possible. Finding relationships within categorical data is graph's strongpoint. Doing so at scale, as Quine now makes possible, has significant implications for cyber security, fraud detection, observability, logistics, e-commerce, and really any use case graph is both well-suited for and which must process high velocity data in real time.

Scaling to 1M+ events/second and demonstrating recovery from various failure scenarios

tl;dr

Our tests delivered the following results:

What is Quine?

For those of you new to Quine, the simplest way to describe it is “real-time graph ETL”.

Quine streaming graph combines the graph data structure and persistence of graph databases (e.g. Neo4J) with the streaming properties of systems like Flink. Drop Quine into a streaming system between two Apache Kafka or Kinesis instances and start materializing and querying your real-time events as a graph.

There’s a lot more to Quine of course, so if you are interested in how it works – asynchronous actor model, caching strategies, "all nodes exist," and more – check out the docs. In particular, see the comparison to standard database systems.

Quine Operational Profiling

The goal of this test is to demonstrate a high-volume of sustained ingest, that is resilient to cluster node failure in both Quine and the persister using commodity infrastructure, and to share performance results along with details of the test for those interested in either reproducing results or running Quine in production.

Infrastructure Used

Quine Cluster

Cassandra Persistor Cluster

Kafka

Infrastructure Update

Our initial testing, provisioned 141 c2-standard-30 hosts. However, as we proceeded with further testing, we made an important discovery. By deploying a higher number of smaller n2-custom-8-16384 hosts, we achieved the same overall performance while significantly reducing our monthly costs.

Using 191 smaller hosts proved to be a more cost-effective solution compared to the initial setup with 141 larger hosts. This adjustment allows us to maintain optimal performance while ensuring budget efficiency.

Regarding the Cassandra persistor layer’s settings, we set a TTL of 15 minutes and replication factor of 1 in order to manage quota limits and spending on cloud infrastructure. This does not fit every possible use case, but it is fairly common. Other scenarios which are more data-storage oriented will often increase the replication factor and/or TTL. In those variations, maintaining the 1 million events/sec processing rate would require increasing the number of Cassandra hosts or disk storage, both of which are budgetary concerns more than technical concerns.

This cluster configuration was meant to demonstrate that high-volume graph processing is possible. In a later post we'll describe how to optimize the cluster to achieve these results and minimize infrastructure costs.

The Test

The plan is set out below, with each action labeled and the results explained. Events are clearly marked by sequence # on the Grafana screen grabs below the table.

A few notes on the test:

Sequence Actions, Expected Results, and Actual Results Overview

Sequence 1

Sequence 2

Sequence 3

Sequence 4

Sequence 5

Sequence 6

Sequence 7

Sequence 8

Sequence 9

Sequence 10

The Results

A diagram showing sustained event processing of 1 million events per second.

Figure 1: Overall Ingest Rate

As you can see from the overall ingest rate results:

We observed that a persistor node high-latency event (7) has a more marked impact on performance than either a Quine node failure (5) or an outright failure of a persistor node (8). In the case of a clear failure, Kubernetes is quick to replace the node, allowing ingest to resume. In cases when a persistence node state is non-responsive but not clearly down, Quine’s response is to back pressure ingest until the node is recovered.

An alternate variation on this test could use more persistor machines to stabilize ingest rates during maintenance events.

A per-host diagram demonstrates when different operational events impacted performance of both individual hosts and the overall cluster.

Figure 2: Per Host Ingest Rate - Quine Only

The individual Quine node ingest graphs indicate when individual nodes are offline and reinforces the observation that Quine Enterprise’s cluster resilience allows for smooth operation during high-volume ingest, even in the face of a Quine node shut down or failure. Quine’s overall performance, and hence an area of operational focus for anyone planning a production deployment, more closely conforms with persistor performance.

Cassandra Latency events line up with overall decreases in cluster performance.

Figure 3: Cassandra Persistor Latency

The median query latency for the Cassandra cluster during this test was <1 ms. Even during/following persistor shutdown (8) or node failure (7), cluster latency stayed < 1.5 ms. Events at (1), (5), and (8), all reflect increased latency for single nodes.

Standing Queries and 1 Million 4-node traversals per second

Figure 4: Standing Query Results (events emitted/second)

The purpose of running any complex event processor, Quine included, is in detecting and acting on high-value events in real time. This could mean detecting indications of a cyber attack, or video stream buffering, or identifying e-commerce upsell opportunities at check out. This is where Quine really excels.

Standing queries are a unique feature of Quine. They monitor streams for specified patterns, maintaining partial matches, and executing user-specified actions the instant a full match is made. Actions can include anything from updating the graph itself by creating new nodes or edges, writing results out to Kafka (or Kinesis, or posting results to a webhook).

In this test, Quine standing queries monitored for specific 4-node patterns requiring a 4-node traversal every time an event was ingested. Traditional graph databases slow down ingest when performing multi-node traversal. Not Quine. Quine’s ability to sustain high-speed data ingest together with simultaneous graph analysis is a revolutionary new capability. Not only did Quine ingest more than 1,000,000 events per second, it analyzed all that data in real-time to find more than 20,000 matches per second for complex graph patterns. This is a whole new world!

Summary Results

Resource Usage and Performance Metrics Overview

Quine Host Metrics

Cassandra Persistor Node Metrics

Performance Metrics

Why Quine Hitting 1 Million Events/Sec Matters

Since its release in 2007 at the start of the NoSQL revolution, Neo4J have proven conclusively the value of graph to connect and find complex patterns in categorical data.

The graph data model is indispensable to everything from fraud detection to network observability to cybersecurity. It is used for recommendation engines, logistics, and XDR/EDR.

But not long after NoSQL hit the scene, Kafka kicked off the movement toward real-time event processing. Soon, event processors like Flink, Spark Streaming and ksqlDB brought the ability to process live streams. These systems relied on less-expressive key-value stores or slower document and relational databases to save intermediate data.

Quine is the graph analog and is important because now you can do what graph is really good at -- finding complex patterns across multiple streams of data using not just numerical but categorical data.

Quine makes all the great graph use cases viable at high volumes and in real time.

Next Steps

If you want help planning your own test, or you would like to try the Quine Enterprise, please contact us. You can also read more about Streaming Graph here.

Or you can start learning about Quine now by visiting the Quine open source project. We have a Slack channel where folks can ask questions and we are always up for a call.