Scaling Quine Streaming Graph to Process 1 Million Events/Second

Learn how Quine achieves groundbreaking performance for real-time complex event processing and how you can reproduce the results.

by
Ryan Wright
Quine
8
min read

Note: If you want to reproduce this test, we have published the test details on Github so that you can understand and run it yourself.

Solving the Unsolvable: Graph that Scales Past 1 Million Events/Second

This is not a blog post about benchmarking Quine streaming graph. This is a post with an operational focus that explains how Quine solves the previously unsolvable: scaling graph data processing past a million events per second. In conventional terms, that means millions of simultaneous writes and multi-node graph traversals per second -- an unprecedented achievement.

The tests this post covers also demonstrate Quine Enterprise's resilience in the face of common failure scenarios.

Most importantly, this blog is about the new use cases for graph this performance makes possible. Finding relationships within categorical data is graph's strongpoint. Doing so at scale, as Quine now makes possible, has significant implications for cyber security, fraud detection, observability, logistics, e-commerce, and really any use case graph is both well-suited for and which must process high velocity data in real time.

Scaling to 1M+ events/second and demonstrating recovery from various failure scenarios [expand image]

tl;dr

Our tests delivered the following results:

  • 1M events/second processed for a 2 hour period
    • 1M+ writes per second
    • 1M 4-node graph traversals (reads) per second
    • 21K results (4-node pattern matches) emitted per second
  • 140 commodity hosts plus 1 hot spare running Quine Enterprise
  • 66 storage hosts using Apache Cassandra persistor
  • 3 hosts for Apache Kafka

What is Quine?

For those of you new to Quine, the simplest way to describe it is “real-time graph ETL”.

Quine streaming graph combines the graph data structure and persistence of graph databases (e.g. Neo4J) with the streaming properties of systems like Flink. Drop Quine into a streaming system between two Apache Kafka or Kinesis instances and start materializing and querying your real-time events as a graph.

There’s a lot more to Quine of course, so if you are interested in how it works – asynchronous actor model, caching strategies, etc,. – check out our technical whitepaper.

Quine Operational Profiling

The goal of this test is to demonstrate a high-volume of sustained ingest, that is resilient to cluster node failure in both Quine and the persister using commodity infrastructure, and to share performance results along with details of the test for those interested in either reproducing results or running Quine in production.

Infrastructure Used

Component # of Hosts Host Types Estimated Cost
Quine Cluster 191
  • n2-custom-8-16384 8 vCPU 16GB Intel Cascade Lake
  • Max heap for JVM set to 12GB
  • 140 cluster size w/ 1 hot spare
$28.73 /hour
Cassandra Persistor Cluster 48
  • n2d-custom-16-131072 16 vCPU 128GB AMD Rome
  • 1 x 375 GB local SSD each
  • durable_writes=false
  • TTL=15 minutes on snapshots (to control disk costs in testing) and journals tables
$21.07 /hour
Kafka 3
  • n2-standard-4 (4 vCPU, 16 GB RAM)
  • Preloaded with 8 billion events (sufficient for a sustained 2 hour ingest at 1 million events per second)
  • 420 partitions
Part of the data pipeline, not estimated

Infrastructure Update

Our initial testing, provisioned 141 c2-standard-30 hosts. However, as we proceeded with further testing, we made an important discovery. By deploying a higher number of smaller n2-custom-8-16384 hosts, we achieved the same overall performance while significantly reducing our monthly costs.

Using 191 smaller hosts proved to be a more cost-effective solution compared to the initial setup with 141 larger hosts. This adjustment allows us to maintain optimal performance while ensuring budget efficiency.

Regarding the Cassandra persistor layer’s settings, we set a TTL of 15 minutes and replication factor of 1 in order to manage quota limits and spending on cloud infrastructure. This does not fit every possible use case, but it is fairly common. Other scenarios which are more data-storage oriented will often increase the replication factor and/or TTL. In those variations, maintaining the 1 million events/sec processing rate would require increasing the number of Cassandra hosts or disk storage, both of which are budgetary concerns more than technical concerns.

This cluster configuration was meant to demonstrate that high-volume graph processing is possible. In a later post we'll describe how to optimize the cluster to achieve these results and minimize infrastructure costs.

The Test

The plan is set out below, with each action labeled and the results explained. Events are clearly marked by sequence # on the Grafana screen grabs below the table.

A few notes on the test:

  • A script is used to generate events
  • Host failures are manually triggered.
  • We used Grafana for the results (and screenshots).
  • We pre-loaded Kafka with enough events to sustain one million events/second for two hours.
  • A Cassandra cluster is used for persistent data storage. The Cassandra cluster is not over-provisioned to accommodate compaction intentionally (a common strategy) so that the effects of database maintenance on the ingest rate can be demonstrated.
  • The cluster is run in a Kubernetes environment
Seq # Actions Expected Results Actual Results
1 Start the Quine cluster and begin ingest from Kafka The ingest rate increase and settle at or above 1 million events per second Observed
2 Let Quine run for 40 minutes to establish a stable baseline Quine does not fail and maintains a baseline ingest rate at or above 1 million events per second. Observed
3 Kill a Quine host Quine ingest is not significantly impacted. The hot spare steps in to recover quickly, and Kubernetes replaces the killed host which becomes a new hot spare. Observed at 17:47. No impact to ingest rate. The hot spare recovered quickly and ingest was not impacted.
4 Persistor Maintenance Cassandra regularly performs maintenance, Quine experiences this as increased latency and should backpressure the ingest to maintain stability during database maintenance. From 17:55 - 18:15 the ingest rate is reduced as a corresponding increase in latency is measured above 1ms across all nodes from the Cassandra persistor.
5 Kill two Quine hosts Observe the following sequence: hot spare recovers one host, whole cluster suspends ingest due to being degraded, Kubernetes replaces killed hosts, first replaced host recovers the cluster, and the second replaced host becomes the new hot spare. Observed from 18:18 - 18:25. Due to Kubernetes the impact was not visible. However, the expected sequence was confirmed in the logs.
6 Stop and resume a Quine host for about 1 minute to inject high latency Quine detects the host is no longer available, boots it from the cluster, and hot spare steps in to recover. When the rejected host resumes, it learns it was removed from the cluster, so it shuts down, is restarted by Kubernetes, and to become the new hot spare Observed from 18:41 - 18:46. No impact to ingest rate as the back pressured ingest was for a single host in the cluster, and the recovery happened quickly.
7 Stop and resume a Cassandra persistor host for about 1 minute to inject high latency Quine back pressures ingest until Cassandra persistor has recovered Observed from 18:47 - 18:54. Due to replication factor = 1, ingest was impacted until Cassandra persistor recovered. Then ingest resumed to > 1M e/s.
8 Kill a Cassandra persistor host Quine suspends ingest until Cassandra persistor recovers with a new host Observed from 18:54 - 19:10. The host was recovered quickly due to kubernetes, and ingest briefly recovered to 1M e/s by 18:58 (only a few minutes).
9 Persistor Maintenance Cassandra regularly performs maintenance, Quine experiences this as increased latency and should backpressure the ingest to maintain stability during database maintenance. From 17:55 - 18:15 the ingest rate is reduced as a corresponding increase in latency is measured above 1ms across all nodes from the Cassandra persistor.
10 Let Quine consume the remaining Kafka stream Observe the Quine hosts drop to zero events per second (not all at once) Observed from 19:10 - 19:35. Around the time Cassandra persistor latency was returning to 1ms, and ingest returned to 1M e/s, the pre-loaded ingest stream began to become exhausted on some hosts. For the following 20 minutes hosts exhausted their partitions in the stream.

The Results

A diagram showing sustained event processing of 1 million events per second.
Figure 1: Overall Ingest Rate [expand image]

As you can see from the overall ingest rate results:

  •  #1 shows an initial peak of 1.25M events/sec
  • #2 Quine settles into a steady ingest rate > 1 million events/sec
  • #3 Quine recovers nicely after killing single node
  • Quine settles into a steady ingest rate > 1 million events/sec
  • #s 4 and 9 show Cassandra maintenance event (see Cassandra Latency - Figure 3)
  • #5 Quine has no problem with two-node failure events.

We observed that a persistor node high-latency event (7) has a more marked impact on performance than either a Quine node failure (5) or an outright failure of a persistor node (8). In the case of a clear failure, Kubernetes is quick to replace the node, allowing ingest to resume. In cases when a persistence node state is non-responsive but not clearly down, Quine’s response is to back pressure ingest until the node is recovered.

An alternate variation on this test could use more persistor machines to stabilize ingest rates during maintenance events.

A per-host diagram demonstrates when different operational events impacted performance of both individual hosts and the overall cluster.
Figure 2: Per Host Ingest Rate - Quine Only [expand image]

The individual Quine node ingest graphs indicate when individual nodes are offline and reinforces the observation that Quine Enterprise’s cluster resilience allows for smooth operation during high-volume ingest, even in the face of a Quine node shut down or failure. Quine’s overall performance, and hence an area of operational focus for anyone planning a production deployment, more closely conforms with persistor performance.

Cassandra Latency events line up with overall decreases in cluster performance.
Figure 3: Cassandra Persistor Latency [expand image]

The median query latency for the Cassandra cluster during this test was <1 ms. Even during/following persistor shutdown (8) or node failure (7), cluster latency stayed < 1.5 ms. Events at (1), (5), and (8), all reflect increased latency for single nodes.

Standing Queries and 1 Million 4-node traversals per second

Figure 4: Standing Query Results (events emitted/second) [expand image]

The purpose of running any complex event processor, Quine included, is in detecting and acting on high-value events in real time. This could mean detecting indications of a cyber attack, or video stream buffering, or identifying e-commerce upsell opportunities at check out. This is where Quine really excels.

Standing queries are a unique feature of Quine. They monitor streams for specified patterns, maintaining partial matches, and executing user-specified actions the instant a full match is made. Actions can include anything from updating the graph itself by creating new nodes or edges, writing results out to Kafka (or Kinesis, or posting results to a webhook).

In this test, Quine standing queries monitored for specific 4-node patterns requiring a 4-node traversal every time an event was ingested. Traditional graph databases slow down ingest when performing multi-node traversal. Not Quine. Quine’s ability to sustain high-speed data ingest together with simultaneous graph analysis is a revolutionary new capability. Not only did Quine ingest more than 1,000,000 events per second, it analyzed all that data in real-time to find more than 20,000 matches per second for complex graph patterns. This is a whole new world!

Summary Results

Description Value
GB RAM used per Quine host 12 GB
CPU% used per Quine host 60%
CPU% used per Cassandra persistor node 80%+
Overall Ingest event records/second >1,000,000
Standing Query results/s  21,000/sec
Average Persistor Latency  1 ms
Data Storage Disk Space Used (Cassandra) 70GB/host

Why Quine Hitting 1 Million Events/Sec Matters

Since its release in 2007 at the start of the NoSQL revolution, Neo4J have proven conclusively the value of graph to connect and find complex patterns in categorical data.

The graph data model is indispensable to everything from fraud detection to network observability to cybersecurity. It is used for recommendation engines, logistics, and XDR/EDR.

But not long after NoSQL hit the scene, Kafka kicked off the movement toward real-time event processing. Soon, event processors like Flink, Spark Streaming and ksqlDB brought the ability to process live streams. These systems relied on less-expressive key-value stores or slower document and relational databases to save intermediate data.

Quine is the graph analog and is important because now you can do what graph is really good at -- finding complex patterns across multiple streams of data using not just numerical but categorical data.

Quine makes all the great graph use cases viable at high volumes and in real time.

Next Steps

If you want help planning your own test, or you would like to try the Quine Enterprise, please contact us. You can also read more about Quine Enterprise here.

Or you can start learning about Quine now by visiting the Quine open source project. We have a Slack channel where folks can ask questions and we are always up for a call.

by
Ryan Wright
Thanks! Your email has been signed up.
Sorry, an error occurred. Please try signing up later.

Read more like this: