Performant Pagination at Scale, Improved Querying and User Docs, Advanced Recipes
It is hard to believe we released Quine 1.2.0 only six weeks ago, especially when I look at the work that has gone into not just Quine but also documentation, how-to blogs and example recipes. Indeed, 1.3.0 cements a pattern of releases made up of a few features needed to achieve performance at scale and loads of smaller usability improvements that has emerged since we released Quine as an open source project in February.
Additions to Quine included vastly improved pagination performance inside of our Cypher compiler, overhauled the API documentation, making journals a default when running recipes, improved Cypher query support, and a number of small but consequential changes to the system’s logging behavior.
In addition, we’ve migrated the documentation to its own site to make it easier to make community contributions and keep docs in sync with releases, added three new recipes and made substantial updates to one of the favorites.
The common theme throughout: usability and performance.
Pagination in Quine Streaming Graph
As part of our work to make all aspects of the system perform predictably and well at throughput rates of hundreds of thousands or even millions of events per second, we have undertaken some plumbing upgrades.
To give you an idea of the engineering involved, check out this blog post about the three most common pagination approaches (or I can save you time and tell you it explains page, point, and keySet-based pagination). We combined aspects of all three in our approach.
The other notable work focused on usability and community enablement.
Quine and API documentation plus Improved Usability
We switched to the Stoplight Elements framework to make API documentation easier to access and migrated from quine.io/docs to docs.quine.io. Not huge changes in themselves, but together they ensure docs are more accessible to the community to modify and never lag releases.
Ingest Streams from Kafka and other Sources
We also completed five blog posts on ingesting streams, ranging from simple CSV files to internet feeds to Kafka integration.
- Real-time Graph Analytics for Kafka Streams with Quine
- Building a Quine Streaming Graph: Ingest Streams
- Ingesting data from the Internet into Quine Streaming Graph
- Ingesting From Multiple Data Sources into Quine Streaming Graph
- Ingest and Analyze Log Files Using Streaming Graph
Live event stream, log, and network observability recipes
And of course when we wrote an explainer on ingesting and processing log files, we couldn’t resist a recipe that uses Quine logs as the source. We all know that consuming, parsing, and visualizing Java log output is a huge challenge, one that lacks a widely available solution. We think Quine might be an answer. Use the Quine Log Recipe as a baseline, then modify the regular expression inside the ingest stream Cypher query to fit your logs.
In addition to the Quine Java log ingest recipe, we’ve created a recipe showing how to ingest and build a streaming graph from a feed of IMDB movie data. (For anyone really interested in log processing, there’s also an Apache web logs analytics recipe). Rounding out the trio of new recipes is a fun one: Ethan’s Pi Day recipe using Quine to calculate Pi using Liebniz’s formula.
On the topic of observability and root cause analysis, the CDN Cache Efficiency recipe got a major update:
- Moved shaping the graph from standing queries into the ingest stream.
- Updated code to reflect Cypher best practices.
- Added quick queries to perform efficiency calculations.
- Optimized the manifestation of nodes.
- Added client device nodes.
- Increased the data sample size
Quine Synthetic Data Generator
With Quine v1.3.0 we also introduced a powerful series of built-in synthetic data Cypher functions. The synthetic data functions can be used within ingest streams to create booleans, bytes, floats, integers, strings, or nodes. This allows you to generate streaming synthetic data that can be used for testing or development purposes.
Search for `gen.` to check out how to use the functions on the Cypher Functions page of docs.quine.io.
Quine is open source if you want to explore standing queries for yourself using your own data. Download a precompiled version or build it yourself from the codebase from the Quine Github codebase.
Have a question, suggestion, or improvement? I welcome your feedback! Please drop into Quine Slack and let me know. I'm always happy to discuss Quine or answer questions.