Hey — I’m going to walk you through how Xorq takes plain ol’ Python and turns it into reusable, optimizable, and cached compute artifacts.
The way we accomplish this is in the form of a compute catalog that is both multi-engine and portable and is built on open standards like Arrow Flight.
We’ve had catalogs for data at rest — systems like Iceberg handle that well.
In fact, Iceberg’s REST catalog as a standard.
But for data in motion — the actual transformations, features, and ML steps — we’ve been missing an equivalent – a open expression format that can be easily ported across compute and storage engines.
Reusability, especially in the context of multiple engines, inside the enterprise is still a pipe dream.
Behind the scenes, it builds Python expressions into a YAML format and exposes partial expressions — so each step can be composed, shared, and served on demand with built-in lineage and observability, as Arrow Flight endpoints.
