Universal Data Processing Engine (UDPE) is a software layer created by DataPelago that integrates the compute core directly into the data storage system (SDS/Storage).
Previously, data had to be mandatory «offloaded» from the lake, cleansed in a separate cluster, and only then sent to the GPU. UDPE makes this offloading unnecessary. It acts as a universal translator that allows application software to use any available accelerators — whether CPU, GPU, or FPGA — directly at the disk shelf level.
The Architecture of Universal Data Processing Engine
The UDPE architecture is built on three levels that allow it to be flexible and fast:
Integration Layer (DataApp): Allows legacy analytical engines to work with new accelerators without rewriting code. For a data engineer, this looks like magic: they run a familiar SQL query, and the UDPE decides on its own how to accelerate it.
Management Layer (DataOS): This is a resource dispatcher. It sees the entire hardware configuration of the storage server and dynamically distributes tasks. If there are free tensor cores available, computations go there. If GPUs are busy, the load shifts to the CPU. No resource sits idle.
Execution Core (DataVM): The deepest part — a virtual machine with its own instruction set, specifically tailored for data operations. It performs filtering and aggregation at speeds unattainable by standard libraries.
Why It Changes the Economics of AI
In the GPU-as-a-Service model, where a minute of cluster operation costs hundreds of dollars, every second of waiting for data from the SDS is a literal burning of money.
Elimination of the «Movement Tax»: UDPE implements the Zero-Copy principle. Data is processed where it resides. It does not need to be copied into server RAM and then into the GPU’s VRAM. This saves up to 80% of infrastructure costs.
Pure Math for GPUs: A traditional GPU spends most of its time on I/O Wait (waiting for data). UDPE prepares, cleanses, and vectorizes information before it even touches the accelerator. As a result, the GPU focuses solely on what it was designed for: heavy matrix computations.
Hardware-level Speed: Thanks to deep optimization, the engine performs typical tasks (such as table filtering) up to 10 times faster than standard solutions.
The Technological Foundation of Universal Data Processing Engine
The efficiency of UDPE is not built on «black boxes,» but on open standards. The engine utilizes the developments of the Gluten, Velox, and Substrait projects. This allows it to integrate into existing analytics ecosystems and accelerate them without forcing companies to migrate to new databases or change their SQL queries.
Activation Instead of Storage
With the advent of UDPE, the role of the storage system (SDS) changes fundamentally. It ceases to be a «deaf» warehouse:
Vectorization on Write: When a new document enters the storage, UDPE can immediately convert its text into a vector (embedding) right inside the storage controller. By the time an LLM needs this document, it is already in the required format.
Instant Analytics: Fraud detection happens in the microsecond of a transaction. The engine processes data «on the fly,» without waiting in a queue for GPU cluster training.
Procurement Savings: Since UDPE handles the preprocessing, companies no longer need the most expensive and power-hungry GPUs for simple tasks; more affordable accelerators are sufficient.
The Universal Data Processing Engine is becoming the foundation of Intelligent Data Infrastructure. It closes the final gap between physical hardware and applied AI. In a world where data grows faster than we can build data centers, the ability to process information in-place, without moving it, becomes the primary weapon in competitive struggle.
UDPE turns a disk shelf from a passive safe into an active brain that prepares data for a meeting with artificial intelligence even before it is written.
Contact our experts to select the optimal hardware configuration and calculate rental pricing for your AI workloads.