Enabling Scalable Real-Time Sensor Stream Processing Across Decentralized Privacy-Preserving Storage Solutions

Abstract

Internet of Things (IoT), Smart Health, and Smart City applications generate continuousdata streams that may contain sensitive personal information. Decentralized storagesystems such as Solid provide data ownership, access control, and interoperable sharing butoffer limited support for real-time stream processing. We present Heimdall, an intermediaryanalytics service that registers RSP-QL queries over streams stored in Solid Pods to producequery results and reuses existing query executions when supported reuse conditionsare satisfied. We evaluate Heimdall against Client-Side Processing and a notificationintermediary using a wearable-sensor workload with concurrent client scaling, query anddata heterogeneity, and concurrent non-reusable queries. For equivalent queries, Heimdallmaintains nearly constant latency as client count increases, while Client-Side Processingshows substantial latency growth and instability at higher concurrency. Heimdall alsoreduces accumulated CPU and memory consumption and client-side network traffic bysharing stream retrieval and query execution. When query execution cannot be reused,shared stream acquisition still improves scalability, although degradation appears as thenumber of independent queries and physical streams increases with no-reuse. These resultsshow that shared stream acquisition and continuous-query execution can reduce duplicatedcomputation and communication in decentralized stream processing.