Enabling Organizations to Participate in the Ad Hoc Scheduling of a Cross-Organizational Data Pipeline

Abstract

In case of an emergency situation, it is required to come up with solutions quickly. The supporting decision-making process should therefore be based on relevant data sources which are fed to data processing pipelines. These data sources may however be located in different domains of distinct organizations. Although the technical realisation of cross-organizational data pipelines is possible, for example using a federated Kubernetes cluster managed by a central operator, it is unclear how those pipelines should be scheduled among the participating organizations. Firstly, the cross-organizational infrastructure is unknown and highly heterogeneous, and secondly, there may be undiscovered scheduling preferences present. The first issue can be solved using software probes, while the second issue will be solved through an extension of this probing concept. During the proposed inter-domain scheduling process, organizations may specify monetary reward requirements and requirements on the maximum load they wish to bear. The former allows administrators to specify a lower limit with respect to payment they request for their contribution, the latter allows the specification of an upper limit on the workload an organization wishes to process. This way, mismatching expectations on contribution level, which may potentially cause harm from the hosting organization perspective, are prevented, increasing the trust level of contributors. These kinds of requirements have nothing to do with the technical assessment of a node, but they do impact scheduling decisions and performance of an application, as shown in our evaluation. The proposed scheduling flow not only allows organizations to steer scheduling decisions, but also to negotiate requirements among each other, giving rise to ad hoc conflict resolution all the while collaborating in solving the emergency situation.