Shashwat Jaiswal

The shared and distributed memory capabilities of the emerging Compute Express Link (CXL) interconnect urge us to rethink the traditional interfaces of system software. In this paper, we explore one such interface: remote fork using CXL-attached shared memory for cluster-wide process cloning. We present CXLfork, a remote fork interface that realizes close to zero-serialization, zero-copy process cloning across nodes over CXL fabrics. CXLfork utilizes globally-shared CXL memory for cluster-wide deduplication of process states. It also enables fine-grained control of state tiering between local and CXL memory. We use CXLfork to develop CXL-porter, an efficient horizontal autoscaler for serverless functions deployed on CXL fabrics. CXLfork minimizes cold-start overhead without sacrificing local memory. CXLfork attains restore latency close to that of a local fork, outperforming state-of-practice by 2.26x on average, and reducing local memory consumption by 87% on average.

Global cloud service providers handle inference workloads for LLMs that span latency-sensitive (e.g., chatbots) and insensitive (e.g., report writing) tasks, with diverse and often conflicting Service Level Agreement (SLA) needs. Serving such mixed workloads is challenging due to the multiple models, diverse GPU hardware, and globally distributed data centers. Solutions that silo fast and slow tasks on separate GPU pools cause under-utilization of expensive accelerators. Here, we characterize the large-scale LLM inference workloads (> 10M requests per day) at Microsoft Office 365, and highlight key observations in one of the first Internet-scale public studies. These insights drive SageServe, an LLM serving framework that dynamically adapts to workload demands. It combines short-term request routing to data centers with long-term scaling of GPU VMs and model placement with higher lead times, and co-optimizes the routing and resource allocation problem using a traffic forecast model and an ILP solution. We evaluate SageServe on traces comprising 10M production requests across three regions and four open-source models. SageServe achieves up to 25% savings in GPU-hours compared to the current baseline deployment and reduces GPU-hour wastage due to inefficient auto-scaling by 80%, potentially resulting in monthly cost savings of up to $2.5 million, while maintaining tail latency and meeting SLAs.

Recent years have seen an unprecedented explosion in research that leverages the newest computing paradigm of Internet of Drones comprised of a fleet of connected Unmanned Aerial Vehicles (UAVs) used for a wide range of tasks such as monitoring and analytics in highly mobile and changing environments characteristic of disaster regions. Given that the typical data (i.e., videos and images) collected by the fleet of UAVs deployed in such scenarios can be considerably larger than what the onboard computers can process, the UAVs need to offload their data in real-time to the edge and the cloud for further processing. To that end, we present the design of AerialDB- a lightweight decentralized data storage and query system that can store and process time series data on a multi-UAV system comprising: (A) a fleet of hundreds of UAVs fitted with onboard computers, and (B) ground-based edge servers connected through a cellular link. Leveraging lightweight techniques for content-based replica placement and indexing of shards, AerialDB has been optimized for efficient processing of different possible combinations of typical spatial and temporal queries performed by real-world disaster management applications. Using containerized deployment spanning up to 400 drones and 80 edges, we demonstrate that AerialDB is able to scale efficiently while providing near real-time performance with different realistic workloads. Further, AerialDB comprises a decentralized and locality-aware distributed execution engine which provides graceful degradation of performance upon edge failures with relatively low latency while processing large spatio-temporal data. AerialDB exhibits comparable insertion performance and 100 times improvement in query performance against state-of-the-art baseline. Moreover, it experiences a 10 times improvement in performance with insertion workloads and 100 times improvement with query workloads over the cloud baseline.