CSI Driver X FUSE Drivers: a Kubernetes Object Storage Solution for AI/ML Data Portab... Jiaxun Song

CSI Driver X FUSE Drivers: a Kubernetes Object Storage Solution for AI/ML Data Portability - Jiaxun Song, Google
Object storage is a common storage choice for AI/ML workloads, but it has lacked a Kubernetes volume interface. Workloads that consume training data from a bucket have to be written using object storage APIs, even though these read-heavy, throughput-oriented workloads are a good fit for data consumption over FUSE. The practical production use of FUSE as a Kubernetes volume layer has three requirements. First, the lifetime of the FUSE process must match that of the consuming Pod. Second, the FUSE process can consume significant resources which must be associated with the consuming Pod. Thirdly, unlike conventional storage such as disks, object storage is usually accessed using workload specific credentials. We present a solution to these problems based on sidecar injection, taking advantage of kernel cross-process file descriptor transfer to cleanly separate privileges, authentication, workload lifecycle and resource usage. The solution can be adopted by any FUSE volume.

[@ianbyte5891]

👍👍