Prophet: An Efficient Feature Indexing Mechanism for Similarity Data Sharing at Network Edge

Abstract

As a promising infrastructure, edge storage systems have drawn many attempts to efficiently distribute and share data among edge servers. However, it remains open to meeting the increasing demand for similarity retrieval across servers. The intrinsic reason is that the existing solutions can only return an exact data match for a query while more general edge applications require the data similar to a query input from any server. To fill this gap, this paper pioneers a new paradigm to support high-dimensional similarity search at network edges. Specifically, we propose Prophet, the first known architecture for similarity data indexing. We first divide the feature space of data into plenty of subareas, then project both subareas and edge servers into a virtual plane where the distances between any two points can reflect not only data similarity but also network latency. When any edge server submits a request for data insert, delete, or query, it computes the data feature and the virtual coordinates; then iteratively forwards the request through greedy routing based on the forwarding tables and the virtual coordinates. By Prophet, similar high-dimensional features would be stored by a common server or several nearby servers. Compared with distributed hash tables in P2P networks, Prophet requires logarithmic servers to access for a data request and reduces the network latency from the logarithmic to the constant level of the server number. Experimental results indicate that Prophet achieves comparable retrieval accuracy and shortens the query latency by 55%~70% compared with centralized schemes.