Mega-KV is the fastest in-memory key-value store on commodity processors, which can be up to 100X faster than Memcached.
Mega-KV makes innovative use of GPUs to offload the index data structure and the corresponding operations.
Mega-KV takes future growth into consideration, and has a good extensibility to more abstractions, interfaces, and functions.
Mega-KV is an open source, MIT licensed, advanced in-memory key-value store.
With the recent advancement on hardware and software techniques for high-speed network processing, network processing is no longer the bottleneck for an in-memory key-value store. Instead, accessing the index data structure and key-value items in the main memory have become the two major operations with the highest overhead in the query processing. However, as the memory performance significantly lags behind the CPU computing power, the key-value store performance is hard to be further improved under the conventional Multicore architectures.
Mega-KV adopts GPU in the query processing by offloading the index data structure and its corresponding operations to GPUs. By best utilizing the high-memory bandwidth and the memory access latency hiding capability of GPUs, Mega-KV achieves significant performance improvement.
Kai Zhang, Kaibo Wang, Yuan Yuan,
Lei Guo, Rubao Lee, Xiaodong Zhang.
Mega-KV: A Case for GPUs to Maximize the Throughput of In Memory Key-Value Stores.
VLDB 2015, Kohala, Hawaii, August 31 -
September 4, 2015.
We are continuously improving the
project. If you have any suggestions
or find any bugs in the code, you
can email the author at
kay21s [AT] gmail [DOT] com
My Homepage: https://kay21s.github.io/.
Required Hardware: Intel CPU (with SSE instruction set), Nvidia GPU, and Intel 10Gbps NIC.
Prerequisite Software: Nvidia CUDA SDK 6.5, and Intel DPDK 1.7.1.
1. Setup NIC with DPDK. Then run 'export RTE_SDK=DPDK_PATH', DPDK_PATH is the path of DPDK directory.
2. Go to libgpuhash directory, run 'make'. This generates the GPU hash table library.
3. Go to src directory, modify configurations including memory limit and CPU core mappings, then run 'make'.
4. Run 'sudo ./build/megakv' to start Mega-KV.
5. On another machine connected via a 10Gbps network, go to benchmark directory and run 'make'.
6. Run 'sudo ./build/benchmark' to generate queries.
For more information on the macros for setup, please refer to README.