|
|
Test and analysis of big data system in different network environment |
School of Computer Science and Telecommunication Engineering, Jiangsu University, Zhenjiang, Jiangsu 212013, China |
|
|
Abstract To investigate the effects of network architecture and communication protocol on large data processing and application system efficiency,the introduction and the analysis of different network architecture and communication protocol were given. The Ethernet, Infiniband, TCP/IP, IPoIB and RDMA protocols were used to construct the prototypes of Hadoop, Tachyon and Spark. Some common test tools and applications were used to evaluate the performance of prototypes. The test results show that compared to TCP/IP protocol, the I/O performance of Hadoop can be improved by 46 to 56 times with IPoIB protocol, and the time overhead of Tachyon data processing can be reduced up to 2%~27% and 90%~95% for spark. The performance of Spark is improved by 46 times. Compared to IPoIB, the system overhead can be decreased by 3%~15% by RDMA protocol. The highspeed network architecture and efficient communication protocol can effectively improve I/O performance, efficiency and adaptability of big data system.
|
|
|
|
|
|
|
|