大数据系统方面的经典论文

说明：下面倾向选取已经在工业界广泛使用的系统论文，还有很多优秀论文没有在列表中，可以查阅近年来SOSP/OSDI/EuroSys/USENIX ATC/SIGMOD/VLDB/NIPS/ICML/KDD等相关会议获取。

分布式数据并行处理框架与编程模型

[Google MapReduce] Jeffrey Dean, Sanjay Ghemawat:
MapReduce: Simplified Data Processing on Large Clusters. OSDI 2004: 137-150

[Microsoft Dryad] Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, Dennis Fetterly:
Dryad: distributed data-parallel programs from sequential building blocks. EuroSys 2007: 59-72

[Microsoft DryadLINQ] Yuan Yu, Michael Isard, Dennis Fetterly, Mihai Budiu, Úlfar Erlingsson, Pradeep Kumar Gunda, Jon Currey:
DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language. OSDI 2008: 1-14

[Google FlumeJava] Craig Chambers, Ashish Raniwala, Frances Perry, Stephen Adams, Robert R. Henry, Robert Bradshaw, Nathan Weizenbaum:
FlumeJava: easy, efficient data-parallel pipelines. PLDI 2010: 363-375

[Apache Spark Core] Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauly, Michael J. Franklin, Scott Shenker, Ion Stoica:
Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. NSDI2012: 15-28

[Google Cloud Dataflow] Tyler Akidau, Robert Bradshaw, Craig Chambers, Slava Chernyak, Rafael Fernández-Moctezuma, Reuven Lax, Sam McVeety, Daniel Mills, Frances Perry, Eric Schmidt, Sam Whittle:
The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing. PVLDB 8(12): 1792-1803 (2015)

[Apache Tez] Bikas Saha, Hitesh Shah, Siddharth Seth, Gopal Vijayaraghavan, Arun C. Murthy, Carlo Curino:
Apache Tez: A Unifying Framework for Modeling and Building Data Processing Applications. SIGMOD Conference 2015: 1357-1369

[Apache Flink] Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, Kostas Tzoumas:
Apache Flink™: Stream and Batch Processing in a Single Engine. IEEE Data Eng. Bull. 38(4): 28-38 (2015)

大数据SQL

[Google Sawzall] Rob Pike, Sean Dorward, Robert Griesemer, Sean Quinlan:
Interpreting the data: Parallel analysis with Sawzall. Scientific Programming 13(4): 277-298 (2005)

[Apache Pig] Christopher Olston, Benjamin Reed, Utkarsh Srivastava, Ravi Kumar, Andrew Tomkins:
Pig latin: a not-so-foreign language for data processing. SIGMOD Conference 2008: 1099-1110

[Apache Hive] Ashish Thusoo, Joydeep Sen Sarma, Namit Jain, Zheng Shao, Prasad Chakka, Suresh Anthony, Hao Liu, Pete Wyckoff,Raghotham Murthy:
Hive - A Warehousing Solution Over a Map-Reduce Framework. PVLDB 2(2): 1626-1629 (2009)

[Berkeley Spark Shark] Reynold S. Xin, Josh Rosen, Matei Zaharia, Michael J. Franklin, Scott Shenker, Ion Stoica:
Shark: SQL and rich analytics at scale. SIGMOD Conference 2013: 13-24

[Apache Spark SQL] Michael Armbrust, Reynold S. Xin, Cheng Lian, Yin Huai, Davies Liu, Joseph K. Bradley, Xiangrui Meng, Tomer Kaftan, Michael J. Franklin, Ali Ghodsi, Matei Zaharia:
Spark SQL: Relational Data Processing in Spark. SIGMOD Conference 2015: 1383-1394

[Google Tenzing] Biswapesh Chattopadhyay, Liang Lin, Weiran Liu, Sagar Mittal, Prathyusha Aragonda, Vera Lychagina, Younghee Kwon, Michael Wong:
Tenzing A SQL Implementation On The MapReduce Framework. PVLDB 4(12): 1318-1327 (2011)

大规模图计算

[Google Pregel] Grzegorz Malewicz, Matthew H. Austern, Aart J. C. Bik, James C. Dehnert, Ilan Horn, Naty Leiser, Grzegorz Czajkowski:
Pregel: a system for large-scale graph processing. SIGMOD Conference 2010: 135-146

[CMU GraphLab] Yucheng Low, Joseph Gonzalez, Aapo Kyrola, Danny Bickson, Carlos Guestrin, Joseph M. Hellerstein:
Distributed GraphLab: A Framework for Machine Learning in the Cloud. PVLDB 5(8): 716-727 (2012)

[CMU PowerGraph] Joseph E. Gonzalez, Yucheng Low, Haijie Gu, Danny Bickson, Carlos Guestrin:
PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs. OSDI 2012: 17-30

[CMU GraphChi] Aapo Kyrola, Guy E. Blelloch, Carlos Guestrin:
GraphChi: Large-Scale Graph Computation on Just a PC. OSDI 2012: 31-46

[Apache Spark GraphX] Joseph E. Gonzalez, Reynold S. Xin, Ankur Dave, Daniel Crankshaw, Michael J. Franklin, Ion Stoica:
GraphX: Graph Processing in a Distributed Dataflow Framework. OSDI 2014: 599-613

分布式机器学习

[Google Distbelief] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc'Aurelio Ranzato, Andrew W. Senior, Paul A. Tucker, Ke Yang, Andrew Y. Ng:
Large Scale Distributed Deep Networks. NIPS 2012: 1232-1240

[CMU Parameter Server] Mu Li, David G. Andersen, Jun Woo Park, Alexander J. Smola, Amr Ahmed, Vanja Josifovski, James Long,Eugene J. Shekita, Bor-Yiing Su:
Scaling Distributed Machine Learning with the Parameter Server. OSDI 2014: 583-598

[CMU Petuum] Eric P. Xing, Qirong Ho, Wei Dai, Jin Kyu Kim, Jinliang Wei, Seunghak Lee, Xun Zheng, Pengtao Xie, Abhimanu Kumar, Yaoliang Yu:
Petuum: A New Platform for Distributed Machine Learning on Big Data. KDD 2015: 1335-1344

[Google TensorFlow] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis,Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard,Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore,Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, Xiaoqiang Zheng:
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. CoRR abs/1603.04467 (2016)

[Open-source MXNet] Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, Zheng Zhang:
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems. CoRRabs/1512.01274 (2015)

[Apache Spark MLlib] Xiangrui Meng, Joseph K. Bradley, Burak Yavuz, Evan R. Sparks, Shivaram Venkataraman, Davies Liu, Jeremy Freeman, D. B. Tsai, Manish Amde, Sean Owen, Doris Xin, Reynold Xin, Michael J. Franklin, Reza Zadeh, Matei Zaharia, Ameet Talwalkar:
MLlib: Machine Learning in Apache Spark. CoRR abs/1505.06807 (2015)

[CMU SSP Protocol] Henggang Cui, James Cipar, Qirong Ho, Jin Kyu Kim, Seunghak Lee, Abhimanu Kumar, Jinliang Wei, Wei Dai,Gregory R. Ganger, Phillip B. Gibbons, Garth A. Gibson, Eric P. Xing:
Exploiting Bounded Staleness to Speed Up Big Data Analytics. USENIX Annual Technical Conference2014: 37-48

流式数据处理

[Apache Spark Streaming] Matei Zaharia, Tathagata Das, Haoyuan Li, Timothy Hunter, Scott Shenker, Ion Stoica:
Discretized streams: fault-tolerant streaming computation at scale. SOSP 2013: 423-438

[Google MillWheel] Tyler Akidau, Alex Balikov, Kaya Bekiroglu, Slava Chernyak, Josh Haberman, Reuven Lax, Sam McVeety, Daniel Mills, Paul Nordstrom, Sam Whittle:
MillWheel: Fault-Tolerant Stream Processing at Internet Scale. PVLDB 6(11): 1033-1044 (2013)

[Microsoft TimeStream] Zhengping Qian, Yong He, Chunzhi Su, Zhuojie Wu, Hongyu Zhu, Taizhi Zhang, Lidong Zhou, Yuan Yu, Zheng Zhang:
TimeStream: reliable stream computation in the cloud. EuroSys 2013: 1-14

资源管理与任务调度

[Apache Hadoop YARN] Vinod Kumar Vavilapalli, Arun C. Murthy, Chris Douglas, Sharad Agarwal, Mahadev Konar, Robert Evans,Thomas Graves, Jason Lowe, Hitesh Shah, Siddharth Seth, Bikas Saha, Carlo Curino, Owen O'Malley, Sanjay Radia, Benjamin Reed, Eric Baldeschwieler:
Apache Hadoop YARN: yet another resource negotiator. SoCC 2013: 5:1-5:16

[Apache Mesos] Benjamin Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D. Joseph, Randy H. Katz, Scott Shenker, Ion Stoica:
Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center. NSDI 2011

[Google Borg] Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune, John Wilkes:
Large-scale cluster management at Google with Borg. EuroSys 2015: 18:1-18:17