Big Data

A view of Microsoft Data Center, 来源: Armando Fox, Cloud Computing What’s in It for Me as a Scientist?

---

来源: Open Source Data Infrastructure

Batch Processing (offline systems)

• HDFS

• MapReduce

Stream Processing (near-real-time systems)

• A Apache Storm

• P Twitter Heron

• P Apache Flink

• P Spark

Resource Management & Scheduling

• Apache YARN

• Mesos

  An execution platform for hosting applications

• Kubernetes

来源: Omega: flexible, scalable schedulers for large compute clusters

Messaging Systems

• A Apache Kafka

  log based message broker

• D RabbitMQ

  AMQP-style message broker

Further Readings

Google

• MapReduce: Simplified Data Processing on Large Clusters

  MapReduce is a programming model and an associated implementation for processing and generating large data sets.

• Dremel: Interactive Analysis of Web-Scale Datasets

  Dremel is a scalable, interactive ad-hoc query system for analysis of read-only nested data.

• Pregel: A System for Large-Scale Graph Processing

• Distributed Large-scale Natural Graph Factorization

  For large-scale graph decomposition and inference

• Omega: flexible, scalable schedulers for large compute clusters