writing…

Heron, a real-time, distributed, fault-tolerant stream processing engine from Twitter.

Architecture

Heron Architecture

Topology Architecture

1 Users employ the Storm API to submit topologies to a scheduler.

2 The scheduler runs each topology as a job consisting of several containers.

• One of containers runs the topology master, responsible for managing the topology.

• The remaining containers each run a stream manager responsible for data routing, a metric manager that collects and report various metrics and a number of processes called Heron instances which run the user-defined spout/bolt code

These containers and allocated and scheduled by scheduler based on resource availability across the nodes in the cluster.

The metadata for the topology, such as physical plan and execution details, are kept in Zookeeper.

Core components of Heron

• Topology Master

• Container

• Stream Manager

• Metrics Manager

• Heron Instance

• Heron Tracker

Schedulers

…

State Managers

…

Further Reading

• Heron Official Document

• Twitter Heron: Stream Processing at Scale

• Twitter Heron Project - Github

• Optimizing Twitter Heron - Twitter blog