Data Milky Way: A Brief History (Part 4) - Architecture Reference

Typical Data Pipeline

Open Source Example Architecture

Reference

AWS Overview

Reference Architecture

This example uses AWS Glue. There are other AWS services like EMR that can handle even more complex cases.

Batch Processing on Azure

Reference

Batch & Streaming (Lambda Architecture) on Azure

Reference

Lambda vs. Kappa Architecture

Read about the differences.

Lambda: The two layers (speed layer & batch layer) resemble the two legs of the λ symbol.

Kappa: The left stem of the 𝚱 (kappa symbol) signifies unified input data store and processing logic.