Skip to main content
Version: 1.0

Data Milky Way: A Brief History (Part 4) - Architecture Reference

Typical Data Pipeline​

typical-data-pipeline.png

Open Source Example Architecture​

AWS Overview​

This example uses AWS Glue. There are other AWS services like EMR that can handle even more complex cases.

Batch Processing on Azure​

Batch & Streaming (Lambda Architecture) on Azure​

Lambda vs. Kappa Architecture​

Read about the differences

Lambda

  • the two layers (speed layer & batch layer) resemble
  • the two legs of the λ symbol

lambda.png

Kappa the left stem of the 𝚱 (kappa symbol) signifies unified input data store and processing logic

kappa.png