Project Introduction

In the next few sections, we'll walk through some of those landmarks that we just talked about, namely

• Data Ingestion
• Data Transformation
• Data Visualisation

And we'll do that in the context of a small project.

We'll take Example 1: Jess as our core use case.

Our goal is to study the relationship between CO2 and Temperature around the world. Some questions that we aim to answer:

• Which countries are worse-hit (higher temperature anomalies)?
• Which countries are the biggest emitters?
• What are some attempts of ranking “biggest polluters” in a sensible way?

We'll focus first on developing our core Ingestion logic by leveraging the enahanced learning experience in Databricks notebooks. We'll do the same for our Data Transformation logic. After this, we'll demonstrate how this comes together in an automated pipeline. And of course, lastly, we'll visualise our data.