You've come so far!

During the course, you've probably been thinking about all the amazing things you could do with your newly acquired knowledge. Now it's time to bring it all together and impress each other with vivid graphs that show correlations and answer questions you (or a business) might have.

Pairing or working in a group is highly recommended. You can also work alone, but you will have to do a lot more work.

Instructions

1. Pick a dataset. You can find some suggestions below. You can also use your own data.

   • https://dev.to/seattledataguy/5-data-sources-for-data-engineering-projects-3j21

   • https://github.com/awesomedata/awesome-public-datasets

   • https://www.tableau.com/learn/articles/free-public-data-sets

   • https://www.google.com/publicdata/directory

   • https://ourworldindata.org (/world-population-growth)

   • https://open.esa.int

   • https://www.kaggle.com/datasets

     You are free to use other sources of data, but do not spend too much time here!

2. Write a five-sentence management-summary of the goals to achieve. This means we first look for data, and then we build a business case. This is the opposite of what you should do in real life, but in the interest of time we're doing it this way around.

3. Share your management-summary with the team in our chat topic.

4. Decide if you want to use batch processing or to stream the data.

5. Ingest the data and clean them up properly.

6. Transform the data into the target structure that enables you to answer the questions adequately.

7. Run analytics, build a dashboard, or automate a trigger to do something with the data, according to your business goal.

8. Prepare a five-minute presentation of your work to show it to the team. Include:

   • The question you want to answer
   • Talk about the methods you've used in your code (why and how)
   • Show the outcome
   • Talk about the learnings you've made & possible extensions of the project

Note for this final project you don't have much time. Therefore, be well-structured and do not take on too much. A complete walkthrough is more important than a nice detailed solution. Hint: Take ten minutes to do a project plan first.