With the increased availability of data generation approaches, focus on how to handle data is becoming a central topic in research. The ability to collect, process, and derive insights from large datasets is crucial for making informed decisions and driving discovery. Raw data is available in various format, whether structured or unstructured, and is often located in different places in large amounts. Therefore, reliable strategies to support scalable ingestion, transformation for downstream analysis, and storage for easy retrieval are greatly needed. Data engineering (DE) focuses on the design, implementation, evaluation and monitoring of data pipelines to ensure reliability and reproducibility. In this hands-on workshop, we will explore the different data pipelines, e.g. ETL (Extract, Transform, and Load) or ELT data pipelines, that can be implemented to prepare data for visualization, analysis or modeling. Since raw data is seldom ready for use; we will explore techniques to enable us to make sense or augment the data, e.g. data visualization, feature engineering, data cleaning, and data drift. Moreover, we will examine the medallion architecture and how it can help store and organize data in a flexible manner to support easy retrieval and the iterative process of data engineering. Finally, we explore the different tools that can support better data and pipeline management. By the end of this workshop, researchers will gain an understanding of basic data engineering principles and insight into how to apply these skills to their own projects. This will enable them to collect, process, and analyze large datasets efficiently and effectively. While no prior experience in data engineering is required, familiarity with Python programming language and standard libraries, e.g. pandas, scikit-learn or matplotlib, will be helpful.
---
Die Veranstaltung wird von der Landesinitiative Forschungsdatenmanagement Niedersachsen (FDM-NDS) organisiert. FDM-NDS ist ein Verbundprojekt unter dem Dach der Hochschule.digital Niedersachsen und wird im Rahmen von zukunft.niedersachsen, einem Förderprogramm von Niedersächsischem Ministerium für Wissenschaft und Kultur (MWK) und VolkswagenStiftung gefördert.
