Data Engineering and Visualization

What you'll learn

• Collect data from databases, files, logs and APIs responsibly, respecting licences, copyright and privacy law, and design representative samples that avoid leakage
• Write SQL that filters, joins, groups and uses window functions and CTEs correctly, avoiding fan-out and NULL pitfalls, to build time-bounded training sets
• Reshape, group and merge data with pandas, checking key relationships, and handle time zones and point-in-time joins
• Profile, clean and validate data, handle outliers and missing values, engineer features and build leakage-free scikit-learn pipelines
• Prepare text, time series, location and image data, building features that respect time and avoiding leakage and spurious cues
• Design labelling tasks, measure annotator agreement, find label errors, split data by group and time, de-identify data and document data sets
• Apply data governance, privacy law principles, de-identification, keyed pseudonymization, differential privacy and security controls to data work
• Choose storage systems and file formats, and build idempotent, validated, orchestrated batch and streaming pipelines with versioned data
• Process data that exceeds memory using compact types, columnar formats, partitioning, chunking, distributed engines and approximate sketches
• Explore data visually and communicate findings with honest, accessible charts that show uncertainty and support decisions
• Define metrics precisely, design decision-focused dashboards, recognize Simpson's paradox, and plan and analyse A/B tests without peeking
• Keep production model data reliable with data contracts, drift and skew monitoring and attention to feedback loops, and carry out an end-to-end data project