Designed and optimised ELT/ETL pipelines on Databricks and Azure — implementing Medallion Architecture (Bronze–Silver–Gold) using PySpark and SQL to transform multi-TB data streams into analytics-ready data warehouse layers, with a focus on query performance, clustering, and cost efficiency., Integrated Azure Data Factory, ADLS Gen2, and event-driven triggers (SQS/SNS) to build reliable, automated data ingestion pipelines from 6+ source systems — a pattern directly aligned to Snowflake + ADF + ADLS integration architectures., Designed and enforced secure data flows using RBAC, row-level security, and tenant-aware access patterns across data platform layers — ensuring governed, auditable access aligned to enterprise data governance standards., Implemented data quality and observability frameworks using Great Expectations, schema enforcement, and audit logging — achieving 99.9% data integrity across all transformation layers and enabling fast incident identification and resolution., Optimised high-volume SQL query performance and Spark job execution through Broadcast Joins, Z-Order indexing, Data Skipping, and Liquid Clustering — achieving 30% compute reduction and 60% improvement in query response times on large-scale workloads., Enforced CI/CD deployment workflows with Git-based version control and automated unit testing for ETL logic, reducing production incidents by 50% and ensuring controlled, reproducible pipeline releases., Provisioned cloud data environments using Terraform IaC, maintaining Dev/Test/Prod parity and supporting deployment documentation aligned to Agile sprint ceremonies and technical documentation standards., Collaborated with cross-functional Product, Analytics, and Engineering stakeholders to gather requirements, participate in Agile planning and estimation, and translate business needs into scalable data pipeline and warehouse designs., Designed Star Schema models and unified KPI semantic layers, reducing dashboard latency by 40% and ensuring metric consistency across all downstream analytics consumers.
