DATA ENGINEER AZURE DATABRICKS & PYSPARK

Duncan & Ross is hiring a Data Engineer for a leading client in Dubai. Youll design build and run scalable data pipelines and data warehouse solutions on Azure and Databricks turning raw data from across the business into trusted analytics-ready data.

What youll do

• Design build and maintain ETL/ELT pipelines that ingest transform and load data from multiple source systems into Azure and Databricks.
• Develop data transformations in Python and PySpark optimising Spark jobs for performance scalability and cost.
• Write efficient well-structured SQL for data modelling transformation and analysis across relational databases.
• Build and manage data warehouse and lakehouse solutions on Azure and Databricks including Delta Lake and medallion (bronze silver gold) architectures.
• Orchestrate and schedule data workflows using tools such as Azure Data Factory Databricks Workflows or Apache Airflow with monitoring and alerting.
• Put data quality checks validation rules and reconciliation in place so data is accurate complete and reliable.
• Apply data governance standards including cataloguing lineage access control and security using tools such as Unity Catalog or Microsoft Purview.
• Work with data analysts data scientists and business stakeholders to understand requirements and deliver data products.
• Troubleshoot pipeline failures and performance issues and continuously improve reliability.
• Document pipelines data models and processes and follow CI/CD and version control best practices.

What youll bring

• 3-5 years of hands-on data engineering experience.
• Data pipelines: proven experience building ETL/ELT pipelines for batch and incremental loads.
• SQL and relational databases: advanced SQL and strong experience with databases such as SQL Server Azure SQL PostgreSQL or Oracle.
• Big data: hands-on Apache Spark experience including performance tuning and handling large datasets.
• Cloud platforms: strong experience with Microsoft Azure (Data Lake Storage Data Factory Synapse) and Databricks.
• Data warehousing: experience designing data warehouses or lakehouses on Azure and Databricks including dimensional modelling (star and snowflake schemas) and Delta Lake.
• Programming: strong Python and PySpark skills.
• Orchestration: experience with Azure Data Factory Databricks Workflows Airflow or similar.
• Data quality and governance: experience implementing data quality frameworks data lineage metadata management and access controls.
• Familiarity with Git CI/CD pipelines (Azure DevOps or GitHub Actions) and Agile delivery.
• Bachelors degree in computer science Engineering Information Systems or a related field.
• Fluent written and spoken English.

Nice to have

• Databricks certification (Data Engineer Associate or Professional) or Microsoft Azure Data Engineer Associate (DP-203).
• Experience with streaming data (Kafka Event Hubs Spark Structured Streaming).
• Experience with Unity Catalog Microsoft Purview or Microsoft Fabric.
• Infrastructure as code (Terraform or Bicep).
• Exposure to BI tools such as Power BI.
• Previous experience in the UAE or wider GCC.

Vertical:

Technology