Data Engineer with 3+ years’ experience managing large-scale data environments, optimizing data pipelines, and architecting resilient Data Warehouses. Proficient driving data-driven strategies, enhancing operational effi…
Data Engineer with 3+ years’ experience managing large-scale data environments, optimizing data pipelines, and architecting resilient Data Warehouses. Proficient driving data-driven strategies, enhancing operational efficiency through Big Data technologies. Seeking to contribute knowledge elevates data-driven initiatives to a challenging role.
● Created Azure Data Factory pipelines to import data from text files & on-premises databases into Azure SQL data warehouse, reducing
data ingestion time by 30%.
● Engineered data pipelines for ETL processes, loading data from various sources into Azure data services like Azure Data Lake, Azure SQL
Database, or Azure Synapse Analytics, managing 10TB+ of data. Enabled in-depth analysis and led to a remarkable 10% increase in
customer retention.
● Pioneered the implementation of Databricks notebooks using Python (PySpark) and Spark SQL for transforming the data stored in Azure
Data Lake store Gen2 from Raw to Stage and Curated zones.
● Collaborated with cross-functional teams to identify and resolve data quality issues, leading to a significant decrease in data errors by
50% and contributing to a 70% increase in stakeholder satisfaction.
● Transformed Hive/SQL queries into efficient Spark transformations using Spark RDDs and Python, resulting in a 40% reduction in data
processing time and improved overall data pipeline performance.
● ETL/ELT technologies and scripting languages are used to develop and maintain workflows for real-time data processing.
Created Azure Data Factory pipelines to import data from text files and on-premises databases into Azure SQL data warehouse, reducing data ingestion time by 30%.
Engineered data pipelines for ETL processes, loading data from various sources into Azure data services like Azure Data Lake, Azure SQL Database, or Azure Synapse Analytics, managing 10TB+ of data.
Enabled in-depth analysis and led to a remarkable 10% increase in customer retention.
Pioneered the implementation of Databricks notebooks using Python (PySpark) and Spark SQL for transforming the data stored in Azure Data Lake store Gen2 from Raw to Stage and Curated zones.
Collaborated with cross-functional teams to identify and resolve data quality issues, leading to a significant decrease in data errors by 50% and contributing to a 70% increase in stakeholder satisfaction.
Transformed Hive/SQL queries into efficient Spark transformations using Spark RDDs and Python, resulting in a 40% reduction in data processing time and improved overall data pipeline performance.
Used ETL/ELT technologies and scripting languages to develop and maintain workflows for real-time data processing.
Led development of an open-source tool that prints 50+ Cloudera-hosted Hadoop Cluster health and monitoring metrics such as Hadoop, HDFS, Hive, HBase, Zookeeper, Kafka, Apache Spark, and MapReduce using Shell Scripting, Cloudera API, and Python Libraries.
Conducted auditing, inspecting, and visualizing S3 server access logs using AWS Lambda and AWS Athena.
Automated resulting scripts and workflow using Apache Airflow and shell scripting to ensure daily execution in production.
Utilized AWS Glue for transformations and AWS Lambda, SQS, and SNS to automate processes and worked on AWS Data Pipeline to configure data loads from S3 to Redshift.
Spearheaded development and implementation of data pipelines in Snowflake, automating processes and reducing manual intervention by 99%, while empowering business users with enhanced data ownership.
Worked on Source/Version control Tools using Github, validate the change sets code changes, Check-in/Out, and versioning and developed CI/CD pipelines using Jenkins on AWS to create, test, and deploy code to higher environments.
Authored in-depth analysis of network element functionality trends by executing advanced python and SQL queries on data warehouse (Snowflake) including predictive modeling using scikit-learn.
Conducted 50+ code reviews and mentored 12 junior developers on various projects.
Analyzed S2TM with table descriptions, schema details for both source and target, data types of both source and target while mapping columns, transformations, and Teradata target table structure.
Employed Pig and Hive Scripts to transform raw data based on given transformation provided in S2TM.
Prepared Source and target DDL's based on S2TM and designed tables accordingly in Hive and Hbase.
Solved performance issues in Pig scripts and Hive Scripts with understanding of Joins, Groups, and aggregation.
Developed Tableau dashboards for real-time insights and utilized Apache Airflow for ETL orchestration, ensuring data consistency.
نظام التوظيف التشغيلي المعتمد على الذكاء الاصطناعي أولاً، الذي يوحّد ويؤتمت دورة التوظيف بالكامل للمؤسسات الحديثة حول العالم.
د.جوب منطقة حرة ذ.م.م. 2026 © جميع الحقوق محفوظة
