نبذة عني
Data engineer with three years of experience seeking a challenging position to leverage my technical skills and expertise in designing, implementing, and maintaining data infrastructure solutions. Passionate about optimi…
Data engineer with three years of experience seeking a challenging position to leverage my technical skills and expertise in designing, implementing, and maintaining data infrastructure solutions. Passionate about optimizing data workflows, improving data quality, and enabling data-driven decision-making.
الخبرة
Data Engineer
Built a real-time data streaming pipeline using Apache Kafka and Apache Flink, enabling the processing of high-velocity data streams for immediate insights.
Utilized Sqoop to efficiently import data from diverse sources such as Oracle RDBMS and Teradata into Hadoop, demonstrating extensive experience in data integration.
Developed and maintained efficient database structures using SQL, ensuring data integrity and optimal query performance.
Designed and implemented data pipelines utilizing various GCP services such as Compute Engine, Dataproc, Cloud Storage, Big Query, and Pub/Sub, successfully performing ETL operations on large volumes of data.
Develop and maintain Azure Data Factory pipelines to orchestrate data movement and transformation processes.
Developed PySpark code for Cloud Dataproc jobs and Google Cloud Engine cluster, managed Databricks notebooks and Delta Lake with Python, demonstrating a strong technical background in big data technologies.
Developed and executed Python scripts to effectively parse XML documents and load the extracted data into databases.
Utilized data orchestration tools such as Apache Airflow and Oozie to effectively manage and automate complex data workflows.
Performed in-depth modeling, analysis, and cleansing of structured and unstructured data to support business initiatives, resulting in a significant monthly cost savings of $5,000, exhibiting expertise in data analysis and optimization.
Design and implement scalable and efficient data warehouse solutions using Azure SQL Data Warehouse and Azure Synapse Analytics.
Developed real-time data analysis dashboards using SQL, Python, and HTML/CSS to visualize and analyze streaming data from social media APIs.
Implement data ingestion and integration solutions using Azure Data Factory and event-driven architectures.
Designed and implemented a cloud-based data lake architecture using Amazon S3 and Apache Hive, enabling efficient data storage, retrieval, and analytics for a healthcare analytics company.
Implemented data partitioning and clustering techniques in Apache Hive and Spark, significantly improving query performance on large datasets for a retail company.
Implemented a real-time data pipeline using Apache Kafka and Apache Storm, enabling the processing and analysis of streaming data with sub-second latency.
Utilized advanced SQL functions and joins to perform complex data analysis and generate meaningful insights for business stakeholders.
Build and manage data lake solutions using Azure Data Lake Storage and Azure Blob Storage.
Efficiently managed large volumes of data, up to 400 GB per day, including both batch and real-time data, utilizing Google Cloud Dataprep to streamline data conversions and improve data quality. Resulted in increased lead generation and enhanced customer experience.
Leverage Azure Databricks to perform big data processing, including data exploration, cleansing, and advanced analytics.
Utilized Google Cloud Platform (GCP) services such as Big Query Dataflow, and Cloud Storage to build a scalable and cost-effective data processing pipeline for a digital marketing agency.
Developed and maintained SQL scripts for data validation, ensuring accuracy and consistency across multiple datasets.
Documented data warehouse design, data dictionaries, and ETL processes to facilitate knowledge sharing, maintenance, and future enhancements.
Data Engineer
Built a real-time data streaming pipeline using Apache Kafka and Apache Flink, enabling the processing of high-velocity data streams for immediate insights.
Utilized Sqoop to efficiently import data from diverse sources such as Oracle RDBMS and Teradata into Hadoop, demonstrating extensive experience in data integration.
Developed and maintained efficient database structures using SQL, ensuring data integrity and optimal query performance.
Designed and implemented data pipelines utilizing various GCP services such as Compute Engine, Dataproc, Cloud Storage, Big Query, and Pub/Sub, successfully performing ETL operations on large volumes of data.
Develop and maintain Azure Data Factory pipelines to orchestrate data movement and transformation processes.
Developed PySpark code for Cloud Dataproc jobs and Google Cloud Engine cluster, managed Databricks notebooks and Delta Lake with Python, demonstrating a strong technical background in big data technologies.
Developed and executed Python scripts to effectively parse XML documents and load the extracted data into databases.
Utilized data orchestration tools such as Apache Airflow and Oozie to effectively manage and automate complex data workflows.
Performed in-depth modeling, analysis, and cleansing of structured and unstructured data to support business initiatives, resulting in a significant monthly cost savings of $5,000, exhibiting expertise in data analysis and optimization.
Design and implement scalable and efficient data warehouse solutions using Azure SQL Data Warehouse and Azure Synapse Analytics.
Developed real-time data analysis dashboards using SQL, Python, and HTML/CSS to visualize and analyze streaming data from social media APIs.
Implement data ingestion and integration solutions using Azure Data Factory and event-driven architectures.
Designed and implemented a cloud-based data lake architecture using Amazon S3 and Apache Hive, enabling efficient data storage, retrieval, and analytics for a healthcare analytics company.
Implemented data partitioning and clustering techniques in Apache Hive and Spark, significantly improving query performance on large datasets for a retail company.
Implemented a real-time data pipeline using Apache Kafka and Apache Storm, enabling the processing and analysis of streaming data with sub-second latency.
Utilized advanced SQL functions and joins to perform complex data analysis and generate meaningful insights for business stakeholders.
Build and manage data lake solutions using Azure Data Lake Storage and Azure Blob Storage.
Efficiently managed large volumes of data, up to 400 GB per day, including both batch and real-time data, utilizing Google Cloud Dataprep to streamline data conversions and improve data quality. Resulted in increased lead generation and enhanced customer experience.
Leverage Azure Databricks to perform big data processing, including data exploration, cleansing, and advanced analytics.
Utilized Google Cloud Platform (GCP) services such as Big Query Dataflow, and Cloud Storage to build a scalable and cost-effective data processing pipeline for a digital marketing agency.
Developed and maintained SQL scripts for data validation, ensuring accuracy and consistency across multiple datasets.
Documented data warehouse design, data dictionaries, and ETL processes to facilitate knowledge sharing, maintenance, and future enhancements.
University Project
Designed and developed a centralized data warehouse for a university, consolidating data from multiple sources and enabling efficient data retrieval and analysis.
Plan and execute data migration projects, moving on-premises data to Azure cloud platforms.
Implemented data modeling techniques to create a logical and physical representation of the university's data within the data warehouse, ensuring optimal performance and data integrity.
Built robust ETL (Extract, Transform, Load) processes to extract data from various source systems, perform necessary transformations, and load it into the data warehouse for further analysis.
Implement real-time data processing using Azure Stream Analytics and Apache Kafka.
Develop streaming pipelines to handle high-volume, high-velocity data streams.
Created a data integration solution using Python and RESTful APIs, enabling seamless data exchange and synchronization between multiple systems for a university data interoperability project.
Collaborated with cross-functional teams, including university administrators, IT professionals, and data analysts, to gather requirements and ensure the data warehouse met their needs for reporting and decision-making purposes.
Developed a scalable architecture for the data warehouse, utilizing appropriate technologies and infrastructure to handle the volume and complexity of university data effectively.
Conducted data quality assessments and implemented data cleansing and validation procedures to ensure the accuracy and reliability of data within the data warehouse.
Implemented security measures to protect sensitive data stored in the data warehouse, including access controls, data encryption, and compliance with privacy regulations.
Modernize legacy data systems by transforming and migrating them to modern Azure data services.
Designed and optimized a distributed database system using Hadoop Distributed File System (HDFS) and Apache Hive, enabling efficient storage, and querying of large-scale genomic data for a bioinformatics research project.
Collaborated with stakeholders to identify additional data sources and expand the data warehouse's capabilities, enabling comprehensive data analysis and reporting across various university domains.
Transactional Risk Data Analyst
Performed data extraction to identify business patterns and financial anomalies of clients in the US marketplace, successfully preventing fraudulent transactions valued over $25M through in-depth data analysis and pattern recognition.
Spearheaded the automation of investigation reports, metrics, and audit data, transforming manual processes into efficient, data-driven solutions utilizing Tableau and SQL. This resulted in improved accuracy and productivity in the investigation process.
Provided mentorship to a team of 8 individuals in the development of advanced fraud detection protocols, addressing emerging patterns of group and seller behavior. Through effective guidance, the team achieved a 35% improvement in production quality metrics, demonstrating exceptional leadership and team management skills.
Implemented performance optimization techniques on SQL queries, resulting in a 25% reduction in data retrieval time, demonstrating proficiency in database optimization and performance tuning.
Conducted in-depth analysis of e-commerce data trends using Python, presenting insights through comprehensive reports created using Tableau and MS Excel. Demonstrated expertise in data analysis and reporting, driving informed decision-making for business initiatives.
Designed and implemented views and performed complex data aggregations utilizing SQL.
Transactional Risk Data Analyst
Performed data extraction to identify business patterns and financial anomalies of clients in the US marketplace, successfully preventing fraudulent transactions valued over $25M through in-depth data analysis and pattern recognition.
Spearheaded the automation of investigation reports, metrics, and audit data, transforming manual processes into efficient, data-driven solutions utilizing Tableau and SQL. This resulted in improved accuracy and productivity in the investigation process.
Provided mentorship to a team of 8 individuals in the development of advanced fraud detection protocols, addressing emerging patterns of group and seller behavior. Through effective guidance, the team achieved a 35% improvement in production quality metrics, demonstrating exceptional leadership and team management skills.
Implemented performance optimization techniques on SQL queries, resulting in a 25% reduction in data retrieval time, demonstrating proficiency in database optimization and performance tuning.
Conducted in-depth analysis of e-commerce data trends using Python, presenting insights through comprehensive reports created using Tableau and MS Excel. Demonstrated expertise in data analysis and reporting, driving informed decision-making for business initiatives.
Designed and implemented views and performed complex data aggregations utilizing SQL.