Experienced Data Engineer skilled in Python, SQL, Spark, Kafka, AWS. Proven track record in designing and implementing robust data architecture, models, and integration services. Well-versed in managing batch data proces…
Experienced Data Engineer skilled in Python, SQL, Spark, Kafka, AWS. Proven track record in designing and implementing robust data architecture, models, and integration services. Well-versed in managing batch data processing environments and implementing ETL/ELT pipelines within dynamic Data Lakes. Excited to apply my skills and expertise to contribute to impactful data initiatives and personal growth.
Developed and maintained robust data pipelines using Apache Airflow, ensuring high availability and efficient data transfer between MySQL and S3 (Data Lake).
Orchestrated Apache Airflow’s DAG for optimized task execution, resulting in 30% faster task processing.
Conducted data quality analysis on AWS S3 data lake, ensuring high standards in completeness, cleansing, validity, enrichment, and accuracy.
Collaborated with data architects to transform and store data in Redshift, facilitating seamless visualization in Power BI.
Designed and implemented a real-time data ingestion pipeline using Apache Kafka, processing over 50,000 events per second.
Utilized pySpark on Databricks to develop scalable and performant solutions for stream data integration and analysis.
Developed and maintained robust data pipelines using Apache Airflow to integrate and stitch data from diverse sources, achieving high availability.
Ensured efficient data transfer across MySQL and S3 (Data Lake).
Orchestrated an efficient workflow within Apache Airflow’s Directed Acyclic Graph (DAG), optimizing task execution and scheduling to process tasks 30% faster.
Conducted data quality analysis on AWS S3 data lake, ensuring high data quality standards, including completeness, cleansing, validity, enrichment, and accuracy.
Collaborated with data architects and business partners to transform and store data in Redshift, enabling seamless visualization in a Power BI dashboard.
Designed and implemented a real-time data ingestion pipeline using Apache Kafka, successfully processing over 50,000 events per second, ensuring efficient capture and transmission of webometrics and historical e-commerce data.
Employed pySpark on Databricks to develop highly scalable, extensible, and performant solutions for stream data integration, processing and analyzing.
Developed serverless actors - Web crawlers using Apify and Node.js to extract U.S. residents’ tax histories from over 35 county websites, leveraging advanced proxy and session techniques with tools like Cheerio, Puppeteer, and Playwright for efficient data scraping.
Implemented data wrangling and Scheduler for automated annual data scraping, transferring the data to MySQL via REST API.
Designed and scheduled workflow using python Selenium, Scarpy script that extracts data from predefined websites, cleans, transforms, and dumps into a MySQL database for production use, reducing the amount of manual effort by 70%.
Demonstrated proficiency in bypassing anti-scraping techniques, ensuring the consistent quality and availability of scraped data.
Implemented and managed a Scrapy crawler to systematically extract and parse offers from 40+ diverse Lebanese data sources, ensuring regular updates.
Enhanced metrics and reporting capabilities for web crawling by integrating Linux daemons, ensuring detailed insights into the scraping process and contributing to continuous improvement.
Successfully scraped details from more than 4k restaurants in the USA and Canada from Doordash, showcasing adeptness in anti-scraping techniques to ensure consistent data quality and availability.
نظام التوظيف التشغيلي المعتمد على الذكاء الاصطناعي أولاً، الذي يوحّد ويؤتمت دورة التوظيف بالكامل للمؤسسات الحديثة حول العالم.
د.جوب منطقة حرة ذ.م.م. 2026 © جميع الحقوق محفوظة
