Data Engineer (PySpark)
شارك هذه الوظيفة
موقع الوظيفة:
الراتب شهرياً:
تاريخ النشر:
نُشرت قبل 2 ساعة
عدد الوظائف الشاغرة:
1 عدد الوظائف الشاغرة
ملخص الوظيفة
We are seeking an experienced Data Engineer (PySpark) to design build optimize and maintain scalable data pipelines for production environments. The role requires strong hands-on experience in big data processing pipeline optimization and deployment using modern data engineering tools and frameworks.
Key Responsibilities
Design develop and maintain robust scalable data pipelines using Python and PySpark
Perform data ingestion transformation cleansing and validation across structured and unstructured datasets
Conduct Exploratory Data Analysis (EDA) to identify data patterns anomalies and quality issues
Apply data imputation techniques data linking and cleansing to ensure high data quality
Implement feature engineering pipelines to support analytics and downstream use cases
Optimize Spark jobs for performance scalability and cost efficiency
Deploy and tune production-grade data pipelines ensuring reliability and performance
Automate workflows using Apache Airflow and/or Jenkins
Collaborate with cross-functional teams to integrate data solutions into production systems
Write and maintain unit tests to ensure code quality and reliability
Manage source code CI/CD and deployments using Git GitHub and GitHub Actions
Requirements
Required Technical Skills
Strong proficiency in Python
Extensive hands-on experience with Apache Spark (PySpark)
Experience working with Jupyter Notebooks
Strong knowledge of SQL and NoSQL databases
Proven experience with Git for version control and CI/CD
Hands-on experience with Apache Airflow and/or Jenkins for scheduling and automation
Solid understanding of data engineering best practices in production environments
Demonstrated experience in Spark performance tuning and optimization
Ability to write clean testable and maintainable Python code
Mandatory Requirement
Previous production experience is a MUST specifically in deploying tuning and maintaining data pipelines in production environments
Preferred Qualifications
Experience working in high-volume or big data environments
Strong problem-solving and analytical skills
Ability to work independently in a fast-paced environment
Why Join
Competitive salary package
Opportunity to work on production-scale data platforms
Exposure to modern data engineering tools and practices
Dubai-based role with a dynamic and collaborative work environment
المجال
خدمات تقنية المعلومات واستشارات تكنولوجيا المعلومات
المهارات المطلوبة
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- البيانات الضخمة
- مستودع البيانات
- Kafka
- Scala
عن الشركة
Black Pearl is a progressive, dynamic and well structured HR solution provider that offers permanent recruitment services, HR consultancy, psychometric assessments, coaching and also professional training services for clients from different corporate sectors in the Middle East. Like a ... اعرض المزيد
