Data Platform Engineer

Roles and responsibilities

The client is seeking an exceptional Data Platform Engineer to join the team of AI specialists in the UAE. As a Data Platform Engineer, you will be responsible for designing, implementing, and optimizing data pipelines and infrastructure to support our cutting-edge AI systems. You will collaborate closely with the multidisciplinary team to ensure the efficient collection, storage, processing, and analysis of large-scale data, enabling us to unlock valuable insights and drive innovation across various domains.

Responsibilities:

• Pipeline Development: Design, develop, and maintain scalable and reliable data pipelines to ingest, transform, and load diverse datasets from various sources, including structured and unstructured data, streaming data, and real-time feeds.

• Data Integration: Implement robust data integration processes to seamlessly integrate data from different sources, ensuring consistency, reliability, and data quality.

• Data Storage: Design and optimize data storage solutions, including relational databases, NoSQL databases, data lakes, and cloud storage services, to efficiently store and manage large volumes of data.

• Performance Optimization: Optimize data processing and query performance to enhance system scalability, reliability, and efficiency, leveraging techniques such as indexing, partitioning, caching, and parallel processing.

• Data Governance: Implement data governance frameworks to ensure data security, privacy, integrity, and compliance with regulatory requirements, including data encryption, access controls, and auditing.

• Monitoring and Maintenance: Monitor data pipelines and infrastructure components, proactively identify and address issues, and perform routine maintenance tasks to ensure system stability and reliability.

• Collaboration: Collaborate closely with cross-functional teams, including data scientists, architects, and domain experts, to understand requirements, gather insights, and deliver integrated solutions.

• Documentation: Create comprehensive documentation, including technical specifications, data flow diagrams, and operational procedures, to facilitate understanding, collaboration, and knowledge sharing.

Qualifications:

• Proven experience as a Data Platform Engineer, with a track record of designing and implementing complex data pipelines and infrastructure solutions.
• Expertise in data modelling, ETL (Extract, Transform, Load) processes, and data warehousing concepts, with proficiency in SQL and scripting languages (e.g., Python)
• Strong hands-on experience with Data Bricks.
• Excellent analytical, problem-solving, and communication skills, with the ability to translate complex technical concepts into clear and actionable insights.
• Proven ability to work effectively in a fast-paced, collaborative environment, with a passion for innovation and continuous learning.
• Can explain in detail how you have designed, built and integrated data platforms.
• You also need to be able to explain in detail and show how many integrations points you have worked on, and your work with Platforms.

Desired candidate profile

• Data Engineering: Strong knowledge of data engineering principles and practices, including ETL (Extract, Transform, Load) processes.

• Programming Languages: Proficiency in programming languages such as Python, Java, or Scala, used for data manipulation and processing.

• Database Management: Experience with relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra).

• Big Data Technologies: Familiarity with big data tools and frameworks, such as Apache Hadoop, Spark, or Kafka.

• Cloud Platforms: Experience with cloud services (e.g., AWS, Google Cloud, Azure) for data storage and processing.

• Data Warehousing: Understanding of data warehousing concepts and experience with platforms like Snowflake, Redshift, or Google BigQuery.

• Data Modeling: Skills in designing data models that efficiently organize and structure data for analytics.

• Data Governance: Knowledge of data governance and data quality best practices to ensure data integrity and compliance.

• APIs and Microservices: Familiarity with building and integrating APIs to facilitate data access and sharing.

• Collaboration: Strong communication and collaboration skills to work with data scientists, analysts, and other stakeholders.