Data Engineering Lead

Roles and responsibilities

• Overall 5-7 years of experience in data engineering and transformation on Cloud
• 3+ Years of Very Strong Experience in Azure Data Engineering, Databricks
• Expertise in supporting/developing lakehouse workloads at enterprise level
• Experience in pyspark is required – developing and deploying the workloads to run on the Spark distributed computing
• Candidate must possess at least a Graduate or bachelor’s degree in Computer Science/Information Technology, Engineering (Computer/Telecommunication) or equivalent.
• Cloud deployment: Preferably Microsoft azure
• Experience in implementing the platform and application monitoring using Cloud native tools

• Team Leadership and Management:

  • Team Supervision: Lead and mentor a team of data engineers to ensure high performance, professional growth, and alignment with organizational goals.
  • Task Allocation: Assign tasks and manage workloads effectively across the team, ensuring timely delivery of projects while maintaining quality.
  • Collaboration: Work closely with cross-functional teams (e.g., data scientists, analysts, business stakeholders) to ensure the data infrastructure supports their needs and objectives.
  • Training and Development: Provide guidance, training, and upskilling opportunities to junior and mid-level data engineers.

• Data Architecture Design:

  • Pipeline Development: Design and implement scalable, reliable, and efficient data pipelines to process and transform large datasets from various sources.
  • Architecture Strategy: Lead the design and architecture of data systems and solutions, ensuring data is easily accessible, clean, and properly formatted for analytics.
  • Data Modeling: Work on data models, including data warehouses, lakes, and other data storage solutions, and design the schema that supports efficient querying and reporting.

• Data Integration and ETL:

  • ETL Processes: Lead the development of Extract, Transform, Load (ETL) processes to integrate data from multiple sources, ensuring data quality, consistency, and reliability.
  • Data Quality Assurance: Ensure that the data pipelines and processes meet high standards of data accuracy, consistency, and timeliness. Establish data validation processes and error handling mechanisms.

• Data Warehousing and Storage:

  • Data Storage Solutions: Lead efforts in selecting and implementing data storage solutions, such as data lakes, data warehouses, and cloud storage platforms (e.g., Amazon Redshift, Google BigQuery, Snowflake).
  • Scalability and Performance: Design and optimize data storage systems for performance, scalability, and cost-efficiency, taking into account future growth and business requirements.
  • Database Management: Oversee the management of relational and non-relational databases, ensuring optimal performance and security.

Desired candidate profile

• Technical Expertise:

  • Expertise in data engineering concepts, such as ETL, data warehousing, data lakes, and data pipelines.
  • Proficiency in programming languages such as Python, Java, Scala, SQL, and frameworks like Apache Spark or Apache Flink for big data processing.
  • Strong experience with cloud platforms (AWS, Google Cloud, Azure) and associated services like S3, Redshift, BigQuery, EMR, and Databricks.
  • Familiarity with containerization tools (e.g., Docker, Kubernetes) for orchestrating and deploying data pipelines.

• Database and Storage Knowledge:

  • Deep knowledge of relational and NoSQL databases, including MySQL, PostgreSQL, MongoDB, Cassandra, and HBase.
  • Familiarity with data storage solutions, including Data Lakes (e.g., AWS S3, Google Cloud Storage), Data Warehouses (e.g., Snowflake, BigQuery), and distributed file systems like HDFS.

• Leadership and Management:

  • Experience managing and leading a team of data engineers, with strong organizational, mentoring, and communication skills.
  • Ability to manage multiple projects simultaneously, ensuring deadlines are met without compromising quality.
  • Conflict resolution, team-building, and performance management skills.

• Data Modeling and Architecture:

  • Expertise in data modeling techniques and building scalable data architectures.
  • Ability to design efficient data models that support analytics, business intelligence (BI), and machine learning (ML) use cases.

• Problem Solving and Optimization:

  • Strong analytical and problem-solving skills to optimize data pipelines and ensure efficient data processing and storage.
  • Experience in debugging and troubleshooting complex data issues.

• Soft Skills:

  • Excellent communication skills to interact with both technical and non-technical stakeholders.
  • Strategic thinking to understand business needs and align technical solutions with those needs.
  • Ability to work in a fast-paced, collaborative environment and manage competing priorities.

• Agile Methodologies:

  • Familiarity with agile methodologies (e.g., Scrum, Kanban) for managing projects and ensuring iterative delivery of data solutions.