Associate Data Engineer

Roles and responsibilities

We are in search of a Professional Data Engineer to enhance our data engineering capabilities. In this role, you will support junior team members, driving the development and maintenance of sophisticated data solutions. Beyond that, you will contribute in building and maintaining the infrastructure necessary for data engineering processes. Collaborating with cross-functional teams, you will contribute to building self-service data layers and provide crucial insights to align data architecture with broader business strategies. Your contribution towards innovative solutions and ensuring the highest standards of data quality.

We work with cutting-edge technologies to power our data and engineering workflows. Our stack includes SQL and Python for programming, with Dagster (and Airflow for legacy pipelines) as our orchestration tool. We store and process data using Snowflake, Redshift, Clickhouse, and S3. On the infrastructure side, we leverage Docker and Kubernetes for containerization and orchestration, along with AWS services like ECS/EKS, DMS, Kinesis, Glue, Athena, and more. Our ETL processes are streamlined with FiveTran and DBT for seamless data transformations. For Infrastructure as Code, we rely on Terraform with Terragrunt, ensuring robust and scalable deployments.

Key Responsibilities:

• Design and implement complex ETL solutions for large datasets.
• Optimise and fine-tune SQL and/or Python code for performance.
• Design and maintain data models for efficient storage and retrieval.
• Troubleshoot and resolve issues related to data pipeline performance.
• Build and maintain self-service datasets to accelerate data democratization efforts.
• Contribute to the initiatives for managing and optimizing infrastructure.
• Implement best practices for version control and collaboration in Terraform.
  

The Person

Desired Qualifications:

• 7+ years of experience as a data engineer
• Proficiency in SQL and Python
• Experience with modern cloud data warehousing, data lake solutions like Snowflake, BigQuery, Redshift.
• Experience with ETL/ELT, batch, streaming data processing pipelines.
• Experience with AWS services (like S3, DMS, Glue, Athena, EKS, etc.)
• Proven experience with data warehousing and dimensional data modelling guidelines and best practices
• Familiar with ETL tools like FiveTran, DBT, Airbyte, etc
• Experience with orchestration tools like Dagster, Airflow, AWS Step functions, etc.
• Knowledge of CI/CD pipelines and automation.

Desired candidate profile

Desired Qualifications:

• 7+ years of experience as a data engineer
• Proficiency in SQL and Python
• Experience with modern cloud data warehousing, data lake solutions like Snowflake, BigQuery, Redshift.
• Experience with ETL/ELT, batch, streaming data processing pipelines.
• Experience with AWS services (like S3, DMS, Glue, Athena, EKS, etc.)
• Proven experience with data warehousing and dimensional data modelling guidelines and best practices
• Familiar with ETL tools like FiveTran, DBT, Airbyte, etc
• Experience with orchestration tools like Dagster, Airflow, AWS Step functions, etc.
• Knowledge of CI/CD pipelines and automation.
  

Other Capabilities:

• Experience with Terraform and Terragrunt.
• Experience with data stores like MySQL, MongoDB, DynamoDB, Aurora.
• Experience with pub-sub, queuing, and streaming frameworks such as AWS Kinesis, Kafka, SQS, SNS.
• Experience in real time processing engines like Apache Flink, Apache Spark etc.
• Familiar with BI tools like Tableau, QuickSight, Tableau, PowerBI, MicroStrategy, etc
• Familiar with Tagging and tracking tools like Snowplow, Tealium,Segment.io etc
• Experience with Realtime analytics solutions like Clickhouse, Rockset, Pinot, or Druid.
• Experience with GCP and Google Analytics.