Senior Data Engineer

• olve challenging problems, using python coding skills.
• Design, build and launch new data extraction, transformation & loading processes in production.
• Web crawling, data cleaning, data annotation, data ingestion and data processing.
• Reading and collating complex data sets.
• Creating and maintaining data pipelines.
• Continual focus on process improvement to drive efficiency and productivity within the team.
• Use of Python, SQL, ES, Shell etc. to build the infrastructure required for optimal extraction, transformation, and loading of data.
• Provide insights into key business performance metrics by building analytical tools that utilize the data pipeline.
• Support the wider business with their data needs on an ad hoc basis.
  

Qualifications

To qualify, you must have

• 3+ years of programming experience, solid coding skills in Python, Shell, and Java
• Bachelor's degree in computer engineering, Computer Science, or Electrical Engineering and Computer Sciences.
• Good corporate capacity, good communication skills.
• Experience with Web crawling, cleaning.
• Experience with solution architecture, data ingestion, query optimization, data segregation, ETL, ELT, AWS, EC2, S3, SQS, lambda, Elastic Search, Redshift, CI/CD frameworks and workflows.
• Working knowledge of data platform concepts - data lake, data warehouse, ETL, big data processing (designing and supporting variety/velocity/volume), real time processing architecture for data platforms, scheduling and monitoring of ETL/ELT jobs
• PostgreSQL and programming (preferably Java, Python), proficiency in understanding data, entity relationships, structured & unstructured data, SQL and NoSQL databases
• Knowledge of best practice in optimizing columnar and distributed data processing system and infrastructure