نبذة عني
Professional Big Data Engineer with 5 years of industry experience in Architecture, analysis, design, development, implementation and support in developing strategic methods for deploying Big Data Technologies to efficie…
Professional Big Data Engineer with 5 years of industry experience in Architecture, analysis, design, development, implementation and support in developing strategic methods for deploying Big Data Technologies to efficiently solve business requirements. Expertise in Hadoop, Python, Java, Scala, Spark, Kafka, Sqoop, Hive, NoSQL databases, automation tools like Ansible, Solr and software design process. Outstanding communication skills and maintain up-to-date IT skills and industry knowledge.
الخبرة
Senior Data Engineer
Implemented Univariate using Java and Bivariate Analysis using Scala, Python ML.
Developed code for Fuzzy String match using NLP in Python.
Implemented SaaS & PaaS architecture where user creates EC2, EMR instances, Job, scheduling, etc.
Optimized code performance to extract metadata schema from database using Multiprocessing(Java).
Experience in Hadoop cluster automated installation using Ansible-Playbook.
Implemented two different search engines in SOLR for the two different use case.
Extensively worked in Data Integration like Sqoop, Spark, Talend, NiFi using Java, Python and improved performance in applications.
Designed and implemented real-time data pipeline to process data from 3 different sources like Oracle, Mysql, DB2 using Scala Spark and Kafka and stored in HDFS.
Improved application frequent accessed dictionary data with Redis in-memory database using Java, thereby speeding up the application.
Designed and implemented data loading in Apache Spark and increased performance by 60%.
Contributed to internal activities for overall process improvements, efficiencies and innovation.
Software Engineer
Implementation of HikariCP using java to improve the connection performance for number of request.
Orchestrated efficient large-scale software deployments, including testing features and correcting code.
Worked closely with software development and testing team members to design and develop robust solutions to meet client requirements for functionality, scalability and performance.
Data integration and quality check done in data and worked on data lineage.
Developed Web scraping tool using Java to get the data from customer site.
Developed code to convert all lat and long into address.
Retrieved the data from the HBASE required for the Models using Python spark.
Fuzzy search algorithm to find the misspelled words using Python.
Wrote shell scripts for daily maintenance activities, including indexes and tables analyses.
Implemented ETL (Extract, Transform, Load) tasks, maintaining data integrity and verifying pipeline stability.
Senior Data Engineer
Implemented Univariate using Java and Bivariate
Analysis using Scala, Python ML.
Developed code for Fuzzy String match using NLP in
Python.
Implemented SaaS & PaaS architecture where user
creates EC2, EMR instances, Job, scheduling, etc.
Optimized code performance to extract metadata schema
from database using Multiprocessing(Java).
Experience in Hadoop cluster automated installation
using Ansible-Playbook.
Implemented two different search engines in SOLR for the
two different use case.
Extensively worked in Data Integration like Sqoop, Spark,
Talend, NiFi using Java, Python and improved
performance in applications.
Designed and implemented real-time data pipeline to
process data from 3 different sources like Oracle, Mysql,
DB2 using Scala Spark and Kafka and stored in HDFS.
Improved application frequent accessed dictionary data
with Redis in-memory database using Java, thereby
speeding up the application.
Designed and implemented data loading in Apache Spark
and increased performance by 60%.
Contributed to internal activities for overall process
improvements, efficiencies and innovation
Software Engineer
Implementation of HikariCP using java to improve the
connection performance for number of request.
Orchestrated efficient large-scale software deployments,
including testing features and correcting code.
Worked closely with software development and testing
team members to design and develop robust solutions to
meet client requirements for functionality, scalability and
performance.
Data integration and quality check done in data and
worked on data lineage.
Developed Web scraping tool using Java to get the data
from customer site.
Developed code to convert all lat and long into address.
Retrieved the data from the HBASE required for the
Models using Python spark.
Fuzzy search algorithm to find the misspelled words
using Python.
Wrote shell scripts for daily maintenance activities,
including indexes and tables analyses.
Implemented ETL (Extract, Transform, Load) tasks,
maintaining data integrity and verifying pipeline stability