Afzal

Overall 4+ years of experience in AWS Cloud services for ETL development in core big data projects using Python, Spark, AWS services.
Experienced in building data pipeline for Real-time data ingestion from relational database to cloud storage.
Worked on the latest Cloud computing tool Databricks for big data projects.
Built complicated data pipelines in AWS services to automate data ingestion related to medical insurance and patient details.
Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, Effective & efficient Joins, Transformations and others during ingestion process itself.
Effectively handled the complications for historical and incremental data in data ingestion process.
Worked multiple projects on Hadoop cluster, managed Hive tables correspondingly.
Gained experience in NiFi for automating data ingestions and creating data pipelines.

Build scheduled PySpark SQL data pipeline with complicated transformations in Managed Apache Airflow (MWAA).
Create new databases, EMR clusters, tables and other AWS infrastructures using serverless (sceptre).
Convert numerous SQL scripts to PySpark scripts.

Build real-time streaming data ingestion jobs from MongoDB to Delta Lake with Confluent Connectors, Kafka, and Databricks data pipeline.
Implemented various data cleaning and different levels of transformation for streaming data in Python and Spark.
Generated reports and analysis based on Delta Lake data in Python Spark and to process them on a schedule as Databricks job.
Developed AWS Lambda functions for alerts regarding high count variations and SLA miss for Databricks streaming jobs.

Build real-time streaming data Ingestion jobs from MongoDB to Delta Lake with Confluent Connectors - Kafka - Databricks data pipeline.
Implemented various data cleaning and different levels of transformation for streaming data in Python and Spark.
Generated reports and analysis based on Delta Lake data in Python Spark and to process them on a schedule as Databricks job.
Developed AWS Lambda functions for Alerts regarding high count variations and SLA miss for Databricks streaming jobs.

Build real-time streaming data Ingestion jobs from MongoDB to Delta Lake with Confluent Connectors - Kafka - Databricks data pipeline.
Implemented various data cleaning and different levels of transformation for streaming data in Python and Spark.
Generated reports and analysis based on Delta Lake data in Python Spark and to process them on a schedule as Databricks job.
Developed AWS Lambda functions for Alerts regarding high count variations and SLA miss for Databricks streaming jobs.

Data ingestion with complicated Spark data frames with joins and optimization of large datasets of patient information and their bill payments for medical insurance companies.
Implemented the above project in AWS as a data pipeline using the services AWS Lambda, AWS StepFunction, AWS DynamoDB, AWS Redshift, AWS Athena and AWS EMR.
Parse the transaction datasets containing credit card details, debit card details, gift card details etc in AWS Glue and store it in s3 location and generate report using custom queries in AWS Athena.
Scala Spark streaming project in Maven.

Data ingestion with complicated Spark data frames with joins and optimization of large datasets of patient information and their bill payments for medical insurance companies.
Implemented the above project in AWS as a data pipeline using the services AWS Lambda, AWS StepFunction, AWS DynamoDB, AWS Redshift, AWS Athena and AWS EMR. Parse the transaction datasets containing credit card details, debit card details, gift card details etc in AWS Glue and store it in s3 location and generate report using custom queries in AWS Athena.
Scala Spark streaming project in Maven.

Data ingestion with complicated Spark data frames with joins and optimization of large datasets of patient information and their bill payments for medical insurance companies.
Implemented the above project in AWS as a data pipeline using the services AWS Lambda, AWS StepFunction, AWS DynamoDB, AWS Redshift, AWS Athena and AWS EMR. Parse the transaction datasets containing credit card details, debit card details, gift card details etc in AWS Glue and store it in s3 location and generate report using custom queries in AWS Athena.
Scala Spark streaming project in Maven.