Akhil kumar

End-to-End Data Solutions Development

Design and implement data storage, integration, and processing solutions using Azure technologies.
Build scalable ETL pipelines and automate workflows.
Azure Data Engineering

Develop data frameworks and notebooks using Azure Databricks (PySpark, Spark SQL).
Implement Azure Data Factory (ADF) components such as Linked Services, Datasets, Pipelines, and Activities.
Handle data migration from On-Prem to Azure Cloud with Databricks using Spark API.
Data Processing & Optimization

Design and implement end-to-end data solutions (storage, integration, processing) in Azure.
Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, ADLS, Data warehouse.
Attend daily scrum calls and update the ADO user stories on daily basis.
Extensively involved in writing transforming data in PySpark and pushing into ADLS Gen2.
Experienced in data migration from On-Prem to Azure Cloud with Databricks using Spark API.
Designed and implemented scalable data ingestion pipelines using Azure Data Factory, integrating various data sources like SQL databases, CSV files, and REST APIs.
Responsible for estimating the cluster size, monitoring, and troubleshooting of the Spark data bricks cluster.
Implemented Incremental load strategy for loading on daily basis.
Build Complex distributed systems involving huge amount data handling, collecting metrics building data pipeline, and Analytics.
Work within and across Agile teams to design, implement, test and support technical solutions across a full stack of technologies.
Actively Participated in Daily Meetings to discuss the status and progress of the project.
Worked on Azure Devops to setup infrastructure and to build CI/CD pipelines the and deploy application

Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
Deployed data factory for creating data pipeline to orchestrate the data into data lake.
Implement end-to-end ETL processes ensuring efficient data movement and transformation with a focus on data quality and governance.
Automate processes and set up monitoring for efficiency and reliability.
Maintain comprehensive documentation for knowledge sharing and transparency.
Building pipelines from end-to-end development on modules/functionalities independently.
Optimize data processing workflows for performance and scalability.
Collaborate with stakeholders including data scientists and analysts to understand and meet data requirements.