Arun Kumar

Over 8+ years of experience where 5+ years as Data Engineer with expertise in designing data intensive applications
using Hadoop Ecosystem, Big Data Analytical, Cloud Data engineering, Data Warehouse / Data Mart, Data
Visualization, Reporting, and Data Quality solutions in the domains of Accounting, Finance and Software and 3+ years of
experience as Project Coordinator.
• Extensive experience in Hadoop led development of enterprise level solutions utilizing Hadoop components such
as Apache Spark, MapReduce, HDFS, Sqoop, Hive, HBase, Oozie, Flume, NiFi, Kafka, Zookeeper, and YARN.
• Implemented the security requirements for Hadoop and integrating with Kerberos authentication infrastructureKDC server setup, creating realm /domain, managing.
• Experience of Partitions, bucketing concepts in Hive and designed both Managed and External tables in Hive to
optimize performance.
• Experience with the Spark improving the performance and optimization of the existing algorithms in Hadoop
using Spark Context, Spark-SQL , Dataframe API , Spark Streaming, MLlib , Pair RDD 's and worked explicitly
on PySpark and Scala.
• Experience in building data pipelines using Azure Data factory, Azure databricks and loading data to Azure Data
Lake Store(ADLS), Azure SQL Database, Azure SQL Data warehouse and controlling and granting database
access.
• Hands-on experience in handling database issues and connections with SQL and NoSQL databases such
as MongoDB, HBase, Cassandra, SQL server, and PostgreSQL.
• Experience in designing and creating RDBMS Tables, Views, User Created Data Types, Indexes, Stored
Procedures, Cursors, Triggers and Transactions.
• Expert in designing ETL data flows using creating mappings/workflows to extract data from SQL Server and Data
Migration and Transformation from Oracle/Access/Excel Sheets using SQL Server SSIS.
• Expert in designing Parallel jobs using various stages like Join, Merge, Lookup, remove duplicates, Filter,
Dataset, Lookup file set, Complex flat file, Modify, Aggregator, XML.
• Experience with Amazon EC2, Amazon S3, Amazon RDS, VPC, IAM, Amazon Elastic Load Balancing, Auto
Scaling, Cloud Watch, SNS, SES, SQS, Lambda, EMR and other services of the AWS family.
• Created and maintained CI/CD (continuous integration & deployment) pipelines and apply automation to
environments and applications. Worked on various automation tools like GIT, Terraform, Ansible.
• Experienced with JSON based RESTful web services, and XML/QML based S

Designed and setup Enterprise Data Lake to provide support for various uses cases including Analytics, processing, storing and Reporting of voluminous, rapidly changing data.
Responsible for maintaining quality reference data in source by performing operations such as cleaning, transformation and ensuring Integrity in a relational environment by working closely with the stakeholders & solution architect.
Designed and developed Security Framework to provide fine grained access to objects in AWS S3 using AWS Lambda, DynamoDB.
Responsible for the turnaround time of fixing the errors and incidents by creating the automated production deployments.
Set up and worked on Kerberos authentication principles to establish secure network communication on cluster and testing of HDFS, Hive, Pig and MapReduce to access cluster for new users.
Performed end-to-end Architecture & implementation assessment of various AWS services like Amazon EMR, Redshift, S3.
Used AWS EMR to transform and move large amounts of data into and out of other AWS data stores and databases, such as Amazon Simple Storage Service (Amazon S3) and Amazon DynamoDB.
Used Spark SQL for Scala & amp, Python interface that automatically converts RDD case classes to schema RDD.
Import the data from different sources like HDFS/HBase into Spark RDD and perform computations using PySpark to generate the output response.
Creating Lambda functions with Boto3 to deregister unused AMIs in all application regions to reduce the cost for EC2 resources.
Importing & exporting database using SQL Server Integrations Services (SSIS) and Data Transformation Services (DTS Packages).
Coded Teradata BTEQ scripts to load, transform data, fix defects like SCD 2 date chaining, cleaning up duplicates.
Developed reusable framework to be leveraged for future migrations that automates ETL from RDBMS systems to the Data Lake utilizing Spark Data Sources and Hive data objects.
Conducted Data blending, Data preparation using Alteryx and SQL for Tableau consumption and publishing data sources to Tableau server.
Developed Kibana Dashboards based on the Log stash data and Integrated different source and target systems into Elasticsearch for near real time log analysis of monitoring End to End transactions.
Implemented AWS Step Functions to automate and orchestrate the Amazon SageMaker related tasks such as publishing data to S3, training ML model and deploying it for prediction.
Integrated Apache Airflow with AWS to monitor multi-stage ML workflows with the tasks running on Amazon SageMaker.

Worked on Azure Data Factory to integrate data of both on-prem (MY SQL, Cassandra) and cloud (Blob storage, Azure SQL DB) and applied transformations to load back to Azure Synapse.
Managed, Configured and scheduled resources across the cluster using Azure Kubernetes Service.
Monitored Spark cluster using Log Analytics and Ambari Web UI.
Transitioned log storage from Cassandra to Azure SQL Data warehouse and improved the query performance.
Involved in developing data ingestion pipelines on Azure HDInsight Spark cluster using Azure Data Factory and Spark SQL.
Also Worked with Cosmos DB (SQL API and Mongo API).
Extensively utilized Databricks notebooks for interactive analysis utilizing Spark APIs.
Used Polybase to load tables in Azure synapse.
Performed information purging and applied changes utilizing Databricks and Spark information analysis.
Responsible for error rate by minimizing the disruptions to the users during deployment and by lowering the errors or failures occurs during the deployment.
Develop dashboards and visualizations to help business users analyze data as well as providing data insight to upper management with a focus on Microsoft products like SQL Server Reporting Services (SSRS) and Power BI.
Performed the migration of large data sets to Databricks (Spark), create and administer cluster, load data, configure data pipelines, loading data from ADLS Gen2 to Databricks using ADF pipelines.
Created various pipelines to load the data from Azure data lake (ADLS) into Staging SQLDB and followed by to Azure SQL DB.
Analyzed SQL scripts and designed the solutions to implement using PySpark.
Created Databrick notebooks to streamline and curate the data for various business use cases and also mounted blob storage on Databrick.
Utilized Azure Logic Apps to build workflows to schedule and automate batch jobs by integrating apps, ADF pipelines, and other services like HTTP requests, email triggers etc.
Worked extensively on Azure data factory including data transformations, Integration Runtimes, Azure Key Vaults, Triggers and migrating data factory pipelines to higher environments using ARM Templates.
Ingested data in mini-batches and performs RDD transformations on those mini-batches of data by using Spark Streaming to perform streaming analytics in Data bricks.

Evaluated and secured vendor contracts for projects through third party vendors.
Worked with users and department heads in determining project requirements.
Planned, organized and implemented all aspects of projects.
Consistently meet or exceed timeline and budget constraints for all projects.
Managed an average of 8 projects monthly.
Tracked and communicated project status to upper management through weekly Excel reports.
Facilitated conference calls with internal departments to review goals and manage progress.
Recognized by operations department for providing excellent customer service

Developed and maintained project schedules.
Utilized Microsoft Project server platform solution to provide support for resource management such as for planning, reporting, and project team collaboration as using it as a centralized and unified repository for storing all project and resource data.
Updated project plans (simple and master schedules).
Used excel for forecasting, compare planned vs. actual hours, gathers financials and cost estimates for projects, and analyze and report financial data.
Supported network security team to identify and guard against threats to Lowe’s.
Prepared and/or edited meeting agendas/minutes, presentations, and tables.
Facilitated project meetings, created project documentation.
Created project calendars and organized conference calls.