Upendra Kumar

• Implemented Azure Data Factory (ADF) extensively for ingesting data from different source systems like relational and unstructured data to meet business functional requirements.
• Created numerous pipelines in Azure using Azure Data Factory v2 to get the data from disparate source systems by using different Azure Activities like Move &Transform, Copy, filter, for each, Databricks etc.
• Maintain and provide support for optimal pipelines, data flows and complex data transformations and manipulations using ADF and PySpark with Databricks.
• Automated jobs using different triggers like Events, Schedules and Tumbling in ADF.
• Created, provisioned different Databricks clusters, notebooks, jobs and autoscaling.
• Performed data flow transformation using the data flow activity.
• Used Polybase to load tables in Azure Synapse.
• Implemented Azure, self-hosted integration runtime in ADF.
• Improved performance by optimizing computing time to process the streaming data by optimizing the cluster run time.
• Perform ongoing monitoring, automation, and refinement of data engineering solutions.
• Scheduled, automated business processes and workflows using Azure Logic Apps.
• Designed and developed a new solution to process the NRT data by using Azure stream analytics, Azure EventHub, and Service Bus Queue.
• Created Linked services to connect the external resources to ADF.
• Worked with complex SQL views, Stored Procedures, Triggers, and packages in large databases from various servers.
• Used Azure DevOps pipelines to build and deploy different resources (Code and Infrastructure) in Azure.
• Ensure the developed solutions are formally documented and signed off by business.
• Worked with team members to resolve any technical issue, Troubleshooting, Project Risk & Issue identification, and management.
• Worked on the cost estimation, billing, and implementation of services on the cloud.
• Work closely across teams (Support, Solution Architecture) and peers to establish and follow best practices while solving customer problems.

Implemented Azure Data Factory (ADF) extensively for ingesting data from different source systems like relational and unstructured data to meet business functional requirements.
Created numerous pipelines in Azure using Azure Data Factory v2 to get the data from disparate source systems by using different Azure Activities like Move & Transform, Copy, filter, for each, Databricks etc.
Maintain and provide support for optimal pipelines, data flows and complex data transformations and manipulations using ADF and PySpark with Databricks.
Automated jobs using different triggers like Events, Schedules and Tumbling in ADF.
Created, provisioned different Databricks clusters, notebooks, jobs and autoscaling.
Performed data flow transformation using the data flow activity.
Used Polybase to load tables in Azure Synapse.
Implemented Azure, self-hosted integration runtime in ADF.
Improved performance by optimizing computing time to process the streaming data by optimizing the cluster run time.
Perform ongoing monitoring, automation, and refinement of data engineering solutions.
Scheduled, automated business processes and workflows using Azure Logic Apps.
Designed and developed a new solution to process the NRT data by using Azure stream analytics, Azure EventHub, and Service Bus Queue.
Created Linked services to connect the external resources to ADF.
Worked with complex SQL views, Stored Procedures, Triggers, and packages in large databases from various servers.
Used Azure DevOps pipelines to build and deploy different resources (Code and Infrastructure) in Azure.
Ensure the developed solutions are formally documented and signed off by business.
Worked with team members to resolve any technical issue, Troubleshooting, Project Risk & Issue identification, and management.
Worked on the cost estimation, billing, and implementation of services on the cloud.
Work closely across teams (Support, Solution Architecture) and peers to establish and follow best practices while solving customer problems.

Attended requirement calls and worked with Business Analyst and Solution Architects to understand the requirements.
Created pipelines in Azure using ADF to get the data from different source systems and transform the data by using many activities.
Design and developed Batch processing and real-time processing solutions using ADF, Databricks clusters and stream Analytics.
Experience in designing, developing, and implementing ETL pipelines using Azure Databricks.
Ingested huge volume and variety of data from disparate source systems into Azure DataLake Gen2 using Azure Data FactoryV2.
Created reusable pipelines in Data Factory to extract, transform and load data into Azure SQL DB and SQL Data warehouse.
Implemented both ETL and ELT architectures in Azure using Data Factory, Databricks, SQL DB and SQL Data warehouse.
Proficiency in using Apache Spark and PySpark to process large datasets, including data ingestion, transformation, and aggregation.
Proficiency in using Delta Lake with various data formats, including Parquet, Avro, JSON, and CSV, and experience in reading and writing data from/to Delta tables using Databricks notebooks and Spark SQL.
Experience in using Databricks Delta Lake, a scalable and performance storage layer for Delta tables, which provides ACID Transactions, schema enforcement, and time travel capabilities.
Created, provisioned multiple Databricks clusters needed for batch and continuous streaming data processing and installed the required libraries for the clusters.
Experienced in developing audit, balance and control framework using SQL DB audit tables to control the ingestion, transformation, and load process in Azure.
Solid experience in Data Warehousing best practices working with Metadata, repositories, and experience within adisciplined lifecycle methodology.
Managing Databricks Notebooks Delta Lake with Python Delta Lake with Spark SQL
Developed and executed migration strategies to move workloads from on-premises or other cloud platforms to Azure, leveraging OCI for supporting components.
Used Azure Logic Apps to develop workflows which can send alerts/notifications on different jobs in Azure.
Used Azure Devops to build and release different versions of code in different environments.
Well-versed with Azure authentication mechanisms such as Service principal, Managed Identity, Key vaults.
Created External tables in Azure SQL Database for data visualization and reporting purposes.
Worked with complex SQL views, Stored Procedures, Triggers, and packages in large databases from various servers.

Involved with architecture team and data analysts for processing customer related user data based on the business requirements.
Actively working on technologies like AWS Glue, S3, Cloudera, Spark, Scala, Python, pyspark, Hive, snowflake.
Created Python job to create or delete the AWS resource like EC2, Athena table, Glue Job etc on ad-hoc request.
Captured the data of insurance members and analyse the interest of members in the insurance process.
Create S3 Buckets to store the raw data and to maintain the ETL scripts and Written Glue jobs to ingest the historical data into the Athena tables.
Developed spark programs with python and applied principles of functional programming to process complex unstructured and structured data sets on top of EMR.
Created Airflow DAGs to schedule the ingestions, ETL jobs and various business reports.
Support Production Environment and debug issues using Splunk logs.
Stage the API or Kafka Data (in JSON file format) into Snowflake Database by flattening the same for different functional services.
Created data ingestion framework in snowflake for both Batch and Real-time data from different file format (XML, JSON, Avro) using snowflake stage and Snowflake Data Pipe.