Vineeth

Completed a data migration from antiquated databases and implemented an ETL workflow using Databricks tables and Azure Synapse.
Built complex ETL jobs that transform data visually with data flows or by using compute services Azure Databricks and Azure SQL Database.
Engineered, designed, developed, and advanced Query Processing and Self-Tuning functionality using Synapse SQL, including integration with Snowflake for data processing and querying.
Created pipelines in ADF using Linked Services, Datasets, and Pipeline to extract, transform, and load data from Azure SQL, Blob storage, Azure SQL Data Warehouse, write-back tool, and backwards.
Led multiple Azure Big Data and data transformation implementations in Banking and Financial Services, High Tech, and Utilities industries.
Implemented large Lambda architectures using Azure Data platform capabilities like Azure Data Lake, Azure Data Factory, HDInsight, Azure SQL Server, Azure ML, and Power BI.
Implemented cloud based Linux OS in AWS to develop scalable applications with Python.
Created data ingestion systems to pull data from traditional RDBMS platforms and store it in NoSQL databases such as MongoDB.
Worked on continuous integration and continuous deployment solutions using Git, Jenkins, and Dockers to setup and configure big data architecture on the AWS cloud platform.
Worked on Azure Data Factory and designed and deployed scalable data intake pipelines for consuming data from SQL databases, CSV files, and REST APIs.
Used complex SQL queries to extract data from Oracle and Teradata databases for analysis and reporting.
Designed and implemented Azure ADF pipelines, incorporating lift and shift methodologies, and leveraging ADF and SSIS packages for data loading into Azure databases.
Established connections between different sources, both on-premises and in the cloud, to serve as the unified source for Power BI reports.
Wrote complex PL/SQL queries and procedures to extract, transform, and load data from various sources, ensuring data accuracy and completeness.
Used Spark and Spark-SQL to read the parquet data and create the tables in Hive using the Scala API.
Utilized Waterfall methodology for team and project management.
Used JIRA as the Scrum tool for the Scrum task board and work on incidents.
Involved in loading data from the Linux file system to Hadoop Distributed File System (HDFS) and setting up HIVE, PIG, HBASE, and SQOOP on Linux/Solaris Operating System.

 Build complex ETL jobs that transform data visually with data flows or by using compute services Azure Data bricks, and Azure SQL Database.
 Engineered, designed, developed, and advanced Query Processing and Self-Tuning functionality using Synapse SQL, including integration with Snowflake for data processing and querying.
 Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
 Strong experience of leading multiple Azure Big Data and Data transformation implementations in Banking and Financial Services, High Tech and Utilities industries.
 Implemented large Lambda architectures using Azure Data platform capabilities like Azure Data Lake, Azure Data Factory, HDInsight, Azure SQL Server, Azure ML and Power BI.
 Experience implementing Cloud based Linux OS in AWS to Develop Scalable Applications with Python.
 Created data ingestion systems to pull data from traditional RDBMS platforms and store it in NoSQL databases such as MongoDB.
 Worked on continuous integration and continuous deployment of (CI/CD) solutions, using Git, Jenkins, and Dockers to setup and configure big data architecture on the AWS cloud platform.
 Worked on Azure Data Factory, designed and deployed scalable data intake pipelines for consuming data from a variety of sources, including SQL databases, CSV files, and REST APIs.
 Used complex SQL queries to extract data from Oracle and Teradata databases for analysis and reporting.
 Designed and implemented Azure ADF pipelines, incorporating lift & shift methodologies, and leveraging ADF and SSIS packages for data loading into Azure databases.
 Established connections between different sources, both on-premises and in the cloud, to serve as the unified source for Power BI reports.
 Writing complex PL/SQL queries and procedures to extract, transform, and load data from various sources, ensuring data accuracy and completeness.
 Used Spark and Spark-SQL to read the parquet data and create the tables in Hive using the Scala API.
 Utilized Waterfall methodology for team and project management. Used JIRA as the Scrum Tool for the Scrum Task board and work on incidents
 Involved in loading data from the Linux file system to Hadoop Distributed File System (HDFS) and setting up HIVE, PIG, HBASE, and SQOOP on Linux/Solaris Operating System.

Extracted, transformed, and loaded data from source systems to Azure Data Storage services using Azure Data Factory, T-SQL, Spark SQL, and U-SQL.
Performed data ingestion to Azure services including Azure Data Lake, Azure Storage, Azure SQL, and Azure DW, and processed the data in Azure Databricks.
Optimized Spark application code to improve the performance of overall pipelines by implementing performance tuning techniques in Databricks.
Helped individual teams set up repositories in Bitbucket and maintain their code and jobs for CI/CD environment.
Wrote UDF in Scala and PySpark to meet specific business requirements.
Analyzed large data sets using Hive queries.
Monitored daily incremental loads from various RDBMSs such as MongoDB, MS SQL, and MySQL.
Worked on evaluating and comparing different tools for test data management with Hadoop.
Analyzed large and critical datasets using Cloudera, HDFS, MapReduce, Hive, Hive UDF, Pig, Sqoop, and Spark.
Worked on developing and setting up Snowpipe to ingest files in near real time to Snowflake tables from Data Lake.
Used Kafka producer to ingest raw data into Kafka topics and ran Spark Streaming app to process click stream events.
Implemented continuous integration and deployment pipelines through Jenkins to automate Hadoop job deployment and managed Hadoop clusters with Cloudera.
Used Spark Streaming to receive real time data from Kafka and store the stream data to HDFS using Python and databases such as HBase.
Worked on Spark SQL for joining multi hive tables and writing them to a final Hive table and stored them on S3.
Managed project workflows using JIRA and maintained version control using Git.
Created UNIX shell scripts to load data from flat files into Oracle tables.