Kavya

Around 5 years of work experience in IT consisting of Data Analytics Engineering, Visualization and database programming.
Experience in Business and Data Analysis, Data Profiling, Data Migration, Data Conversion, Data Quality, Data Integration and Metadata Management Services and Configuration Management.
Strong experience in Data Modeling with expertise in creating Star Snow-Flake Schemas, FACT and Dimensions Tables, Physical and Logical Data Modeling.

Worked on gathering security (equities, options, derivatives) data from different exchange feeds and storing historical data.
Designed and deployed a Kubernetes-based containerized infrastructure for data processing and analytics, leading to a 20% increase in data processing capacity.
Set up base Python structure with the create python-App package, SRSS, PySpark.
Wrote data ingestion systems to pull data from traditional RDBMS platforms such as Oracle and Teradata and store it in NoSQL databases such as MongoDB.
Designed and build scalable data pipelines to ingest, translate, and analyze large sets of data.
Implemented airflow for workflow automation and scheduling tasks and created DAGs tasks.
Responsible for Building and Testing of applications.
Experience in handling database issues and connections with SQL and NoSQL databases like MongoDB by installing and configuring various packages in python (Teradata, MySQL, MySQL connector, PyMongo and SQLAlchemy).
Pipelines were created in Azure Data Factory utilizing Linked Services to extract, transform, and load data from many sources such as Azure SQL Data warehouse, write-back tool, and backwards.
Imported real time weblogs using Kafka as a messaging system and ingested the data to Spark Streaming and did data quality checks using Spark Streaming and arranged bad and passable flags on the data.
Experience in creating Kubernetes replication controllers, Clusters and label services to deployed Microservices in Docker.
Developed JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data using the Sql Activity and creating UNIX shell scripts for database connectivity and executing queries in parallel job execution.
Consult leadership/stakeholders to share design recommendations and thoughts to identify product and technical requirements, resolve technical problems and suggest Big Data based analytical solutions.
Responsible for estimating the cluster size, monitoring, and troubleshooting of the Spark Databricks cluster and Ability to apply the spark Data Frame API to complete Data manipulation within spark session.
Instantiated, created, and maintained CI/CD (continuous integration & deployment) pipelines and apply automation to environments and applications.
Worked on various automation tools like GIT, Terraform, and Ansible.
Responsible for loading the data from BDW Oracle database, Teradata into HDFS using Sqoop.
Implemented AJAX, JSON, and Java script to create interactive web screens.
Designing and implementing data integration solutions using Azure Data Factory to move data between various data sources, including on-premises and cloud-based systems.
Used AWS to create storage resources and define resource attributes, such as disk type or redundancy type, at the service level.
Strong at testing and debugging tested the applications, Rest APIs using Pytest, Unit-test, requests libraries.
Implemented automated Data pipelines for Data migration, ensuring a smooth and reliable transition to the Cloud environment.
Supported development of Web portals, completed Database Modelling in PostgreSQL, front end support in HTML/CSS, jQuery.
Used Pig as ETL tool to do Transformations with joins and pre-aggregations before storing the data onto HDFS and assisted Manager by providing automation strategies, Selenium/Cucumber Automation and JIRA reports.
Ensured data quality and accuracy with custom SQL and Hive scripts and created data visualizations using Python and Tableau for improved insights and decision-making.
Automated and monitored AWS infrastructure with Terraform for high availability and reliability, reducing infrastructure management time by 90% and improving system uptime.
Analyzed data using SQL, Python, Apache Spark and presented analytical reports to management and technical teams.
The AWS Lambda functions were written in Spark with cross-functional dependencies that generated custom libraries for delivering the Lambda function in the cloud.
Performed raw data ingestion into, which triggered a lambda function and put refined data into ADLS.

Looked into existing Java/Scala spark processing and maintained, enhanced the jobs.
Analyzed and developed a modern data solution with Azure PaaS service to enable data visualization.
Understood the application's current Production state and the impact of new installation on existing business processes.
Working on query languages such as SQL, code languages such as Python or C# and scripting languages such as PowerShell, M-Query (Power Query), or Windows batch commands.
Using python NLTK tool kit to be used in smart MMI interactions.
Ability to apply the spark Data Frame API to complete data manipulation with in sparksession.
Handled various data sources, performed importing transformations using Hive, MapReduce, loaded data into HDFS and Extracted the data from SQL into HDFS using Sqoop.
Part of the Data and reporting team creating insights and Visualization for the business to make decisions on.
Built scalable data infrastructure on cloud platforms, such as AWS and GCP, using Kubernetes and Docker.
Well versed with various aspects of ETL processes used in loading and updating Oracle data warehouse.
Created pipelines to load the data using ADF.
Analyzed the SQL scripts and designed it by using Spark SQL for faster performance.
Creating job flow using Airflow in python and automating the jobs.
Airflow will have separate stack for developing DAGs on and will run jobs on EMR or EC2 Cluster.
Written queries in MySQL and Native SQL.
Used Azure Data factory to ingest data from log files and business custom applications, processed data on Data bricks per day-to-day requirements, and loaded them to Azure Data Lakes.
Used Kafka functionalities like distribution, partition, replicated commit log service for messaging systems by maintaining feeds.
Involved in loading data from rest endpoints to Kafka.
Deployed models as python package, as API for backend integration and as services in a microservices architecture with a Kubernetes orchestration layer for the Dockers containers.
Used Python to write Data into JSON files for testing Django Websites, Created scripts for data modelling and data import and export.
Led requirement gathering, business analysis, and technical design for Hadoop and Big Data projects.
Instantiated, created, and maintained CI/CD continuous integration & deployment pipelines and apply automation to environments and applications.
Collected and aggregated large amounts of web log data from different sources such as webservers, mobile and network devices using Apache Flume and stored the data into HDFS for analysis.
Automated AWS infrastructure to be deployed periodically using Docker images and CloudFormation templates.

Controlling and granting database access and migrating on premise databases to Azure data lake store using Azure Data Factory.
Enhanced by adding Python XML SOAP request/response handlers to add accounts, modify trades and security updates.
Involved in database migration methodologies and integration conversion solutions to convert legacy ETL processes into Azure Synapse compatible architecture.
Created clusters to classify control and test groups.
Created Complex Stored Procedures, Slow Changing Dimension Type 2, Triggers, Functions, Tables, Views and other T SQL code and SQL joins to ensure efficient data retrieval.
Worked on Angular JS to augment browser applications with MVC capability.
Build and deployed the code Artefacts into the respective environments in the Confidential Azure cloud.
Installed, configured, administered, monitored Azure, IAAS and PAAS, AzureAD, Azure VMs, Networking (VNet’s, Load Balancers, App Gateway, Traffic Manager,etc.).
Extensively involved in all phases of Data acquisition, data collection, data cleaning, model development, model validation and visualization to deliver business needs of different teams.
Implemented a Reusable plug & play Python Pattern (Synapse Integration, Aggregations, Change Data Capture, Deduplication and High Watermark Implementation.
This process accelerated the development time and standardization across teams.
Worked with Spark Core, Spark ML, Spark Streaming and Spark SQL and data bricks.
Developed analytical components using Scala, Spark, Apache Mesos and Spark Stream and Installed Hadoop, Map Reduce, and HDFS and developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing.
Implemented Kubernetes namespaces and RBAC (Role-Based Access Control) policies to enforce security and access controls in the data infrastructure.
Extracting and Analyzing data from various sources.
Data wrangling and cleanup using Python-Pandas.
Wrote and executed various MYSQL database queries from Python using Python-MySQL connector and MySQL dB package.
Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
Storing different configs in No SQL database Mongo DB and manipulating the configs using PyMongo.

Experience in using different types of stages like Transformer, Aggregator, Merge, Join, Lookup, and Sort, remove duplicated, Funnel, Filter, Pivot for developing jobs.
Build Jenkins jobs for CI/CD Infrastructure for GitHub repos.
Conducted performance tuning and optimization of Kubernetes and Docker deployments to improve overall system performance.
Used Python based GUI components for the Front End functionality such as selection criteria.
Used Python to connect to MySQL using MySQL connectors and extracted various data for customer usage reports.
Performed load testing and optimization to ensure the pipeline's scalability and efficiency in handling large volumes of data.
Developing and maintaining Azure Analysis Services models to support business intelligence and data analytics requirements, creating measures, dimensions, and hierarchies for reporting and visualization.
Involved in various phases of Software Development Lifecycle (SDLC) of the application, like gathering requirements, design, development, deployment, and analysis of the application.
Developed Spark applications using Scala and spark SQL for data extraction, transformation, and aggregation from multiple file formats for analyzing and transforming the data uncover insight into the customer usage patterns and even.
Involved in loading and transforming large sets of Structured, Semi-Structured and Unstructured data and analyzed them by running Hive queries.
Processed the image data through the Hadoop distributed system by using Map and Reduce then stored into HDFS.
Developing scalable and reusable database processes and integrating them.
Created datasets from S3 using AWS Athena and created Visual insights using AWS Quicksight Monitoring Data Quality and integrity end to end testing and reverse engineering and documented existing program and codes.
Successfully completed a POC for Azure implementation, with the larger goal of migrating on-premises servers and data to the cloud.
Developed custom reports using HTML, Python and MySQL.