venkata paladugula

• Worked on Kafka streaming on subscriber side, processing the messages, and inserting them into the db and Apache Spark for real-time data processing.
• Worked extensively with Python in optimization of the code for better performance.
• Extensively involved in all phases of Data acquisition, data collection, data cleaning, model development, model validation and visualization to deliver business needs of different teams.
• Developed data connectors to extract data from social media APIs and integrated them into the pipeline using Python and Apache Kafka Connect.
• Implemented data transformations and enrichment using Apache Spark Streaming to clean and structure the data for analysis.
• Build Data pipelines using Python, Apache Airflow for ETL related jobs inserting data into Oracle.
• Created Data stage and ETL jobs for populating the data into Data Warehouse constantly from different source systems like ODS, flat files, Parquet.
• Developed analytical components using Scala, Spark, Apache Mesos and Spark Stream and Installed Hadoop, Map Reduce, and HDFS and developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing.
• Converted SAS codes to Python for predictive models with Pandas, NumPy and scikit-learn.
• Extensively used Data bricks spark and Jupyter notebooks for Data Analytics.
• Develop metrics based on SAS scripts on legacy system, migrating metrics to snowflake (AWS).
• Maintain code version control and hold accountability for Prod-stage validation and deployment.
• Responsible for Building and Testing of applications. Experience in handling database issues and connections with SQL and NoSQL databases like MongoDB by installing and configuring various packages in python (Teradata, MySQL, MySQL connector, PyMongo and SQLAlchemy).
• Design and configure database, Back-end applications and programs. Managed large datasets using Pandas data frames and SQL.
• Provisioned high availability of AWS EC2 instances, migrated legacy systems to AWS, and developed Terraform Plugins, modules, and templates for automating AWS infrastructure.

Worked on Kafka streaming on subscriber side, processing the messages, and inserting them into the db and Apache Spark for real-time data processing.
Worked extensively with Python in optimization of the code for better performance.
Extensively involved in all phases of Data acquisition, data collection, data cleaning, model development, model validation and visualization to deliver business needs of different teams.
Developed data connectors to extract data from social media APIs and integrated them into the pipeline using Python and Apache Kafka Connect.
Implemented data transformations and enrichment using Apache Spark Streaming to clean and structure the data for analysis.
Build Data pipelines using Python, Apache Airflow for ETL related jobs inserting data into Oracle.
Created Data stage and ETL jobs for populating the data into Data Warehouse constantly from different source systems like ODS, flat files, Parquet.
Developed analytical components using Scala, Spark, Apache Mesos and Spark Stream and Installed Hadoop, Map Reduce, and HDFS and developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing.
Converted SAS codes to Python for predictive models with Pandas, NumPy and scikit-learn.
Extensively used Data bricks spark and Jupyter notebooks for Data Analytics.
Develop metrics based on SAS scripts on legacy system, migrating metrics to snowflake (AWS).
Maintain code version control and hold accountability for Prod-stage validation and deployment.
Responsible for Building and Testing of applications.
Experience in handling database issues and connections with SQL and NoSQL databases like MongoDB by installing and configuring various packages in python (Teradata, MySQL, MySQL connector, PyMongo and SQLAlchemy).
Design and configure database, Back-end applications and programs.
Managed large datasets using Pandas data frames and SQL.
Provisioned high availability of AWS EC2 instances, migrated legacy systems to AWS, and developed Terraform Plugins, modules, and templates for automating AWS infrastructure.

Used Python to write Data into JSON files for testing Django Websites, Created scripts for data modeling and data import and export.
Building/Maintaining Docker container clusters managed by Kubernetes Linux, Bash, GIT, Docker.
Worked with AWS Terraform templates in maintaining the infrastructure as code.
Worked on Jenkins pipelines to run various steps including unit, integration and static analysis tools.
Worked with Docker containers in developing the images and hosting them in antifactory.
Led requirement gathering, business analysis, and technical design for Hadoop and Big Data projects.
Architected Python scripts for automated data extraction and loading from web server output files, reducing manual data entry, and processing time by 75%.
Spearheaded HBase setup and utilized Spark and SparkSQL to develop faster data pipelines, resulting in a 60% reduction in processing time and improved data accuracy.
Achieved 70% faster EMR cluster launch and configuration, optimized Hadoop job processing by 60%, improved system stability, and utilized Boto3 for seamless file writing to S3 bucket.
Ensured data quality and accuracy with custom SQL and Hive scripts and created data visualizations using Python and Tableau for improved insights and decision-making.
Converted and parsed data formats using PySpark Data Frames, reducing time spent on data conversion and parsing by 40%.
Implemented Python automation for Capital Analysis and Review, leveraging Pandas and NumPy modules to manipulate and analyze data, ensuring accurate reporting and streamlined decision - making.
Led migration to AWS, leveraging Amazon Redshift for data warehousing and utilizing HiveQL for reporting, reducing data retrieval and processing time by 30%.

Performed data wrangling to clean, transform and reshape the data utilizing panda’s library.
Analyzed data using SQL, Python, Apache Spark and presented analytical reports to management and technical teams.
Deployed models as python package, as API for backend integration and as services in a microservices architecture with a Kubernetes orchestration layer for the Dockers containers.
Integrated Azure Data Factory with Blob Storage to move data through Databricks for processing and then to Azure Data Lake Storage and Azure SQL data warehouse.
Used Azure Data factory to ingest data from log files and business custom applications, processed data on Data bricks per day-to-day requirements, and loaded them to Azure data lakes.
Expertise in Business intelligence and Data Visualization tools like Tableau Used Tableau to connect to various sources and build graphs.
Handled importing data from various Data sources, performed transformations using Pandas, Spark and loaded into Hive as external files using HDFS.
Involved in building database Model, APIs and Views utilizing Python, in order to build interactive web-based solutions.
Created Complex Stored Procedures, Slow Changing Dimension Type 2, Triggers, Functions, Tables, Views and other T SQL code and SQL joins to ensure efficient data retrieval.

Developed custom reports using HTML, Python and MySQL.
Developed Monitoring and notification tools using Python.
Documented the revised workflow using Confluence Wiki.
Created Data tables utilizing PyQt to display customer and policy information and add, delete, update customer records.
Involved in various phases of B of the application, like gathering requirements, design, development, deployment, and analysis of the application.
Used Python based GUI components for the Front End functionality such as selection criteria.
Implemented Navigation rules for the application and page outcomes, written controllers using annotations.
Written queries in MySQL and Native SQL.
Added the Navigations and paginations and filtering columns and adding and removing the desired columns for view.
Build Jenkins jobs for CI/CD Infrastructure for GitHub repos.
Implemented Synapse Integration with Azure Databricks notebooks which reduce about half of development work.
Achieved performance improvement on Synapse loading by implementing a dynamic partition switch.
Developed business logic using Kafka & Spark Streaming and implemented business transformations.
Supported Continuous storage in ADLS and configured Snapshots and wrote entities in spark along with named queries to interact with database.
Integrated Kubernetes with cloud-native services, such as AWS EKS and GCP GKE, to leverage additional scalability and managed services.
Used Continuous Delivery Pipeline.
Deployed microservices, including provisioning Azure environments and developed modules using Python scripting and Shell Scripting.