Honeyshmitha

Orchestrated Data Lake setup, merging diverse Teradata and multiple source data through Hadoop stack (SQOOP, Hive/HQL).
Developed a data ingestion tool using PySpark capable of gathering data from diverse sources such as APIs, RDBMS, files, and Kafka streams.
Addressed challenges including updating hive table structures to accommodate incoming data for incremental loads, performing data type conversions, and identifying error records during file ingestions.
Enhanced ingestion performance by 20 times compared to existing tools like Stream Sets, leveraging multi-core processing tailored to the data pipeline's workload.
Optimized SQL queries for reading tables as partitions to achieve better performance through parallel processing.
Handled importing of data from various data sources, performed transformations using spark and storing into Hive.
Scheduled Autosys jobs to execute multiple Spark tasks via shell scripts, ensuring independent execution based on time, seamless data loading and resource availability.
Designed robust big data ingestion pipelines in Spark, ensuring data quality checks, transformations, and efficient storage.
Conducted comprehensive Data Wrangling on diverse datasets using PySpark across various sources.
Solved performance issues in Pyspark with understanding of groups, joins and aggregation functions and Scheduled Pyspark jobs in Cluster by using Spark UI.
Leveraged Scala scripts in Spark for aggregation, querying, and HDFS data writing via data frames/SQL and RDD/MapReduce.
Developed SQL and Hadoop artifacts: stored procedures, views, UDFs, and CTEs for streamlined data operations.
Created UNIX scripts for data reconciliation and engineered Spark jobs for user data analysis.
Demonstrated adeptness in RDBMS data extraction and transformation for ingestion, optimizing computational logic with efficient Spark code using Scala and spark-SQL.

• Orchestrated Data Lake setup, merging diverse Teradata and multiple source data through Hadoop
stack (SQOOP, Hive/HQL).
• Developed a data ingestion tool using PySpark capable of gathering data from diverse sources such as
APIs, RDBMS, files, and Kafka streams.
• Addressed challenges including updating hive table structures to accommodate incoming data for
incremental loads, performing data type conversions, and identifying error records during file
ingestions.
• Enhanced ingestion performance by 20 times compared to existing tools like Stream Sets, leveraging
multi-core processing tailored to the data pipeline's workload.
• Optimized SQL queries for reading tables as partitions to achieve better performance through parallel
processing.
• Handled importing of data from various data sources, performed transformations using spark and
storing into Hive.
• Scheduled Autosys jobs to execute multiple Spark tasks via shell scripts, ensuring independent
execution based on time, seamless data loading and resource availability.
• Designed robust big data ingestion pipelines in Spark, ensuring data quality checks, transformations,
and efficient storage.
• Conducted comprehensive Data Wrangling on diverse datasets using PySpark across various sources.
• Solved performance issues in Pyspark with understanding of groups, joins and aggregation functions
and Scheduled Pyspark jobs in Cluster by using Spark UI.
• Leveraged Scala scripts in Spark for aggregation, querying, and HDFS data writing via data frames/SQL
and RDD/MapReduce.
• Developed SQL and Hadoop artifacts: stored procedures, views, UDFs, and CTEs for streamlined data
operations.
• Created UNIX scripts for data reconciliation and engineered Spark jobs for user data analysis.
• Demonstrated adeptness in RDBMS data extraction and transformation for ingestion, optimizing
computational logic with efficient Spark code using Scala and spark-SQL.

Developed PySpark tasks to handle data processing, including reading from external sources, merging, enriching, and loading into target destinations.
Created efficient Hive tables with appropriate static and dynamic partitions to meet specific requirements.
Managed data importation from diverse sources, performed transformations using Spark, and stored results in Hive and S3 buckets.
Imported RDBMS tables into Hive using Sqoop and created data visualizations in Tableau.
Scheduled Apache Airflow DAGs to execute multiple Hive and Spark jobs independently based on time and data availability.
Migrated an on-premises application to AWS, utilizing EMR and Glue jobs for processing and storing small datasets.
Created on-demand tables on S3 files using Lambda Functions and AWS Glue with Python and Spark.
Developed and optimized complex SQL queries to extract data from SQL server databases, tuning ETL processes and SQL Queries for improved performance.
Maintained technical documentation for executing Hive queries, and actively participated in code reviews and bug fixing to enhance performance.

Engaged in all project stages including analysis, design, development, and testing.
Proficient in understanding Business Requirement Documents and crafting detailed Low-Level Design Documents.
Managed Database Objects, Tables, and Views effectively.
Developed APIs for new clients and devised corresponding business logic for scheduled and immediate transfers.
Designed projects utilizing Python Modules and Packages.
Created comprehensive unit test cases and maintained a release tracker for each software release.
Designed and developed webpages using CSS, HTML, and JavaScript, while also implementing automation resources through Jenkins.
Utilized various Python modules and controls to rapidly build applications, leveraging Python data structures and advanced coding techniques like list comprehensions, generators, lambda functions, and built-in functions such as map and filter.
Addressed performance issues through efficient coding and optimized SQL queries for improved response time, while providing development support by coordinating with production teams.