Spoorthi

Spoorthi

Data Engineer
United States of America

نبذة عني

Data Engineer with experience in Python, SQL, Snowflake, dbt, AWS, MySQL, and ETL tools including Talend and Informatica. Experience includes data loading, secure SFTP connections, data warehousing, REST APIs, Java, and …

الخبرة

Data Engineer

Apple
Jul 2023 - حتى الآن · 3 سنوات

• Utilized Snowflake's Python connector to establish a connection and efficiently transfer data from MySQL to the Snowflake data warehouse.
• Designed the script to handle large datasets, ensuring scalability for future data loading requirements and adherence to Snowflake best practices.
• Implemented optimizations in the Python script to enhance the performance of data loading and transformation processes, achieving 25% improvement in processing time.
• Mapped the necessary Snowflake tables that align with the data structures and semantics of the original SQL queries, facilitating a smooth transition to Snowflake as the underlying data platform.
• Established documentation for the table mapping process, providing a comprehensive reference for future updates, maintenance, and knowledge transfer within the team.
• Worked closely with dashboard developers and data analysts to address any data-related challenges during the migration, fostering a collaborative environment for successful implementation.
• Employed regular expressions and string manipulation techniques in Python to identify camel case patterns and systematically convert them to snake case.
• Conducted thorough testing and validation to verify the accuracy of the script's output, guaranteeing a seamless transition to the new naming convention.
• Implemented dbt models in accordance with industry best practices and established dbt standards, ensuring consistency, reusability, and maintainability across the data transformation processes.
• Utilized dbt's version control features to manage and track changes to dbt models, ensuring transparency and collaboration within the team.

Data Engineer

APPLE (Austin, TX)
Jul 2023 - Dec 2023 · 5 أشهر

Developed a Python script to establish secure connections to PMWeb via SFTP, ensuring efficient and secure data transfers.
Implemented a modular and reusable loader by leveraging Python libraries such as Paramiko and Sops, enhancing adaptability and scalability for diverse data integration needs.
Utilized the Paramiko library for its robust support of the SSH protocol, enabling secure and efficient communication with PMWeb's SFTP server.
Incorporated the Sops library for encryption and decryption, securing the private key used in the SFTP connection.
Enhanced security measures by implementing base64 encoding and decoding for the private key, contributing to the protection of sensitive information during key exchange.
Conducted thorough testing and validation of the script, ensuring compatibility with PMWeb's security protocols and data transfer standards.
Developed a Python script to automate the data loading process, efficiently transferring data from the .sql file to a MySQL database.
Leveraged MySQL as an intermediary database to preprocess and transform the data before loading it into Snowflake, ensuring data quality and compatibility.
Utilized Snowflake's Python connector to establish a connection and efficiently transfer data from MySQL to the Snowflake data warehouse.
Designed the script to handle large datasets, ensuring scalability for future data loading requirements and adherence to Snowflake best practices.
Implemented optimizations in the Python script to enhance the performance of data loading and transformation processes, achieving 25% improvement in processing time.
Mapped the necessary Snowflake tables that align with the data structures and semantics of the original SQL queries, facilitating a smooth transition to Snowflake as the underlying data platform.
Established documentation for the table mapping process, providing a comprehensive reference for future updates, maintenance, and knowledge transfer within the team.
Worked closely with dashboard developers and data analysts to address any data-related challenges during the migration, fostering a collaborative environment for successful implementation.
Employed regular expressions and string manipulation techniques in Python to identify camel case patterns and systematically convert them to snake case.
Conducted thorough testing and validation to verify the accuracy of the script's output, guaranteeing a seamless transition to the new naming convention.
Implemented dbt models in accordance with industry best practices and established dbt standards, ensuring consistency, reusability, and maintainability across the data transformation processes.
Utilized dbt's version control features to manage and track changes to dbt models, ensuring transparency and collaboration within the team.
Documented dbt models, including transformation logic, dependencies, and business rules, enabling knowledge transfer within the team and ensuring transparency in data transformations.

Software Dev Engineer

AMAZON (Austin, TX)
Oct 2022 - Apr 2023 · 6 أشهر

Worked as an SDE on building Distribution as a service and created a new workflow in the existing system to support the external orders.
Worked in the Multi-Channel Replenishment team (Amazon Warehouse Distribution).
Worked on debugging and refactoring the existing system to improve the success rate.
Improved matcher implementation in the Integration test suite.
Wrote Rest APIs in Java 8 and worked on Junit test cases for the APIs and business logic.
Improved the code coverage by 15% by adding unit tests and wrote integration tests also worked on the matcher implementation in the integration test suite.
Developed Java objects using Object Oriented and Multithreading concepts.
Written the use case JUNIT tests as per and used Mockito framework to confirm the expected results.
Used GIT to implement version control system.
Used POJO classes, spring configuration files to use dependency injection.

Data Modeler

LETHYA GROUP (HYD, INDIA)
Jul 2019 - Dec 2020 · 1 سنة 5 أشهر

Participated in all phases including Analysis, Design, Coding, Testing and Documentation.
Gathered Requirements and performed Business Analysis.
Extensively used SQL for Data Analysis.
Designed ER diagrams, logical model (relationship, cardinality, attributes, and, candidate keys) and physical database (capacity planning, object creation and aggregation strategies) as per business requirements.
Created Schema objects like Indexes, Views, and Sequences.
Conducted team meetings and proposed ETL strategy.
Developed ETL scripts on AWS by designing data pipelines that used EC2 instances to synchronize data between MySQL database and the Snowflake cloud data warehouse using Snow pipe, which increased data fetching speed by 40%.
Developed the required data warehouse model using star schema for the generalized model.
Identified and analyzed source data coming from Oracle, SQL server and flat files.
Used Informatica – power center for extracting, transforming and loading.
Designed ETL specification documents to load the data in target using various transformations according to the business requirements.

Software Intern

TELANGANA STATE IT DEPARTMENT (HYD, INDIA)
Mar 2018 - Aug 2018 · 5 أشهر

Worked as an intern in Telangana state IT department for 6 months on a project called “BLOCKCHAIN BASED PROPERTY RECORD MANAGEMENT SYSTEM”.
Assisted the team manager in managing the data.
Dumped the whole data from centralized database to blockchain technology.

Software Intern

S&K TECHNOLOGIES(HYD, INDIA)
Aug 2017 - Feb 2018 · 6 أشهر

Developed an efficient website using file-hierarchy attribute-based encryption scheme in Cloud Computing.
Managed the details and information of a hospital management system.
Used object-oriented languages like C# and Java along with relational databases and HTML.

المهارات

البيانات الضخمة مستودع البيانات هادوب (Hadoop) سكالا سبارك أباتشي هايف أباتشي بيغ خدمات أمازون ويب (AWS) كافكا نوسكيو إل ردشفت (خدمة قاعدة بيانات) إس 3 (خدمة تخزين البيانات على الإنترنت) Python Java Kotlin Data Structures Unix Object-Oriented Programming Spring Boot PySpark Talend Informatica MySQL Relational Database Snowflake HTML CSS JavaScript jQuery AngularJS REST API GitHub JIRA Bitbucket PuTTY WinSCP Postman Insomnia dbt SQL Visual Studio IntelliJ SFTP Paramiko Sops Base64 encoding Base64 decoding SSH JUnit Mockito Multithreading Dependency Injection EC2 Snowpipe Oracle SQL Server ETL Data Modeling Star Schema ER Diagrams
الإبلاغ عن هذا الملف الشخصي؟