Teodoro

Worked on a hybrid kanban team leveraging Atlassian suite to keep track of work and version control.

Create data pipelines utilizing in house spark framework that uses Kubernetes to create Hadoop instance to ingest data from different data sources.

Created POC to connect Hadoop to google cloud platform.

Created pyspark/python scripts to transform and move csv files around google cloud storage

Export data to google cloud storage buckets utilizing pyspark.

Task with converting existing ETL talend work flows and converting them to spark workflow.

Worked on a hybrid kanban team leveraging Atlassian suite to keep track of work and version control.
Created data pipelines utilizing in house spark framework that uses Kubernetes to create Hadoop instance to ingest data from different data sources.
Created a proof of concept to connect Hadoop to Google Cloud Platform.
Created PySpark/Python scripts to transform and move CSV files around Google Cloud Storage.
Exported data to Google Cloud Storage buckets utilizing PySpark.
Converted existing ETL Talend workflows to Spark workflows.

Worked on an agile team with UI and Hadoop developers and data analysts.
Supported the Corporate Investment Banking Risk space and handled credit risk data.
Used Jira to keep track of work.
Used AngularJS to create UI and backend for line of business interaction.
Created data pipelines utilizing in house Spark application to ingest data from different data sources.
Exported data to SQL Server utilizing Sqoop.
Developed SQL logic to transform risk data according to line of business requirements.
Utilized Jenkins to deploy code into DEV, Testing, and production on premise Cloudera Hadoop cluster.
Attended meetings with line of business leaders to help create BRD documents.
Developed Jira epics from BRD documents.

Worked on a technical agile team that supported the bank fraud lines of business.
Utilized an internal Spark-based application called ezflow to create ETL process for Hadoop.
Created and modified configuration files to import correct data from RDBMS utilizing ezflow.
Updated Excel mapping documents for newly ETL process into the bank fraud space.

Participated in meetings with the Lines of Business to gather requirements for quarter planning.
Translated business needs into technical requirements.
Broke down requirements into Jira stories.
Utilized Bitbucket to manage code.
Created Oozie workflow that grabbed CSV files from an SFTP server and copied CSV files into HDFS and created Hive tables from the CSV files.
Analyzed existing code that ran for longer periods of time and optimized the code to reduce runtime.
Created Oozie workflow to query existing Hive tables to create new metrics table and export metrics table to Teradata for data scientist consumption.
Created Oozie workflow that scooped store production data and transformed it based on business specifications.
Exported data to Teradata or created Hive view for business consumption.
Worked on agile team to develop data pipelines utilizing Oozie, Bash, Sqoop, and Scala Spark to ingest files from internal and external servers into Hadoop.
Imported data sets from Teradata, RDBMS, and SQL Server into Hadoop Hive.
Created ETL logic and exposed US retail stores data to data analyst.
Communicated with Store Production Line of Business to determine specific data needs and data location.
Monitored emails for failed Oozie jobs in production and investigated failures to rerun jobs successfully.
Worked on Horton Works Hadoop Unix system to create data pipelines.
Utilized Bash scripting in Hadoop to execute commands including moving files into HDFS, running spark submit jobs, and running beeline commands.
Created Python script to connect Hadoop to Splunk.
Sourced inventory data from Teradata into Hadoop utilizing Sqoop.
Created tables in Teradata to export tables from Hadoop into Teradata utilizing Scala Spark.

Gathered and sourced enterprise-wide datasets into AWS Query cluster for Marketing and Analytics team requirements.
Decommissioned existing on-premises data pipelines and replaced them with AWS processes.
Created new data pipeline that spun up an ingestion cluster in AWS and ran an Oozie workflow to download CSV files from the sec.gov website and upload them to respective S3 buckets.
Used Bash and Python scripts in the workflow.
Developed Dell Boomi proof of concept.
Created a Dell Boomi job to sample query from a DB2 database and write the resulting query to an S3 bucket in AWS.

Worked on an on premise Cloudera Hadoop based system optimizing existing HQL scripts to reduce workflow runtime.
Created proof of concept using PySpark to develop an Oozie workflow that created snapshot table and gathered metric data from auto loans reporting database.
Served as lead analyst for internal audit initiative.
Led team to gather transformation logic of requested source to target elements.
Translated application code to pseudo code and queried sample data for audit team to replicate transformation logic.
Developed data quality rules and controls to test and validate data being sourced and processed in the application.
Translated Pig script logic to SQL logic.
Created PySpark process to get the sum total of metrics table for Volvo loan data.

Worked on a lean agile team of 4 other intern developers.
Created and estimated stories for sprint.
Conducted sprint planning and refinement.
Helped develop and build documentation for an internal management and reporting web-based application.
Used JavaScript and jQuery libraries to add functionality to the web app including displaying error logs, displaying inactive and active reports, and updating, adding, and deleting XML files.
Acted as lead business analyst in a project managed via agile.
Worked with other developers and business analyst to gather user stories.
Assigned development roles to team members to accomplish goals and deadlines.
Helped develop an internal web-based lineage application that visualizes data lineage by scanning source code from on premise Cloudera Hadoop.
Application graphically displays the origin and target location of the data and shows the location path of the source and target files.
Used JavaScript, HTML, and CSS for the project.