Noor Farhan Kapdi

Creating and deploying applications using docker containers.
Use Airflow to orchestrate creation and destruction of cloud instances.
Monitoring and recording system metrics using tools such as node exporter and Prometheus.
Sourcing the metrics from Prometheus by using it as a source in Grafana.
Creating dashboards in Grafana to monitor system metrics and ensure health of cloud instances.
Build Spark application to read data generated from a proprietary tool that scans the open internet for
vulnerable assets.
Schedule the pipeline using Apache Airflow Scheduler.
Perform asset attribution on acquired data set and write attributed data to S3 and Elasticsearch.
Monitoring SLA(s), failures using airflow and set up notification mechanism for stakeholders.
Setup a Kafka pipeline for the company's main platform/portal.
Implement REST APIs using Flask and integrate them with real time data processing.
Work with DevOPS team to deploy changes on Kubernetes.

Build Spark application to read data generated from a proprietary tool that scans the open internet for vulnerable assets.
Schedule the pipeline using Apache Airflow Scheduler.
Perform asset attribution on acquired data set using Pyspark with data-science libraries and write attributed data to S3 and Elasticsearch.
Monitor SLA(s) and failures using Airflow and Grafana.
Set up notification mechanism for stakeholders.
Build a Spark application that uses the Nuclei Vulnerability scanner to discover vulnerabilities on the internet.
Create a mechanism in the Spark application to deploy custom template files to worker nodes.
Execute Nuclei on worker nodes and combine output from workers.
Write the combined output to S3 and Elasticsearch.
Set up Slack notification system to notify user on success, failure and SLA miss using Airflow callback.
Monitor SLA(s) and data quality using Prometheus and Grafana.
Re-implement the tool using Kafka instead of Spark and deploy it to Kubernetes.
Build a Spark application that uses the Nuclei Vulnerability screenshot template to take headless screenshots of provided IPs on a large scale using Spark.
Execute Nuclei screenshot template on worker nodes and combine output files from workers.
Write the combined output to S3 in parallel.
Set up Slack notification system to notify user on success, failure and SLA miss using Airflow callback.
Implement Airflow API to accept requests.
Implement dockerization of applications for ease of deployment.
Create a framework that allows users to call an API and execute any security analysis function on provided data.
Set up scripts to create and spin up nodes and destroy them on completion of pipeline to minimize cloud costs.
Set up notification service to track progress of pipeline at each stage.
Write unit test scripts for modules written by self as well as other peers.
Work with DevOps team to deploy changes on Kubernetes.
Set up Slack notification system to notify user on success, failure and SLA miss using Airflow callback.
Integrate existing and new modules to SiaaS.
Set up a Kafka pipeline for the company's main platform/portal.
Accept data from incoming topics and process them using data science Python libraries and functions.
Implement data pipelines for various modules.
Implement REST APIs using Flask and integrate them with real time data processing.
Write the result to the appropriate destination topic for consumption.
Work with DevOps team to deploy changes on Kubernetes.

Build Spark application to read data generated from a proprietary tool that scans the open internet for vulnerable assets.
Schedule the pipeline using Apache Airflow Scheduler.
Perform asset attribution on acquired data set using Pyspark with data-science libraries and write attributed data to S3 and Elasticsearch.
Monitoring SLA(s), failures using airflow and Grafana and set up notification mechanism for stakeholders.

Migrate all business applications to Data Lake as part of migration strategy for single source of consumption using Sqoop and Spark while maintaining the data lineage.
Orchestrate data flow using Autosys.
Built Spark-streaming application using Scala for processing real-time transactional data consuming from Oracle Golden Gate in JSON format and performing real-time replication in HBase Hadoop environment.
Maintained metadata for all tables in HBase.
Migrated transaction history tables to Hive using Sqoop and later synced over real time transaction using Kafka.
Built NiFi pipeline to store real time interactive voice response JSON data to Hadoop staging in HBase tables and AWS S3 storage layer.
Apply business rules consuming at real-time and provide respective tables to analyze and process threat monitoring and customer respective analytics.
Blacklist and whitelist priority alerts for downstream.
Build data pipeline to move XML and AVRO semi structured files to AWS S3 storage layer.
Loading data files from AWS S3 bucket to Snowflake Internal Stage.
Apply business rules to staging data stored in Snowflake Internal Stage to extract it and store it in structured Snowflake tables for downstream consumption.
Built an NLP project for a local university to extract the sentiments and opinions of students from a feedback comments dataset.
Used Python and associated libraries like pandas, nltk, and sci-kit to extract sentiments from student feedback comments in English and Arabic.
Implemented topic modeling by using the Latent Dirichlet Allocation technique to identify topics in feedback comments.
Implemented code to derive suggestions and improvement areas based on extracted opinions.
Built a real-time ETL streaming pipeline that reads and processes user activity data stored in two different formats from an AWS SQS queue and stores it in a unified SQL structure in Postgres.
Deployed the project using Docker modules.
Built data visualization using Grafana for user activity insights and to monitor data quality issues.
Deployed Prometheus DB on Grafana for storing time-series data used in monitoring the pipeline in real time to gain insight into pipeline efficiency and maintenance.

Migrate all business applications to Data Lake as part of migration strategy for single source of consumption using Sqoop, Spark system maintaining the data lineage.
Orchestrate data flow using Autosys.
Built Spark-streaming application using Scala for processing real-time transactional data consuming coming from Oracle Golden Gate in JSON format performing real-time replication in HBase Hadoop environment.
Maintained metadata for all tables in HBase.
Migrated transaction history tables to Hive using Sqoop and later synced over real time transaction using Kafka.
Build NIFI pipeline to store real time interactive voice response JSON data to Hadoop staging in HBase tables & AWS S3 storage layer.
Apply business rules consuming at real-time and provide respective tables to analyze and process threat monitoring and customer respective analytics.
Blacklist and whitelist priority alerts for downstream.
Build data pipeline to move XML and AVRO (semi structured files) to AWS S3 storage layer.
Loading data files from AWS S3 bucket to Snowflake Internal Stage.
Apply business rules to staging data stored in Snowflake Internal Stage to extract it and store it in structured Snowflake tables for downstream consumption.