Shadab Barmare

Project 1 (Total Energies): 1. Data migration mission from Azure to AWS. 2. AWS Glue as the main ETL tool to build pipelines to extract data from CSV, Excel files with proper encoding (Chinese, Japanese, French Turkish) from S3 bucket and sinking it to RDS (ms-sql server) using stored schema in Glue Catalog. 3.Merging of SQL, Python and Pyspark in AWS Glue to prepare a data processing pipeline from the S3 bucket to Staging AWS RDS (MSSQL Server) 4.Managing and Monitoring the task performed by juniors in order to avoid failures. 5.Working closely with Data Analyst team to meet the requirements. 6.Job role as Leader and team player. Project 2 (Total Energies – Data Integration & Orchestration ): 1.Integrating Third-party Cloud , Databricks and AWS for Data Transformation. 2. Moving the entire workload from Local to Databricks and mounting the storage in AWS. 3. Developing a code with combination of Linux and Windows server using Databricks and AWS Fargate and Uploading the results on the Private Cloud. 4. Orchestrating the workflow in accordance with the requirement and the processing complexity and time. 5. Working with Data Lead , Head of Data and Data Architect in an Agile Manner. 6. Tools used : AWS (EC2 , Fargate , S3 , Lambda) , Databricks , Private Cloud

Project 1 (Air France) : • Designing a workflow for processing pdf and image data stored in AWS S3 bucket using SNS and Lambda Function and re-stockage of the extracted data back to S3 in cleaned format. • Working with AWS Textract and Comprehend for Processing .Pdf and Image data which uses machine learning algorithm for text detection and classification. • Implementing all of the above steps using AWS CDK in Python for automation. • Providing suitable triggers and event notifications to pass the data from S3, SNS , Lambda Using CDK (Code Development KIt). Tools Used - AWS SNS ,S3 , CDK , Lambda , Textract , Comprehend , Python