Gyaneshwar Prajapati

– Designed a comprehensive data pipeline to process data from diverse sources and transform it for downstream business applications. – Defined the data flow using AWS Glue Crawlers, Jobs, and Triggers for schema generation and automated ETL execution. – Configured AWS Glue to connect to various databases and data lakes, ensuring proper access permissions and connectivity. – Integrated Pandas and AWS Lambda for additional data processing tasks. – Developed Python scripts for advanced data transformations, including XML and JSON processing for data exchange and business logic implementation. – Utilized AWS Glue Jobs to clean, enrich, and aggregate data, meeting specific pipeline requirements. – Built ETL workflows using AWS Glue, defining dependencies and execution order to ensure correct sequencing of data processing tasks. – Optimized workflow performance through directed acyclic graph (DAG) design. – Leveraged Apache Spark (with Python) for high-performance batch and interactive data processing using DataFrames and Spark SQL API. – Worked with Hive to enhance data querying capabilities within the pipeline. – Configured and managed a robust AWS environment using services such as EC2, S3, RDS, VPC, and Redshift. – Integrated tools like Hadoop, HDFS, Sqoop, and Talend for additional data management and ETL tasks.

– Develop, deploy, and maintain data pipelines using Google Dataflow to extract, transform, and load (ETL) data. – Implement efficient data ingestion and integration processes from various sources into BigQuery. – Design and optimize BigQuery datasets, tables, and schemas for efficient querying and storage. – Create and manage BigQuery views and stored procedures to facilitate data access and transformation. – Develop serverless functions using Google Cloud Functions for automation and integration purposes. – Implement event-driven processes and data triggers using cloud functions.

– Design, develop, and maintain data pipelines using Apache Airflow to automate and schedule data workflows. – Extract, transform, and load (ETL) data between different systems, including Snowflake, S3, and SFTP servers. – Connect to Snowflake databases to extract and load data, ensuring data accuracy, reliability, and performance. – Optimize SQL queries and data transfer processes within Snowflake. – Utilize PySpark for data processing, including data cleansing, transformation, and aggregation. – Establish secure connections to SFTP servers to exchange data with external partners or systems. – Integrate with Amazon S3 for storing, retrieving, and managing data objects.

– Created Script for Amazon, Ali expresses, Walmart, Asos.com to Scrap data and store the data inside MongoDB database. – Used Python for XML, JSON processing, data exchange and business logic implementation. – Expertise in developing data driven applications using Python 2.7, Python 3.0 on PyCharm and Anaconda Spyder IDE's. – Created Script for Order automation for these websites. – Developed Script on AWS server with MongoDB database.